I try to trigger another repo only if in current repo I add TAG to commit. Unfortunately when I want to pass variable as well, this job not starting, if i comment out variable key - its working.
test:
stage: build
trigger:
project: php/project-to-run
branch: develop
rules:
- if: $CI_COMMIT_TAG == 'tst'
variables:
CI_COMMIT_TAG: "$CI_COMMIT_TAG"
Instead of rules i also tried "only" and it works ONLY without passing variable
only:
refs:
- tags
variables:
- $CI_COMMIT_TAG =~ /^stage_[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/
Suppose we have a .gitlab-ci.yml file that reads in a common.gitlab-ci.complete.yml file:
include: common.gitlab-ci.complete.yml
Suppose that common.gitlab-ci.complete.yml has the following stages:
stages:
- build
- unit-test
- mutation-test
In the .gitlab-ci.yml file, how do we ignore the unit-test and mutation-test stages? Would it be something like:
include: common.gitlab-ci.complete.yml
stages:
- build
#- unit-test
#- mutation-test
Would these stages override the one in the common.gitlab-ci.complete.yml file?
(I have not yet checked this code for accuracy. Treat it as pseudocode.)
You're asking to remove all jobs in a stage. If you look at the documentation for the include: keyword, you'll note that the YAML is always merged with your current CI script. So no, you can't just add a statement like "Disable all jobs in stage A".
Here's an idea... recall that
If a stage is defined but no jobs use it, the stage is not visible in the pipeline...
https://docs.gitlab.com/ee/ci/yaml/#stages
So what you could do is add a mechanism for disabling all jobs in that stage. Then the stage will disappear as well. That mechanism might be checking a variable in rules.
So if you wrote the common.gitlab-ci.complete.yml with templates that check a variable to disable the job...
stages:
- build
- unit-test
- package
.unit-test-stage:
stage: unit-test
when: on_success
rules:
- if: $DISABLE_UNIT_TESTS
when: never
unit-test-job-a:
extends: .unit-test-stage
script:
- echo "do stuff"
unit-test-job-b:
extends: .unit-test-stage
script:
- echo "do more stuff"
Then maybe your top-level .gitlab-ci.yml would simply set that known variable in its global variables section like this:
include: common.gitlab-ci.complete.yml
variables:
DISABLE_UNIT_TESTS: 1
I am using below to trigger pipeline if any changes are made in the folders under root directory.
trigger_serviceA:
stage: triggers
rules:
- if: '$CI_COMMIT_BRANCH == "dev"'
changes:
- serviceA/*
when: always
trigger:
include: serviceA/.gitlab-ci.yml
strategy: depend
However, pipeline is not getting triggered if there are any changes in the subfolders under serviceA.
When using only:changes the single wildcard does not covers subdirs - you'll have to use /**/* instead.
Altough the documentation does not say it, i could imagine that it's the same for rules:changes - so maybe you'll want to try that out.
I am trying to set up a CI with minimal code duplication using .gitlab-ci.yml.
With that, I am separating the configuration in separate files and reusing parts of it that are common.
I have a separate repository with Gitlab CI settings: gitlab-ci and several projects that use it to form their own CI pipelines.
Contents of gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core
Project with code contents:
In the actual project (separate repositories per project, and I have a lot of them), I have the following:
.gitlab-ci.yml:
stages:
- test
include:
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
This configuration works fine and allows to include only some architectures for some modules while sharing rules between the job templates.
However, it's easy to notice one code duplication: both jobs_architectureA.yml and jobs_architectureB.yml contain a common section:
.script_core: &script_core
- echo "Running stage"
It would be ideal to move it into a separate file: template_scripts.yml and include from both jobs_architectureA.yml* and jobs_architectureB.yml. However, that results in the invalid YAML (at least from Gitlab's point of view).
With that, I make a conclusion that I can share the rules as the mechanism of their usage is via extends keyword; however, I am not able to do it with the scripts: as it uses &/* anchoring mechanic on the YAML level.
Ideally, I want something along the lines of:
Contents of the ideal (conceptually) gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
template_scripts.yml:
.script_core: &script_core
- echo "Running stage"
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
Am I doing something wrong?
Am I hitting a limitation of the Gitlab mechanic? Is it the implementation of the include directive in this specific YML type, that limits me?
Do I have options to achieve something close to the desired behaviour?
Note, while this might not look like a big deal, in reality, I have many more pieces to the scripts, and the actual script is much larger. Thus, currently, it is duplicated code all over the place which is very prone to mistakes.
my solution is to not include template_jobs.yml and template_scripts.yml directly in jobs_architectureA.yml but only in the "final" .gitlab-ci.yml
taking you exemple, /template_jobs.yml//template_scripts.yml do not change.
jobs_architectureA.yml loses the include:
test_archA:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
and .gitlab-ci.yml becomes:
stages:
- test
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
in reality, I have many more pieces to the scripts, and the actual script is much larger
Adding to Cyril's solution, GitLab 13.12 (May 2021) can help scale those includes:
Support wildcards when including YAML CI/CD configuration files
The includes: keyword for CI/CD pipelines lets you break one long .gitlab-ci.yml file into multiple smaller files to increase readability.
It also makes it easier to reuse configuration in multiple places.
Frequently there are multiple files included into a single pipeline, and they all might be stored in the same place.
In this release, we add support to use the * wildcard with the local includes: keyword. You can now make your includes: sections more dynamic, less verbose, and easier to read, check out how we are dogfooding it in GitLab.
See Documentation and Issue.
The extends: keyword can be used with multiple jobs to extend from (on recent versions of GitLab). You can simply do the following:
include:
- local: '/template_jobs.yml'
.script_core:
- echo "Running stage"
test_archA:
extends:
- .sample
- .script_core
stage: test
tags:
- architectureA
And after that move the .script_core to an import.
I’d like to use the artifacts created by the Security/SAST.gitlab-ci.yml template in my final pipeline stage (reporting).
How can I modify the Security/SAST.gitlab-ci.yml template to store the artifacts somewhere in my project dir? I tried to define the following for this template, but this is not working:
artifacts:
paths:
- binaries/
I’d be happy for every kind of support.
Thank you
Solution
Your parameters need to be updated. Since SAST.gitlab-ci.yml cannot be updated directly, you need to either override one of the blocks from your gitlab-ci.yml which includes the file, or define and include your custom SAST.gitlab-ci.yml. It seems like you can get away with simply overriding the sast block. Specifically, override the artifacts -> reports -> sast parameter.
Example
sast:
stage: test
artifacts:
reports:
sast: gl-sast-report.json
You also need to ensure the stages and build step is something resembling
stages:
- build
- test
include:
- template: Security/SAST.gitlab-ci.yml
build:
stage: build
script:
- ...
artifacts:
paths:
- binaries/
References
Gitlab SAST: https://docs.gitlab.com/ee/user/application_security/sast/