How to reference a changes array hash from a block this is itself referenced on a CI configuration? - gitlab

I'm writing a GitLab CI workflow in order to be re-used accross by project, using Docker for build and deployment.
On each project, I would like to only specify the needed changes to trigger this workflow under a merge request like so:
# .gitlab-ci.yml
include:
- { project: internal, ref: main, file: workflow.gitlab-ci.yml }
.workflow:changes:
- Dockerfile
- docker-compose.yml
- 'package*.json'
I was able to achieve it with the following:
# workflow.gitlab-ci.yml
build:
stage: build
script: docker build
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes: !reference [.workflow:changes]
test:
stage: test
script: docker test
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes: !reference [.workflow:changes]
However, I also want to store my repeated rules conditions onto configuration I can reference to them like described on the official documentation:
# workflow.gitlab-ci.yml
.rules:production:
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
.rules:staging:
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
.rules:review:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes: !reference [.workflow:changes]
build:
stage: build
script: docker build
rules:
- !reference [.rules:production, rules]
- !reference [.rules:staging, rules]
- !reference [.rules:review, rules]
test:
stage: test
script: docker test
rules:
- !reference [.rules:staging, rules]
- !reference [.rules:review, rules]
The .rules:review section has the exact same configuration as when it was placed under the build and test section of the first example. However, it give me the following invalid configuration error:
Unable to create pipeline
jobs:build:rules:rule:changes should be an array or a hash
Is that a GitLab CI limitation of the !reference keyword? Is there any workaround?
Note: The other .rules:production and .rules:staging configuration block are working properly.

Related

Gitlab CI execute SAST jobs only when merging branch to master

Hello I'm trying to figure out how to run SAST jobs only when merging branch into master because they last 5 minutes and they are being run in every push for any branch.
This means that every time someone makes a push to their MR branch the security stage is executed with all SAST jobs.
What I want to achieve is that SAST jobs are executed when the branch is going to be merged to master.
gitlab-ci.yml:
include:
- template: Jobs/SAST.gitlab-ci.yml
stages:
- security
- tests
my_tests:
stage: tests
script:
- echo Running tests ...
sast:
stage: security
What I tried so far is using:
sast:
stage: security
only:
- master
But it fails because the included template Jobs/SAST.gitlab-ci.yml already uses rules and rules with only/except can't be used together.
jobs:sast config key may not be used with rules: only
In the sourcecode Jobs/SAST.gitlab-ci.yml does not use except but rules which also are incompatible with only.
But you could also switch to rules syntax:
sast:
stage: security
rules:
- if: "$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH"
when: always
That should do the trick
You can use this configuration:
include:
- template: Jobs/SAST.gitlab-ci.yml
phpcs-security-audit-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.php'
semgrep-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.go'
- '**/*.html'
- '**/*.js'
- '**/*.jsx'
- '**/*.ts'
- '**/*.tsx'
gosec-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.go'
nodejs-scan-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/package.json'
bandit-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.py'
flawfinder-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.c'
- '**/*.cpp'
eslint-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.html'
- '**/*.js'
- '**/*.jsx'
- '**/*.ts'
- '**/*.tsx'
spotbugs-sast:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
exists:
- '**/*.groovy'
- '**/*.java'
- '**/*.scala'
- '**/*.kt'

How to avoid duplicated pipeline start in case of commit to branch related to merge request?

My aim: add "run pipeline" button to merge request -> pipelines page
Initially I had following .gitlab-ci.yaml file:
include:
- project: <erased>
ref: latest
file:
- /pipelines/my_chart.yaml
...
...
my_chart.yaml:
workflow:
rules:
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAG
..
I started to google and faced following article:
https://docs.gitlab.com/ee/ci/pipelines/merge_request_pipelines.html#:~:text=Run%20when%20you%3A,tab%20in%20a%20merge%20request
so I've modified(added the single line) my_chart.yaml like this:
workflow:
rules:
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAG
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
...
This changed added "run pipeline" button to merge request pipeline page but if I have a MR and make additional commit to the branch - 2 pipelines are started simultaneously. It is redundant - only 1 pipeline should be started.
How can I achieve it ?
P.S.
this one doesn't work
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
- if: $CI_COMMIT_BRANCH && $CI_PIPELINE_SOURCE != 'merge_request_event'
- if: $CI_COMMIT_TAG

Rules:changes runs always while only:changes runs never

I have a parent pipeline as follows:
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "web"
- if: $CI_MERGE_REQUEST_IID
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- if: $CI_COMMIT_BRANCH == "master"
stages:
- triggers
deidentify:
stage: triggers
trigger:
include: microservice_a/.gitlab-ci.a.yml
strategy: depend
only:
changes:
- microservice_a/*
ebridge_export:
stage: triggers
trigger:
include: microservice_b/.gitlab-ci.b.yml
strategy: depend
only:
changes:
- microservice_b/*
If I push any changes under microservice_a/ e.g. microservice_a/.gitlab-ci.a.yml it NEVER starts any pipeline, but if I change
only:
changes:
- microservice_a/*
To
rules:
- changes:
- microservice_a/*
On both, it will always run both regardless of whether the change was actually under microservice_a or microservice_b.
Now, the child pipelines both have a small test job that looks like this:
stages:
- test
test:
stage: test
rules:
- when: always
If I remove the rules section on both (while using rules in the parent pipeline), both will fail because it couldn't find any jobs to add to the pipeline
What am I doing wrong?

Exclude merge_request, push to create jobs in gitlab CI pipeline

workflow:
rules:
- if : '$CI_COMMIT_BRANCH == "Sprint-Release-Branch"'
when: never
- if : '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_PIPELINE_SOURCE == "push"'
when: never
- when: always
stages:
- Stage1
- Stage2
- Stage3
Task1:
stage: Stage1
script:
- echo "Stage1"
rules:
- if: '$CI_COMMIT_TAG =~ /^\d+\.\d+\.DEV\d+/'
tags:
- Runner
Task2:
stage: Stage1
script:
- echo "Checking code standard as per "Coding Standards""
rules:
- if: '$CI_COMMIT_TAG =~ /^\d+\.\d+\.DEV\d+/'
allow_failure: true
tags:
- Runner
Task3:
stage: Stage2
script:
- echo "Stage2"
when: manual
tags:
- Runner
Task4:
stage: Stage3
script:
- echo "Stage3"
when: manual
tags:
- Runner
Above is my Gitlab CI file, where i am trying
pipeline should not add jobs when there is merge & push requests happened on "Sprint-Release-Branch"
but whenever any merge request done "feature branches" onto "Sprint-Release-Branch" jobs which are defined as "when: manual" get added in pipeline.
So in my situation, Dev team is creating different feature branch for different user-stories, and then merging those features branches onto Sprint-Release-Branch having above yml file. So multiple jobs are getting added in pipeline continuously for every merge_request which are defined with "manual" trigger
How can i make optimized my yml so that jobs having manual trigger should not get added in pipeline.
which are defined as "when: manual" get added in pipeline.
You have to repeat the whole logic when you overwrite it.
Task3:
...
rules:
- if : '$CI_COMMIT_BRANCH == "Sprint-Release-Branch"'
when: never
- if : '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_PIPELINE_SOURCE == "push"'
when: never
- when: manual
Also, it's better to use when: on_success, not always.
Do something like the following with yaml anchors:
.myrules: &myrules
if: $CI_COMMIT_BRANCH == "Sprint-Release-Branch" || ($CI_PIPELINE_SOURCE == "merge_request_event" && $CI_PIPELINE_SOURCE == "push")
when: never
workflow:
rules:
- *myrules
- when: on_success
Task3:
...
rules:
- *myrules
- when: manual

Gitlab-ci: Run CI job with rules, but use a definition to avoid CI run for tags

I'm using rule method in my stage. But with rule method, I don't want to trigger CI run for tags.
I need to find a way to avoid tags like except: - tags. Unfortunately except not working with rules. In shortly how can I avoid it?
I use this snippet in my stage but it did not work, it still triggers the pipeline after the creation of the tag.
- if: '$CI_COMMIT_TAG'
when: never
This is my main deployment stage:
deploy:
stage: deployment
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $sample == "one"'
when: manual
- if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
when: never
- if: '$CI_COMMIT_TAG'
when: never
- if: '$CI_COMMIT_BRANCH && $sample == "one"'
when: manual
script:
- ...
- ...
I resolved the problem. Just I added - when: never at the end of the rules. As a separate rule, it will block all different trigger cases.
deploy:
stage: deployment
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $sync == "one"'
when: manual
- if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
when: never
- if: '$CI_COMMIT_TAG'
when: never
- if: '$CI_COMMIT_BRANCH && $sync == "false"'
when: manual
- when: never
script:
- ...
- ...

Resources