Don't launch a job if the only changes are on a folder - gitlab

I want to disable some jobs if the only changes that happen in last git commit are in a docs folder. The idea is to be able to push new documentation on master (in a docs folder) without launching jobs responsible for pushing to production.
What I tried:
prod:
script:
- whatever
rules:
- changes:
- docs/*
when: never
It does not work because I want to be able to launch this job if there are changes in docs AND other files.
I also tried with a regular expression (probably wrong) but it does not seem to be taken in charge by gitlab. Something like that:
prod:
script:
- whatever
rules:
- changes:
- (docs/*)!*
when: never
I also tried a suboptimal solution, with a CI variable $CI_COMMIT_MESSAGE:
prod:
script:
- whatever
rules:
- if: $CI_COMMIT_MESSAGE == "doconly"
when: never
and it does not work, except if the variable is set manually. Or this:
prod:
script:
- whatever
except:
variables:
- if: $CI_COMMIT_MESSAGE == "doconly"
I also tried with a bash script that would set up a variable but variables can't be passed between jobs.
Here the bash script:
LAST_COMMIT=$(git log --pretty=format:'%H' -n 1)
FILES=$(git diff-tree --no-commit-id --name-only -r ${LAST_COMMIT})
doconly=true
while read -r line && $doconly; do
if [[ ! "$line" =~ docs/* ]] ; then
doconly=false
exit 1
fi
done <<< "$FILES"
exit 0

Related

`rules` condition in gitlab ci does not work

I am currently trying to build a pipeline in gitlab ci. It contains a job (here called speed) that should only be run on a specific day (e.g. the 6th of every month). The configuration yaml looks like this:
stages:
- speed
- watcher
variables:
# setup cache in root folder so gitlab cache can pick it up
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
TODAYS_DATE: "$(date +%Y-%m-%d)"
TODAYS_DAY: "$(date +%d)"
TODAYS_MONTH: "$(date +%m)"
include:
- local: "ci/ci_job_speed.yml"
- local: "ci/ci_job_watcher.yml"
speed:
extends: .speed
stage: speed
rules:
- if: '$TODAYS_DAY == "06"'
when: always
watcher:
when: always
extends: .watcher
stage: watcher
rules:
- if: '($CI_COMMIT_AUTHOR !~ /.*Gitlab Runner.*/ && $CI_COMMIT_BRANCH == "test") || $CI_PIPELINE_SOURCE != "push"'
cache:
paths:
- .cache/pip
- .cache/poetry
- .venv
key: "globalcache"
My problem is, that the speed job seems to be ignored, and only the watcher job is started, even though I expect speed to be run (since today is the 6th). The ci linter in GitLab validates my yaml as correct. Does anyboy have an idea what I am doing wrong?
I have tried different combinations as '$TODAYS_DAY == '06'', $TODAYS_DAY == '06' etc, none of them worked yet (the job is simply ignored).
Thanks in advance for any advice :)
Looking at your yaml i would suggest, that TODAYS_DAY: "$(date +%d)" is not executed as intended.
If you assigned this Variable in a bash script, the command date +%d would be executed and the result would be stored in the variable, but i think in your case $(date +%d) just gets stored as literal string.
I could be wrong about this variable, but this is my best guess.
You could consider using GitLabs scheduled Pipelines Feature: https://docs.gitlab.com/ee/ci/pipelines/schedules.html#add-a-pipeline-schedule
#joreign you are correct, the variable TODAYS_DATE was not assigned the day of the current date, but a string value "$(date +%d)". After trying a bit, I could not find a way to enforce the desired behaviour.
Instead, I used a workaround as described in https://gitlab.com/gitlab-org/gitlab/-/issues/20769#note_215109166 and set a random variable in a Pipeline scheduler and gave the ci job a corresponding run condition.
speed:
extends: .speed
stage: speed
only:
refs:
- schedules
variables:
- $REFSPEED_SCHEDULE

Suppress gitlab CI stage if push only changes README.md

I have a CI build stage that runs whenever someone pushes to the repo, and it takes a long time to run. So I want to configure a .gitlab-ci.yml rule that says if the user is only updating the documentation in README.md, it doesn't need to execute the build stage.
Following the tips in How to exclude gitlab-ci.yml changes from triggering a job, it seems like the way to do this would be to add something like:
stages:
- A
Stage A:
stage: A
rules:
- changes:
- "README.md"
when: never
However, based on the documentation in https://docs.gitlab.com/ee/ci/yaml/#ruleschanges, I think this will also suppress the stage if the push contains multiple files, if one of them is README.md.
In that situation, I want the stage to run, not be suppressed. Is there a way to handle this distinction?
I use this syntax to not trigger pipelines when modifying *.md files
workflow:
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BEFORE_SHA !~ /0{40}/'
changes:
- "{*[^.]md*,*.[^m]*,*.m,*.m[^d]*,*.md?*,*[^d]}"
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: '$CI_PIPELINE_SOURCE == "pipeline"'
- if: '$CI_COMMIT_TAG'
Following the idea from #j_b, I was able to solve this (with the caveat that it exits with pipeline having been marked as failed, as described in the code below). Here is the relevant code I added to my .gitlab-ci.yml.
stages:
- Check Only Docs Changed
- mybuild
# Stage that checks whether only documentation changed. If so, then we won't build a new release.
check-only-docs-changed:
stage: Check Only Docs Changed
script:
- BRANCH=origin/$CI_COMMIT_BRANCH
- echo "Checking commit to see if only docs have changed. "
# The line below is from
# https://stackoverflow.com/questions/424071/how-do-i-list-all-the-files-in-a-commit
- GET_CMD="git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA"
- FILES_CHANGED=`$GET_CMD`
- echo "Files in this commit:" $FILES_CHANGED
- NUM_FILES_CHANGED=`$GET_CMD | wc -l`
# We consider any file that ends in .md to be a doc file
# The "|| true" trick on the line below is to deal with the fact that grep will exit with non-zero if it doesn't find any matches.
# See https://stackoverflow.com/questions/42251386/the-return-code-from-grep-is-not-as-expected-on-linux
- NUM_DOC_FILES_CHANGED=`$GET_CMD | grep -c ".*\.md$" || true`
- echo $NUM_FILES_CHANGED "files changed," $NUM_DOC_FILES_CHANGED "of which were documentation."
- |
# We have to test whether NUM_FILES_CHANGED is > 0 because when one branch gets merged into another
# it will be 0, as will NUM_DOC_FILES_CHANGED.
if [[ $NUM_FILES_CHANGED -gt 0 && $NUM_FILES_CHANGED -eq $NUM_DOC_FILES_CHANGED ]]
then
DID_ONLY_DOCS_CHANGE="1"
# Write out the env file before we exit. Otherwise, gitlab will complain that the doccheck.env artifact
# didn't get generated.
echo "DID_ONLY_DOCS_CHANGE=$DID_ONLY_DOCS_CHANGE" >> doccheck.env
echo "Only documentation files have changed. Exiting in order to skip further stages of the pipeline."
# Ideally it would be great to not have to exit with a non-zero code, because this will make the gitlab pipeline
# look like it failed. However, there is currently no easy way to do this, as discussed in
# https://stackoverflow.com/questions/67269109/how-do-i-exit-a-gitlab-pipeline-early-without-failure
# The only way would be to use child pipelines, which is more effort than it's worth for this.
# See https://stackoverflow.com/questions/67169660/dynamically-including-excluding-jobs-in-gitlab-pipeline and
# https://stackoverflow.com/questions/71017961/add-gitlab-ci-job-to-pipeline-based-on-script-command-result
exit 1
else
DID_ONLY_DOCS_CHANGE="0"
echo "DID_ONLY_DOCS_CHANGE=$DID_ONLY_DOCS_CHANGE" >> doccheck.env
fi
# The section below makes the environment variable available to other jobs, but those jobs
# unfortunately cannot access this environment variable in their "rules:" section to control
# whether they execute or not.
artifacts:
reports:
dotenv: doccheck.env

Changes not triggering workflow and jobs based on defined condition

I have following workflow configurations:
workflow:
rules:
- if: '$CI_COMMIT_REF_NAME == "master"'
changes:
- /*script.yaml
variables:
CI_COMMIT_TIMESTAMP: $CI_COMMIT_TIMESTAMP
ROOT_DIR: ${CI_PROJECT_DIR}
SCRIPT_PATH: /etc/executable/scripts/
A_REPOSITORY_SCRIPT_PATH: /folder/a/script.yaml
B_REPOSITORY_SCRIPT_PATH: /folder/b/script.yaml
stages:
- init
# A Workflow:
deploycheck a 1/6:
rules:
- changes:
- ${A_REPOSITORY_SCRIPT_PATH}
stage: init
script: |
[ -f "${SCRIPT_PATH}/script.yaml" ] && cp -v "${SCRIPT_PATH}/script.yaml" ${SCRIPT_PATH}/SCRIPT_yaml_$CI_COMMIT_TIMESTAMP.txt
# B Workflow:
deploycheck b 1/6:
rules:
- changes:
- ${B_REPOSITORY_SCRIPT_PATH}
stage: init
script: |
[ -f "${SCRIPT_PATH}/script.yaml" ] && cp -v "${SCRIPT_PATH}/script.yaml" ${SCRIPT_PATH}/SCRIPT_yaml_$CI_COMMIT_TIMESTAMP.txt
I want this workflow to trigger when a change is made to any script.yaml file anywhere inside the repository and pushed to master branch.
And once the workflow is triggers depending on the path and which script.yaml was changes respective jobs should run.
However, nothing happens with this and no matter which script.yaml I change this workflow never gets triggered.
What am I missing here?
Your glob pattern would only match the root directory.
To match in any directory, use "**/script.yaml"

Gitlab CI CD only run Pipeline Step when certain criteria is met

I am trying to configure my CI/CD pipeline so that a certain step is only executed if some conditions are met.
My current rule definition looks like this:
rules:
- if: '($CI_PIPELINE_SOURCE == "push" || $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "merge_request_event") && $CI_COMMIT_BRANCH == "develop" '
exists:
- $MY_FILE
What I want to achieve:
Execute step only if a merge from branch_x was done into develop
Or Execute step when pipeline triggered over the GitLab UI
Or Execute step if a commit is pushed directly to develop
And the file in $MY_FILE is present in the repository
Other than expected, the step gets not executed on
Push
Merge Branch_X into develop
Manually trigger over GitLab UI
The corresponding file exists within the repository.
The step work is if I change back to
only:
- develop
But then I can not set the condition that the file needs to be present.
Part of the issue you're running into is that some Predefined Variables only exist in certain types of pipelines. For example, the CI_COMMIT_TAG variable, which would hold the name of a tag, only exists if the pipeline is a "Tag pipeline" (ran after a tag is created). However, if a tag points to a commit that is the HEAD of a branch that is the Source of a Merge Request, it is still just a Tag pipeline, so none of the Merge Request specific variables will exist.
Due to this, your conditional will get slightly more complex since we'll need some more parenthesis, or more if clauses for our rules section:
rules:
- if: ($CI_PIPELINE_SOURCE == 'push' || $CI_PIPELINE_SOURCE == 'web') && $CI_COMMIT_REF_NAME == 'develop'
exists:
- $MY_FILE
when: always
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'develop'
exists:
- $MY_FILE
when: always
- when: never
Note: you could write this using only one if clause and just use more parenthesis.
Let's look at these changes one by one.
First, the push and web pipeline sources are more or less identical in terms of how the pipeline is ran, and which Predefined Variables are available.
Second, the variable $CI_COMMIT_BRANCH is only available in certain pipelines. Most noticeably, it only exists when there is a Branch. If the pipeline runs for a tag, the variable will not exist, it's not just empty. If it's a merge request event, the variable will not exist. $CI_COMMIT_REF_NAME is more reliable, even though it can have more values (Commit SHAs, Branch names, or Tag names).
Third, the merge_request_event source is a totally different animal compared to the other sources we're dealing with here. The Predefined Variables available are totally different since there are now two branches (the source and the target). Also, a merge_request_event pipeline can only ever run if you have rules in your pipeline definition for merge_request_event's.
I ran some tests with this rules scenario, with this pipeline definition:
stages:
- run
Run Job:
stage: run
image: alpine:latest
script:
- echo $CI_PIPELINE_SOURCE
- echo $CI_COMMIT_REF_NAME
- echo $CI_COMMIT_BRANCH
- echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
- echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
Second Run Job:
stage: run
image: alpine:latest
rules:
- if: ($CI_PIPELINE_SOURCE == 'push' || $CI_PIPELINE_SOURCE == 'web') && $CI_COMMIT_REF_NAME == 'develop'
exists:
- 'a_file'
when: always
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'new_branch'
exists:
- 'a_file'
when: always
- when: never
script:
- echo $CI_PIPELINE_SOURCE
- echo $CI_COMMIT_REF_NAME
- echo $CI_COMMIT_BRANCH
- echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
- echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
Here's the job output for a normal Push event to a branch other than develop:
Executing "step_script" stage of the job script
00:02
$ echo $CI_PIPELINE_SOURCE
push
$ echo $CI_COMMIT_REF_NAME
other_branch
$ echo $CI_COMMIT_BRANCH
other_branch
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
Note: only the first job runs since the branch isn't develop. If the branch is develop and a_file exists, both jobs run, and the output is identical.
Here's the output of a job with the 'web' source, to a branch other than develop:
Executing "step_script" stage of the job script
00:00
$ echo $CI_PIPELINE_SOURCE
web
$ echo $CI_COMMIT_REF_NAME
other_branch
$ echo $CI_COMMIT_BRANCH
other_branch
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
Again, in this case there is only one job in the pipeline. If instead we hit Run Pipeline for the develop branch and a_file exists, we run both jobs and the output is identical:
Executing "step_script" stage of the job script
00:00
$ echo $CI_PIPELINE_SOURCE
web
$ echo $CI_COMMIT_REF_NAME
develop
$ echo $CI_COMMIT_BRANCH
develop
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
Lastly, if we push to a branch that's the source of a merge request targeting develop with a_file in the repository, we get the merge_request_event, but only get the second job due to the way Merge Request Pipelines work:
Executing "step_script" stage of the job script
00:01
$ echo $CI_PIPELINE_SOURCE
merge_request_event
$ echo $CI_COMMIT_REF_NAME
some_other_branch
$ echo $CI_COMMIT_BRANCH
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
some_other_branch
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
new_branch
Here's the documentation for Merge Request Pipelines for more information.

how to build a rule which is based on dotenv variable which was defined previously in same pipeline

I'm setting a variable in a job in the pipeline and use it in the artifacts / dotenv.
It doesn't seem to work to use the variable as a condition in the rules.
How can I set a env variable and use it in another Job's rule?
Thanks a lot for your thoughts!
include:
- local: '/gitlab/cicd/.gitlab-ci_test.yml'
stages:
- build
- test
build_rule:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_MESSAGE =~ /R::/m
script:
- echo "Hello World"
- |
echo "RUNTESTS=1" > gitlabcicd.env
artifacts:
reports:
dotenv: gitlabcicd.env
test_always:
stage: test
script:
- echo "TestEnv" $RUNTESTS
# prints TestEnv 1
test_sometimes:
stage: test
rules:
- if: $RUNTESTS
# $RUNTESTS == "1" doesnt work either
script:
- echo "Runs only if variable was set"
# doesnt run, even if it prints above
As I posted in a comment before, but just so people can find it more easily, the answer is that, unfortunately, right now it is not possible to do that.
There is an issue about it here: https://gitlab.com/gitlab-org/gitlab/-/issues/235812
No satisfactory workaround that I'm aware of.
Is needed to load the .env file on every job
"Set variables":
stage: preparation
script:
- echo version=1.1 > varfile
- echo foo=bar >> varfile
artifacts:
paths:
- varfile
Dothings:
stage: run
script:
- load varfile
- echo ${foo}

Resources