gitlab has introduces rules and changes that allow specifying jobs to run when changes occur.
I wonder how can I configure the job to be skipped or ignore a path?
For example, I want to run the ci for any change unless it is a change in the docs path.
If a change occured both in the src and docs then the job should be executed,
but if the change was only in the docs folder, then the job should not be executed.
Update 2
GitLab uses globs for their rules: changes. There is no direct way of excluding folders in standard globs, but we can work around it and include everything except files in the folder docs:
myjob:
script: build myproject
rules:
- changes:
- "*"
- "?/**/*"
- "??/**/*"
- "???/**/*"
- "[^d]???/**/*"
- "?[^o]??/**/*"
- "??[^c]?/**/*"
- "???[^s]/**/*"
- "?????*/**/*"
when: on_success
The first * matches all files. The second to forth glob lines matches everything in folders that have 1-3 characters. The fifth line matches everything inside of folders that are four characters long, except if it starts with a d. The line after this matches all folders that are four characters long, except if the second char is a o. The two next lines does the same for c and s, and together these four lines matches all folders that have four characters, except docs.
The last line matches everything inside of folders that are five or more characters long.
Update
When I understood the question properly:
myjob:
script: build myproject
rules:
- changes:
- docs/**/*
when: never
- when: on_success
This job is skipped if there are changes to docs, and otherwise it behaves as a normal job and is run if all jobs from the previous stage has succeeded.
Old answer
Something like this should work:
myjob:
script: build myproject
rules:
- changes:
- src/**/*
when: on_success
- changes:
- docs/**/*
when: never
The rules are evaluated from top to bottom. If there are changes to any file in the src directory, the job will get the attribute when: on_success. This is the default for GitLab jobs, and means that this job will be executed if all jobs of the previous stage has been executed.
If the first rule doesn't match, the second job will be evaluated. If there are any changes to the docs folder, the job will get the attribute when: never, and the job will never execute.
The second rule could be changed to:
- when: never
But I imagine that your real task is a bit more complicated that your question, and that you might need this whole rule.
Related
I am currently trying to build a pipeline in gitlab ci. It contains a job (here called speed) that should only be run on a specific day (e.g. the 6th of every month). The configuration yaml looks like this:
stages:
- speed
- watcher
variables:
# setup cache in root folder so gitlab cache can pick it up
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
TODAYS_DATE: "$(date +%Y-%m-%d)"
TODAYS_DAY: "$(date +%d)"
TODAYS_MONTH: "$(date +%m)"
include:
- local: "ci/ci_job_speed.yml"
- local: "ci/ci_job_watcher.yml"
speed:
extends: .speed
stage: speed
rules:
- if: '$TODAYS_DAY == "06"'
when: always
watcher:
when: always
extends: .watcher
stage: watcher
rules:
- if: '($CI_COMMIT_AUTHOR !~ /.*Gitlab Runner.*/ && $CI_COMMIT_BRANCH == "test") || $CI_PIPELINE_SOURCE != "push"'
cache:
paths:
- .cache/pip
- .cache/poetry
- .venv
key: "globalcache"
My problem is, that the speed job seems to be ignored, and only the watcher job is started, even though I expect speed to be run (since today is the 6th). The ci linter in GitLab validates my yaml as correct. Does anyboy have an idea what I am doing wrong?
I have tried different combinations as '$TODAYS_DAY == '06'', $TODAYS_DAY == '06' etc, none of them worked yet (the job is simply ignored).
Thanks in advance for any advice :)
Looking at your yaml i would suggest, that TODAYS_DAY: "$(date +%d)" is not executed as intended.
If you assigned this Variable in a bash script, the command date +%d would be executed and the result would be stored in the variable, but i think in your case $(date +%d) just gets stored as literal string.
I could be wrong about this variable, but this is my best guess.
You could consider using GitLabs scheduled Pipelines Feature: https://docs.gitlab.com/ee/ci/pipelines/schedules.html#add-a-pipeline-schedule
#joreign you are correct, the variable TODAYS_DATE was not assigned the day of the current date, but a string value "$(date +%d)". After trying a bit, I could not find a way to enforce the desired behaviour.
Instead, I used a workaround as described in https://gitlab.com/gitlab-org/gitlab/-/issues/20769#note_215109166 and set a random variable in a Pipeline scheduler and gave the ci job a corresponding run condition.
speed:
extends: .speed
stage: speed
only:
refs:
- schedules
variables:
- $REFSPEED_SCHEDULE
Given the following very simple .gitlab-ci.yml pipeline:
---
variables:
KEYCLOAK_VERSION: 20.0.1 # this should be populated from reading a file from the repo...
stages:
- test
build:
stage: test
script:
- echo "$KEYCLOAK_VERSION"
As you might see, this simply outputs the value of KEYCLOAK_VERSION defined in the variables section.
Now, the Git repository contains a env.properties file with KEYCLOAK_VERSION=20.0.1 as content. How would I read the variable from that file and use it in the GitLab pipeline?
The documentation mentions import but this seems to be using YAML files.
To read variables from a file you can use the source or . command.
script:
- source env.properties
- echo $KEYCLOAK_VERSION
Attention:
One reason why you might not want to do it this way is because whatever is in env.properties will be run in your shell, such as rm -rf /, which could be very dangerous.
Maybe you can take a look here for some other solutions.
I have a CI build stage that runs whenever someone pushes to the repo, and it takes a long time to run. So I want to configure a .gitlab-ci.yml rule that says if the user is only updating the documentation in README.md, it doesn't need to execute the build stage.
Following the tips in How to exclude gitlab-ci.yml changes from triggering a job, it seems like the way to do this would be to add something like:
stages:
- A
Stage A:
stage: A
rules:
- changes:
- "README.md"
when: never
However, based on the documentation in https://docs.gitlab.com/ee/ci/yaml/#ruleschanges, I think this will also suppress the stage if the push contains multiple files, if one of them is README.md.
In that situation, I want the stage to run, not be suppressed. Is there a way to handle this distinction?
I use this syntax to not trigger pipelines when modifying *.md files
workflow:
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BEFORE_SHA !~ /0{40}/'
changes:
- "{*[^.]md*,*.[^m]*,*.m,*.m[^d]*,*.md?*,*[^d]}"
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: '$CI_PIPELINE_SOURCE == "pipeline"'
- if: '$CI_COMMIT_TAG'
Following the idea from #j_b, I was able to solve this (with the caveat that it exits with pipeline having been marked as failed, as described in the code below). Here is the relevant code I added to my .gitlab-ci.yml.
stages:
- Check Only Docs Changed
- mybuild
# Stage that checks whether only documentation changed. If so, then we won't build a new release.
check-only-docs-changed:
stage: Check Only Docs Changed
script:
- BRANCH=origin/$CI_COMMIT_BRANCH
- echo "Checking commit to see if only docs have changed. "
# The line below is from
# https://stackoverflow.com/questions/424071/how-do-i-list-all-the-files-in-a-commit
- GET_CMD="git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA"
- FILES_CHANGED=`$GET_CMD`
- echo "Files in this commit:" $FILES_CHANGED
- NUM_FILES_CHANGED=`$GET_CMD | wc -l`
# We consider any file that ends in .md to be a doc file
# The "|| true" trick on the line below is to deal with the fact that grep will exit with non-zero if it doesn't find any matches.
# See https://stackoverflow.com/questions/42251386/the-return-code-from-grep-is-not-as-expected-on-linux
- NUM_DOC_FILES_CHANGED=`$GET_CMD | grep -c ".*\.md$" || true`
- echo $NUM_FILES_CHANGED "files changed," $NUM_DOC_FILES_CHANGED "of which were documentation."
- |
# We have to test whether NUM_FILES_CHANGED is > 0 because when one branch gets merged into another
# it will be 0, as will NUM_DOC_FILES_CHANGED.
if [[ $NUM_FILES_CHANGED -gt 0 && $NUM_FILES_CHANGED -eq $NUM_DOC_FILES_CHANGED ]]
then
DID_ONLY_DOCS_CHANGE="1"
# Write out the env file before we exit. Otherwise, gitlab will complain that the doccheck.env artifact
# didn't get generated.
echo "DID_ONLY_DOCS_CHANGE=$DID_ONLY_DOCS_CHANGE" >> doccheck.env
echo "Only documentation files have changed. Exiting in order to skip further stages of the pipeline."
# Ideally it would be great to not have to exit with a non-zero code, because this will make the gitlab pipeline
# look like it failed. However, there is currently no easy way to do this, as discussed in
# https://stackoverflow.com/questions/67269109/how-do-i-exit-a-gitlab-pipeline-early-without-failure
# The only way would be to use child pipelines, which is more effort than it's worth for this.
# See https://stackoverflow.com/questions/67169660/dynamically-including-excluding-jobs-in-gitlab-pipeline and
# https://stackoverflow.com/questions/71017961/add-gitlab-ci-job-to-pipeline-based-on-script-command-result
exit 1
else
DID_ONLY_DOCS_CHANGE="0"
echo "DID_ONLY_DOCS_CHANGE=$DID_ONLY_DOCS_CHANGE" >> doccheck.env
fi
# The section below makes the environment variable available to other jobs, but those jobs
# unfortunately cannot access this environment variable in their "rules:" section to control
# whether they execute or not.
artifacts:
reports:
dotenv: doccheck.env
I have a folder structure like
foo/bar/lorem/a.txt
foo/bar/lorem/b.txt
foo/bar/lorem/c.ext
foo/bar/ipsum/p.txt
foo/bar/ipsum/q.ext
In GitLab CI's yml artifacts I want to include everything in foo/bar, exclude *.txt but include b.txt
The GitLab CI reference for artifacts says that:
Wildcards can be used that follow the glob patterns and golang's filepath.Match.
Try 1:
job1:
artifacts:
paths:
- foo/bar/
- foo/bar/lorem/b.txt
exclude:
- foo/bar/**/*.txt
Try 2:
job1:
artifacts:
paths:
- foo/bar/
exclude:
- foo/bar/**/!(b).txt
Expected output:
foo/bar/lorem/b.txt
foo/bar/lorem/c.ext
foo/bar/ipsum/q.ext
What paths and exclude combination do I use to achieve this?
You can do this:
job1:
artifacts:
paths:
- foo/bar/lorem/b.txt
- foo/bar/*/*.ext
Although the previous answer is good and correct, it relies on a particular case where there are only two types of files. Having a whole hierarchy with multiple file types, that could possibly be different for each job, makes it very hard to apply to other cases.
Here is an answer, a bit more difficult to set up, but that covers more cases, if someone else is looking for an answer:
At the end of your script section, delete the files you don't want. For example:
job1:
script:
- # Perform the actions you already do
- mkdir keep # Create a place to temporarly keep your files
- mv foo/bar/lorem/b.txt keep/ # Move the file to the safe place
- rm foo/bar/*.txt # Delete all files you want to delete
- mv keep/b.txt foo/bar/lorem/ # Put back the file you want to keep
artifacts:
paths:
- foo/bar/ # Save everything, the *.txt are already removed
Note for the anxious: CI jobs are run in a container (at least it's the default behavious), and the files you work on are a copy of your project. You can delete whatever you want, nothing will be lost outside the container.
I am trying to build a Gitlab pipeline that is made up of 4 jobs. The stages I have are:
stages:
- compare
- build
- test
- deploy
The compare stage is taking a dump from an API on another server, comparing it to the same dump from the last successful pipeline run (it's made available as an artifact) then comparing the two.
If there is any difference I would like the pipeline to move onto the next stage, if there is no difference then I would like it to exit gracefully.
I have it working but rather than exiting gracefully if there are no differences it fails and the pipeline is marked as failed, here is how it looks.
Here is the important code from my .gitlab-ci.yaml (with some identifying information removed )
Get_inventory_dump:
stage: compare
only:
- schedules
script:
- 'curl -k --output "previous-inventory.json" --header "PRIVATE-TOKEN: $user_token" "https://url/to/get/artifact/from/last/successful/run"'
- python3 auto_config_scripts/dump_device_inventory_api_to_json.py -p $pass -o /inventory.json -u https://url/for/inventory/dump -y
- /usr/bin/cmp previous-inventory.json inventory.json && echo "No Change in inventory since last successful run" && exit 1 || echo "Inventory has changed since last run, continue" && exit 0
artifacts:
when: on_success
expire_in: 4 weeks
paths:
- inventory.json
Generate_icinga_config:
stage: build
only:
- schedules
when: on_success
script:
Everything is behaving as I would expect but I feel like there is a better way to do this.
Is there a way, if the comparison is the same to simply skip the next stages of the pipeline but still have the pipeline completed as 'passed' rather than 'failed'?
There are two solutions I can think of. Unfortunately, they either come slightly confusing UI behavior or you have to adapt all jobs.
Job attributes like only or changes are only concerned with the state of or the files of the git repository (see https://docs.gitlab.com/ee/ci/yaml/) and therefore not of use here as the file is only created during CI and not part of the repository.
Solution 1: You can allow_failure: true to the first job. This will mark the pipeline as successful despite the job failing and subsequent jobs will not be executed as the first job did not succeed. The drawback is that when you investigate the pipeline there will be an exclamation mark instead of a green check for this job.
Solution 2: Instead of failing the first job when there are no changes the inventory.json file is removed. And all subsequent jobs directly terminate with exit code 0 when the file doesn't exist. Note that this only works because inventory.json is marked as an artifact.
Based on Fzgregors suggestion, this is how I solved my problem:
If there was a difference and I wanted my second stage to actually do some work I created a file called "continue" and made it available as an artifact.
The second stage will look for that file and use an IF statement to decide if it should do something or just exit nicely
Get_inventory_dump:
stage: compare
only:
- schedules
script:
- 'curl -k --output "previous-inventory.json" --header "PRIVATE-TOKEN: $user_token" "https://url/to/get/artifact/from/last/successful/run"'
- python3 auto_config_scripts/dump_device_inventory_api_to_json.py -p $pass -o /inventory.json -u https://url/for/inventory/dump -y
- /usr/bin/cmp previous-inventory.json inventory.json && echo "No Change in inventory since last successful run" || echo "Inventory has changed since last run, continue" && touch continue
artifacts:
when: on_success
expire_in: 4 weeks
paths:
- inventory.json
- continue
Generate_icinga_config:
stage: build
only:
- schedules
when: on_success
script:
- if [[ -f continue ]]; then
do some stuff;
else
echo "No Change in inventory, nothing to do";
fi
This allowed me to keep my inventory artifact but at the same time let the next stage know if it needed to do some work or just do nothing and exit with code 0
I have a substantially similar construction and I'm looking for essentially the same solution.
When I allow_failure: true my subsequent jobs DO execute.
If I use a stamp file, all of the subsequent jobs also execute taking up queue, runners, etc, even though they aren't needed.
I was hoping for an easier solution but I think I'm going to have to go with generated yaml files. That seems to be the only way to inject dynamic information like decisions into a pipeline.