I am trying to pass the variable to parallel/matrix and do not see that getting expanded and the job failing. This is being set in the job from the environment variable. I am trying to echo the variable in script and see it shows the right value, but does not get substituted in parallel/matrix. Am I missing anything?
.common_deploy:
script:
- |
echo "showing the regions from environment"
echo $qa_regions
echo "showing the regions from job variable"
echo $REGIONS
parallel:
matrix:
- REGION: "${REGIONS}"
DeployToQA:
variables:
ENVIRONMENT: qa
REGIONS: $qa_regions
extends:
- .common_deploy
stage: deploy
rules:
- if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "master"'
allow_failure: true
Here the variable $qa_regions has the value of "us-west-2,us-east-1", I was expecting to see the 2 jobs for those regions , but i am seeing the job as DeployToQA: [${REGIONS}]
Variable expansion for the parallel keyword is currently not supported. There is an open issue for this request.
You can take a look at the documentation where variables can be used.
Related
I am currently trying to build a pipeline in gitlab ci. It contains a job (here called speed) that should only be run on a specific day (e.g. the 6th of every month). The configuration yaml looks like this:
stages:
- speed
- watcher
variables:
# setup cache in root folder so gitlab cache can pick it up
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
TODAYS_DATE: "$(date +%Y-%m-%d)"
TODAYS_DAY: "$(date +%d)"
TODAYS_MONTH: "$(date +%m)"
include:
- local: "ci/ci_job_speed.yml"
- local: "ci/ci_job_watcher.yml"
speed:
extends: .speed
stage: speed
rules:
- if: '$TODAYS_DAY == "06"'
when: always
watcher:
when: always
extends: .watcher
stage: watcher
rules:
- if: '($CI_COMMIT_AUTHOR !~ /.*Gitlab Runner.*/ && $CI_COMMIT_BRANCH == "test") || $CI_PIPELINE_SOURCE != "push"'
cache:
paths:
- .cache/pip
- .cache/poetry
- .venv
key: "globalcache"
My problem is, that the speed job seems to be ignored, and only the watcher job is started, even though I expect speed to be run (since today is the 6th). The ci linter in GitLab validates my yaml as correct. Does anyboy have an idea what I am doing wrong?
I have tried different combinations as '$TODAYS_DAY == '06'', $TODAYS_DAY == '06' etc, none of them worked yet (the job is simply ignored).
Thanks in advance for any advice :)
Looking at your yaml i would suggest, that TODAYS_DAY: "$(date +%d)" is not executed as intended.
If you assigned this Variable in a bash script, the command date +%d would be executed and the result would be stored in the variable, but i think in your case $(date +%d) just gets stored as literal string.
I could be wrong about this variable, but this is my best guess.
You could consider using GitLabs scheduled Pipelines Feature: https://docs.gitlab.com/ee/ci/pipelines/schedules.html#add-a-pipeline-schedule
#joreign you are correct, the variable TODAYS_DATE was not assigned the day of the current date, but a string value "$(date +%d)". After trying a bit, I could not find a way to enforce the desired behaviour.
Instead, I used a workaround as described in https://gitlab.com/gitlab-org/gitlab/-/issues/20769#note_215109166 and set a random variable in a Pipeline scheduler and gave the ci job a corresponding run condition.
speed:
extends: .speed
stage: speed
only:
refs:
- schedules
variables:
- $REFSPEED_SCHEDULE
I have a CI build stage that runs whenever someone pushes to the repo, and it takes a long time to run. So I want to configure a .gitlab-ci.yml rule that says if the user is only updating the documentation in README.md, it doesn't need to execute the build stage.
Following the tips in How to exclude gitlab-ci.yml changes from triggering a job, it seems like the way to do this would be to add something like:
stages:
- A
Stage A:
stage: A
rules:
- changes:
- "README.md"
when: never
However, based on the documentation in https://docs.gitlab.com/ee/ci/yaml/#ruleschanges, I think this will also suppress the stage if the push contains multiple files, if one of them is README.md.
In that situation, I want the stage to run, not be suppressed. Is there a way to handle this distinction?
I use this syntax to not trigger pipelines when modifying *.md files
workflow:
rules:
- if: '$CI_COMMIT_BRANCH && $CI_COMMIT_BEFORE_SHA !~ /0{40}/'
changes:
- "{*[^.]md*,*.[^m]*,*.m,*.m[^d]*,*.md?*,*[^d]}"
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: '$CI_PIPELINE_SOURCE == "pipeline"'
- if: '$CI_COMMIT_TAG'
Following the idea from #j_b, I was able to solve this (with the caveat that it exits with pipeline having been marked as failed, as described in the code below). Here is the relevant code I added to my .gitlab-ci.yml.
stages:
- Check Only Docs Changed
- mybuild
# Stage that checks whether only documentation changed. If so, then we won't build a new release.
check-only-docs-changed:
stage: Check Only Docs Changed
script:
- BRANCH=origin/$CI_COMMIT_BRANCH
- echo "Checking commit to see if only docs have changed. "
# The line below is from
# https://stackoverflow.com/questions/424071/how-do-i-list-all-the-files-in-a-commit
- GET_CMD="git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA"
- FILES_CHANGED=`$GET_CMD`
- echo "Files in this commit:" $FILES_CHANGED
- NUM_FILES_CHANGED=`$GET_CMD | wc -l`
# We consider any file that ends in .md to be a doc file
# The "|| true" trick on the line below is to deal with the fact that grep will exit with non-zero if it doesn't find any matches.
# See https://stackoverflow.com/questions/42251386/the-return-code-from-grep-is-not-as-expected-on-linux
- NUM_DOC_FILES_CHANGED=`$GET_CMD | grep -c ".*\.md$" || true`
- echo $NUM_FILES_CHANGED "files changed," $NUM_DOC_FILES_CHANGED "of which were documentation."
- |
# We have to test whether NUM_FILES_CHANGED is > 0 because when one branch gets merged into another
# it will be 0, as will NUM_DOC_FILES_CHANGED.
if [[ $NUM_FILES_CHANGED -gt 0 && $NUM_FILES_CHANGED -eq $NUM_DOC_FILES_CHANGED ]]
then
DID_ONLY_DOCS_CHANGE="1"
# Write out the env file before we exit. Otherwise, gitlab will complain that the doccheck.env artifact
# didn't get generated.
echo "DID_ONLY_DOCS_CHANGE=$DID_ONLY_DOCS_CHANGE" >> doccheck.env
echo "Only documentation files have changed. Exiting in order to skip further stages of the pipeline."
# Ideally it would be great to not have to exit with a non-zero code, because this will make the gitlab pipeline
# look like it failed. However, there is currently no easy way to do this, as discussed in
# https://stackoverflow.com/questions/67269109/how-do-i-exit-a-gitlab-pipeline-early-without-failure
# The only way would be to use child pipelines, which is more effort than it's worth for this.
# See https://stackoverflow.com/questions/67169660/dynamically-including-excluding-jobs-in-gitlab-pipeline and
# https://stackoverflow.com/questions/71017961/add-gitlab-ci-job-to-pipeline-based-on-script-command-result
exit 1
else
DID_ONLY_DOCS_CHANGE="0"
echo "DID_ONLY_DOCS_CHANGE=$DID_ONLY_DOCS_CHANGE" >> doccheck.env
fi
# The section below makes the environment variable available to other jobs, but those jobs
# unfortunately cannot access this environment variable in their "rules:" section to control
# whether they execute or not.
artifacts:
reports:
dotenv: doccheck.env
stages:
- test
# Default vars
variables:
DEPLOY_VARIABLE: "dev-deploy"
workflow:
rules:
- if: '$CI_COMMIT_REF_NAME == "master"'
variables:
DEPLOY_VARIABLE: "master-deploy" # Override globally-defined DEPLOY_VARIABLE
my_project_test:
stage: test
script:
- env | grep CI
- echo $DEPLOY_VARIABLE // this always outputs dev-deploy.
Running with gitlab-runner 14.10.1.
No matter if i try that locally or on Gitlab that var is never set.
On local I run it with gitlab-runner exec shell my_project_test.
env | grep CI is:
CI_SERVER_VERSION=
CI_RUNNER_EXECUTABLE_ARCH=darwin/amd64
CI_COMMIT_REF_NAME=master
CI_JOB_TOKEN=
CI_PROJECT_ID=0
CI_RUNNER_REVISION=f761588f
... etc
As per their documentation:
If a rule matches, when: always is the default, and when: never is the default if nothing matches.
I even tried if: '1 == 1' and so on.
gitlab-runner exec has several limitations and does not implement/consider many features of YAML definitions, including workflow:rules:[]variables in this case.
However, when run through gitlab.com or a self-hosted instance of GitLab, workflow:rules: will evaluate properly.
Keep in mind, there are a few cases where variables set elsewhere will take precedence over variables defined in the YAML, such as when variables are set in project, group, or instance settings.
the assignment should works if you put the condition in your task.
my_project_test:
stage: test
rules:
- if: '$CI_COMMIT_REF_NAME == "master"'
variables:
DEPLOY_VARIABLE: "master-deploy" # Override globally-defined DEPLOY_VARIABLE
script:
- env | grep CI
- echo $DEPLOY_VARIABLE // this always outputs dev-deploy.
However the variable is only in the scope of your job, which is under your if condition, it won't overwrite the global value in the another job.
What you really need to pass variable between jobs :
set up global variables dynamically in gitlab-ci
If I add variables(build stage), the stages does not work. Where exactly am I going wrong. If remove the variables under only variables section, it runs.
build:
stage: build
script:
- echo "Build is running"
only:
changes:
- Dockerfile
- requirements.txt
- ./configs/*
variables:
- $BUILD == "True"
development:
stage: development
script:
- echo "development"
except:
variables:
- $BUILD == "False"
development_build:
stage: development_build
script:
- echo "BUILD OK"
rules:
- if: $BUILD == "True"
when: always
The documentation mentions
only:variables / except:variables
Use the only:variables or except:variables keywords to control when to add jobs to a pipeline, based on the status of CI/CD variables.
Keyword type: Job keyword. You can use it only as part of a job.
Possible inputs: An array of CI/CD variable expressions.
First: try and use rules:if to see if the issue persists.
job1:
variables:
VAR1: "variable1"
script:
- echo "Test variable comparison"
rules:
- if: $VAR1 == "variable1"
Second, try with a different variable name, in case BUILD is somehow reserved.
Check also where you have BUILD defined (is there a workflow:rules:variables section?)
I'm setting a variable in a job in the pipeline and use it in the artifacts / dotenv.
It doesn't seem to work to use the variable as a condition in the rules.
How can I set a env variable and use it in another Job's rule?
Thanks a lot for your thoughts!
include:
- local: '/gitlab/cicd/.gitlab-ci_test.yml'
stages:
- build
- test
build_rule:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_MESSAGE =~ /R::/m
script:
- echo "Hello World"
- |
echo "RUNTESTS=1" > gitlabcicd.env
artifacts:
reports:
dotenv: gitlabcicd.env
test_always:
stage: test
script:
- echo "TestEnv" $RUNTESTS
# prints TestEnv 1
test_sometimes:
stage: test
rules:
- if: $RUNTESTS
# $RUNTESTS == "1" doesnt work either
script:
- echo "Runs only if variable was set"
# doesnt run, even if it prints above
As I posted in a comment before, but just so people can find it more easily, the answer is that, unfortunately, right now it is not possible to do that.
There is an issue about it here: https://gitlab.com/gitlab-org/gitlab/-/issues/235812
No satisfactory workaround that I'm aware of.
Is needed to load the .env file on every job
"Set variables":
stage: preparation
script:
- echo version=1.1 > varfile
- echo foo=bar >> varfile
artifacts:
paths:
- varfile
Dothings:
stage: run
script:
- load varfile
- echo ${foo}