I have 2 GitLab CI/CD Jobs doing some stuff that generates variables.
For example, the first job saves in a .env file
job1:
script:
- echo "A_VARIABLE=${A_VARIABLE}" >> job.env
artifacts:
reports:
dotenv:
- job.env
And so does the second job.
job2:
script:
- echo "ANOTHER_VARIABLE=${ANOTHER_VARIABLE}" >> job.env
artifacts:
reports:
dotenv:
- job.env
But can it save to that same job.env file without scratching existing data ?
I know I can just group the 2 jobs but I would prefer not mixing them to make it cleaner
I didn't test it, but this should work from my understanding. Saving the file to an artifact will restore it before job2 starts. So it should be possible to append to the file.
Probably would test it with some simpler structure like this
job2:
script:
- echo "ANOTHER_VARIABLE=${ANOTHER_VARIABLE}" >> job.env
artifacts:
paths:
- job.env
Related
I'm following the gitlab docs for multi-project pipelines. I'm running on gitlab.com (not enterprise/self-hosted).
I have successfully set up a multi-project pipeline. My question is - is there a way to pass some but not all variables between stages?
Here's a very simple build script for two projects:
Main project:
variables:
THIS_PROJECT_NAME: trigger-source
SHARED_ARGUMENT: "hello world!"
stages:
- build
- downstream
build-code-job:
stage: build
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
run-trigger-job:
stage: downstream
inherit:
variables: false
variables:
SHARED_ARGUMENT: $SHARED_ARGUMENT
trigger: my-org/triggers_dest
Triggered project:
variables:
THIS_PROJECT_NAME: trigger-dest
SHARED_ARGUMENT: "overwrite me"
stages:
- test
triggered-job:
stage: test
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
only:
- pipelines
when I run this with inherit: variables: false, the output in the triggered project's builds just show the default variables (no variables are passed):
$ echo "${THIS_PROJECT_NAME}"
trigger-dest
$ echo "${SHARED_ARGUMENT}"
overwrite me
However, when I use inherit: variables: true, all variables are passed, except the value of SHARED_ARGUMENT is actually written as the literal "$SHARED_ARGUMENT, which then gets expanded to "overwrite me":
$ echo "${THIS_PROJECT_NAME}"
trigger-source
$ echo "${SHARED_ARGUMENT}"
overwrite me
This is the opposite of what I want! Essentially I want to whitelist variables to pass through, rather than blacklisting them as above. Any way to do this?
Found the answer buried in the docs on the inherit: variables keyword. In addition to true/false, you can specify a list of variables to inherit.
Changing the source project's .gitlab-ci.yml to the following:
variables:
THIS_PROJECT_NAME: trigger-source
SHARED_ARGUMENT: "hello world!"
stages:
- build
- downstream
build-code-job:
stage: build
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
run-trigger-job:
stage: downstream
inherit:
variables:
- SHARED_ARGUMENT
trigger: my-org/triggers_dest
results in the desired output:
$ echo "${THIS_PROJECT_NAME}"
trigger-dest
$ echo "${SHARED_ARGUMENT}"
hello world!
I have the following
stages:
- stage1
- stage2
variables:
MY_ENV_VAR: env_$CI_JOB_ID
stage1_build:
stage: stage1
script:
- echo $MY_ENV_VAR
stage2_build:
stage: stage2
script:
- echo $MY_ENV_VAR
I get different values for $MY_ENV_VAR in the two stages (which means $CI_JOB_ID changes on every stage).
What I want is set $MY_ENV_VAR once with one value of $CI_JOB_ID and make it a constant, so that the same value of $MY_ENV_VAR is used across all stages.
Use $CI_PIPELINE_ID instaed, which will be constant across all jobs in the pipeline.
variables:
MY_ENV_VAR: env_$CI_PIPELINE_ID
See predefined environment variables for additional reference.
If you really want an environment variable to be created in one job and persist for the rest of the pipeline, you can pass variables between jobs using artifacts:reports:dotenv.
stages:
- stage1
- stage2
set_env:
stage: .pre
script:
echo "MY_ENV_VAR=env_$CI_JOB_ID" > .myenv
artifacts:
reports:
dotenv: .myenv
stage1_build:
stage: stage1
script:
- echo $MY_ENV_VAR
stage2_build:
stage: stage2
script:
- echo $MY_ENV_VAR
I have a gitlab-ci like this:
stages:
- calculation
- execution
calculation-job:
stage: calculation
script: ./calculate_something_and_output_results.sh
tags:
- my-runner
execution-job:
stage: execution
script: ./execute_something_with_calculation_results.sh foo
tags:
- my-runner
The foo argument in execution-job is base on the results of calculation-job. I want to pass the results from one job to another job via variables. How can I do that?
If you're looking to get the results without storing a file anywhere you can use artifacts: reports: dotenv. This is taken entirely from DarwinJS shared-variables-across-jobs repo.
stages:
- calculation
- execution
calculation-job:
stage: calculation
script: - |
# stores new or updates existing env variables, ex. $OUTPUT_VAR1
./calculate_something_and_output_results.sh >> deploy.env
tags:
- my-runner
artifacts:
reports:
#propagates variables into the pipeline level, but never stores the actual file
dotenv: deploy.env
execution-job:
stage: execution
script: - |
echo "OUTPUT_VAR1: $OUTPUT_VAR1"
./execute_something_with_calculation_results.sh foo
tags:
- my-runner
AFAIK it is not possible to pass a variable directly from one job to another job. Instead you have to write them into a file and pass that as artifact to the receiving job. To make parsing of the file easy, I recommend to create it with bash export statements and source it in the receiving job's script:
calculation-job:
stage: calculation
script:
- ./calculate_something_and_output_results.sh
- echo "export RESULT1=$calculation_result1" > results
- echo "export RESULT2=$calculation_result2" >> results
tags:
- my-runner
artifacts:
name: "Calculation results"
path: results
execution-job:
stage: execution
script:
- source ./results
# You can access $RESULT1 and $RESULT2 now
- ./execute_something_with_calculation_results.sh $RESULT1 $RESULT2 foo
tags:
- my-runner
needs: calculation-job
Note the ./ when sourcing results might be necessary in case of a POSIX compliant shell that does not source files in the current directory directly like, for example, a bash started as sh.
As a simpler version of what #bjhend answered (no need for export or source statements), since GitLab 13.1 the docs. recommend using a dotenv artifact.
stages:
- calculation
- execution
calculation-job:
stage: calculation
script:
# Output format must feature one "VARIABLE=value" statement per line (see docs.)
- ./calculate_something_and_output_results.sh >> calculation.env
tags:
- my-runner
artifacts:
reports:
dotenv: calculation.env
execution-job:
stage: execution
script:
# Any variables created by above are now in the environment
- ./execute_something_with_calculation_results.sh
tags:
- my-runner
# The following is technically not needed, but serves as good documentation
needs:
job: calculation-job
artifacts: true
If you have a job after the calculation stage that you don't want to use the variables, you can add the following to it:
needs:
job: calculation-job
artifacts: false
I'm setting a variable in a job in the pipeline and use it in the artifacts / dotenv.
It doesn't seem to work to use the variable as a condition in the rules.
How can I set a env variable and use it in another Job's rule?
Thanks a lot for your thoughts!
include:
- local: '/gitlab/cicd/.gitlab-ci_test.yml'
stages:
- build
- test
build_rule:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_MESSAGE =~ /R::/m
script:
- echo "Hello World"
- |
echo "RUNTESTS=1" > gitlabcicd.env
artifacts:
reports:
dotenv: gitlabcicd.env
test_always:
stage: test
script:
- echo "TestEnv" $RUNTESTS
# prints TestEnv 1
test_sometimes:
stage: test
rules:
- if: $RUNTESTS
# $RUNTESTS == "1" doesnt work either
script:
- echo "Runs only if variable was set"
# doesnt run, even if it prints above
As I posted in a comment before, but just so people can find it more easily, the answer is that, unfortunately, right now it is not possible to do that.
There is an issue about it here: https://gitlab.com/gitlab-org/gitlab/-/issues/235812
No satisfactory workaround that I'm aware of.
Is needed to load the .env file on every job
"Set variables":
stage: preparation
script:
- echo version=1.1 > varfile
- echo foo=bar >> varfile
artifacts:
paths:
- varfile
Dothings:
stage: run
script:
- load varfile
- echo ${foo}
Is there any way to use an export variable, defined in the generic before_script:
before_script:
- export UPPERHASH=$(echo $CI_COMMIT_REF_SLUG | md5sum | tr [a-z] [A-Z])
into another job as a variable, because I am gonna use trigger but trigger does not allow to have any script, ex:
test variables:
stage: test-variables
variables:
UPPERHASH_TEST1: $UPPERHASH
trigger:
project: "...\..."
I have tried multiple options but none of them is working.
It will not work this way because "test variables".variables is processed before before_script
You only can refer to this variable in a script:
test variables:
stage: test-variables
script:
UPPERHASH_TEST1=$UPPERHASH
... trigger other project from command line ...
Read here on how to trigger other project from command line
https://docs.gitlab.com/ee/ci/triggers/README.html