I am trying to parallelize flutter build using GitLab using GitLab's parallel keyword and flutter's --total-shards and --shard-index.
Something like below
test_job:
stage: test
parallel: 3
script:
- flutter test --total-shards $CI_NODE_TOTAL --shard-index $CI_NODE_INDEX
However, this script fails in the last job because off-by-one error $CI_NODE_INDEX > $CI_NODE_TOTAL. Seems like it is undocumented that $CI_NODE_INDEX starts from 1 instead of 0.
I wanted to subtract the variables by using VARIABLES to $CI_NODE_INDEX_ZERO because the variable is being used multiple times throughout this long job (the script in the example above is shortened).
I tried this.
test_job:
stage: test
parallel: 3
variables:
CI_NODE_INDEX_ZERO: $( expr $CI_NODE_INDEX - 1 )
script:
- flutter test --total-shards $CI_NODE_TOTAL --shard-index $CI_NODE_INDEX_ZERO
The script still fails since the value of $CI_NODE_INDEX_ZERO is literal string expr $CI_NODE_INDEX - 1 instead of 0 (or whatever integer value needed).
This actually works in my local terminal.
petrabarus#Petras-Air % CI_NODE_INDEX_ZERO=5
petrabarus#Petras-Air % CI_NODE_INDEX=5
petrabarus#Petras-Air % echo $CI_NODE_INDEX
5
petrabarus#Petras-Air % CI_NODE_INDEX_ZERO=$( expr $CI_NODE_INDEX - 1 )
petrabarus#Petras-Air % echo $CI_NODE_INDEX_ZERO
4
How do I fix this?
Variables can only be literal values -- they are not evaluated in any way, like what happens in your bash shell.
If you want to use bash to evaluate and set variables for jobs, you can do that using a dotenv artifact.
make_variables:
stage: .pre # run before all jobs
script:
# evaluate the value of a variable
- DYNAMIC_VARIABLE=$(my-script)
# Add the value to a dotenv file.
- echo "DYNAMIC_VARIABLE=$DYNAMIC_VARIABLE" >> myvariables.env
artifacts:
reports: # set the variables for subsequent jobs
dotenv: myvariables.env
my_job:
script:
- echo "$DYNAMIC_VARIABLE"
Though the easier thing to do would be just to evaluate it directly in your script:
script:
- SHARD_INDEX=$( expr $CI_NODE_INDEX - 1 )
- flutter test --total-shards $CI_NODE_TOTAL --shard-index $SHARD_INDEX
i think because VARIABLES can't read from Gitlab CI
please check it https://docs.gitlab.com/ee/ci/variables/
example
test_variable:
stage: test
script:
- echo "$CI_JOB_STAGE"
please tried it
script:
- flutter test --total-shards "$CI_NODE_TOTAL" --shard-index "$CI_NODE_INDEX"
Related
I'm following the gitlab docs for multi-project pipelines. I'm running on gitlab.com (not enterprise/self-hosted).
I have successfully set up a multi-project pipeline. My question is - is there a way to pass some but not all variables between stages?
Here's a very simple build script for two projects:
Main project:
variables:
THIS_PROJECT_NAME: trigger-source
SHARED_ARGUMENT: "hello world!"
stages:
- build
- downstream
build-code-job:
stage: build
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
run-trigger-job:
stage: downstream
inherit:
variables: false
variables:
SHARED_ARGUMENT: $SHARED_ARGUMENT
trigger: my-org/triggers_dest
Triggered project:
variables:
THIS_PROJECT_NAME: trigger-dest
SHARED_ARGUMENT: "overwrite me"
stages:
- test
triggered-job:
stage: test
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
only:
- pipelines
when I run this with inherit: variables: false, the output in the triggered project's builds just show the default variables (no variables are passed):
$ echo "${THIS_PROJECT_NAME}"
trigger-dest
$ echo "${SHARED_ARGUMENT}"
overwrite me
However, when I use inherit: variables: true, all variables are passed, except the value of SHARED_ARGUMENT is actually written as the literal "$SHARED_ARGUMENT, which then gets expanded to "overwrite me":
$ echo "${THIS_PROJECT_NAME}"
trigger-source
$ echo "${SHARED_ARGUMENT}"
overwrite me
This is the opposite of what I want! Essentially I want to whitelist variables to pass through, rather than blacklisting them as above. Any way to do this?
Found the answer buried in the docs on the inherit: variables keyword. In addition to true/false, you can specify a list of variables to inherit.
Changing the source project's .gitlab-ci.yml to the following:
variables:
THIS_PROJECT_NAME: trigger-source
SHARED_ARGUMENT: "hello world!"
stages:
- build
- downstream
build-code-job:
stage: build
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
run-trigger-job:
stage: downstream
inherit:
variables:
- SHARED_ARGUMENT
trigger: my-org/triggers_dest
results in the desired output:
$ echo "${THIS_PROJECT_NAME}"
trigger-dest
$ echo "${SHARED_ARGUMENT}"
hello world!
Here's what I am trying to do.
in .gitlab-ci:
Check schedules pass:
stage: check
image: ${myimage}
script:
- MY_CI_VAR=aVeryLongVariable
- echo "$MY_SCHEDULE_VAR"
In schedules:
Which is not working.
The reason I want to do this is for picking different variable (out of many in the job) on each schedule.
Yes, it is possible to use variables within other variables. This feature was released in GitLab 14.3.
However, since you are using GitLab 13.x, this feature won't be available to you.
You may be able to get around this limitation by using a static value and altering your job script accordingly.
myjob:
before_script: |
if [[ "$SCHEDULE_VAR" == "abc" ]]; then
export FOO="$MY_CI_VAR"
fi
# ...
In versions of GitLab < 14.3 you can still make use of other variables within variables, but instead by using $$ to preserve variables from evaluation.
Example from the docs:
variables:
FLAGS: '-al'
LS_CMD: 'ls "$FLAGS" $$TMP_DIR'
script:
- 'eval "$LS_CMD"' # Executes 'ls -al $TMP_DIR'
I have a gitlab-ci like this:
stages:
- calculation
- execution
calculation-job:
stage: calculation
script: ./calculate_something_and_output_results.sh
tags:
- my-runner
execution-job:
stage: execution
script: ./execute_something_with_calculation_results.sh foo
tags:
- my-runner
The foo argument in execution-job is base on the results of calculation-job. I want to pass the results from one job to another job via variables. How can I do that?
If you're looking to get the results without storing a file anywhere you can use artifacts: reports: dotenv. This is taken entirely from DarwinJS shared-variables-across-jobs repo.
stages:
- calculation
- execution
calculation-job:
stage: calculation
script: - |
# stores new or updates existing env variables, ex. $OUTPUT_VAR1
./calculate_something_and_output_results.sh >> deploy.env
tags:
- my-runner
artifacts:
reports:
#propagates variables into the pipeline level, but never stores the actual file
dotenv: deploy.env
execution-job:
stage: execution
script: - |
echo "OUTPUT_VAR1: $OUTPUT_VAR1"
./execute_something_with_calculation_results.sh foo
tags:
- my-runner
AFAIK it is not possible to pass a variable directly from one job to another job. Instead you have to write them into a file and pass that as artifact to the receiving job. To make parsing of the file easy, I recommend to create it with bash export statements and source it in the receiving job's script:
calculation-job:
stage: calculation
script:
- ./calculate_something_and_output_results.sh
- echo "export RESULT1=$calculation_result1" > results
- echo "export RESULT2=$calculation_result2" >> results
tags:
- my-runner
artifacts:
name: "Calculation results"
path: results
execution-job:
stage: execution
script:
- source ./results
# You can access $RESULT1 and $RESULT2 now
- ./execute_something_with_calculation_results.sh $RESULT1 $RESULT2 foo
tags:
- my-runner
needs: calculation-job
Note the ./ when sourcing results might be necessary in case of a POSIX compliant shell that does not source files in the current directory directly like, for example, a bash started as sh.
As a simpler version of what #bjhend answered (no need for export or source statements), since GitLab 13.1 the docs. recommend using a dotenv artifact.
stages:
- calculation
- execution
calculation-job:
stage: calculation
script:
# Output format must feature one "VARIABLE=value" statement per line (see docs.)
- ./calculate_something_and_output_results.sh >> calculation.env
tags:
- my-runner
artifacts:
reports:
dotenv: calculation.env
execution-job:
stage: execution
script:
# Any variables created by above are now in the environment
- ./execute_something_with_calculation_results.sh
tags:
- my-runner
# The following is technically not needed, but serves as good documentation
needs:
job: calculation-job
artifacts: true
If you have a job after the calculation stage that you don't want to use the variables, you can add the following to it:
needs:
job: calculation-job
artifacts: false
I want to use arrays in variables of my gitlab ci/cd yml file, something like that:
variables:
myarrray: ['abc', 'dcef' ]
....
script: |
echo myarray[0] myarray[1]
But Lint tells me that file is incorrect:
variables config should be a hash of key value pairs, value can be a hash
I've tried the next:
variables:
arr[0]: 'abc'
arr[1]: 'cde'
....
script: |
echo $arr[0] $arr[1]
But build failed and prints out bash error:
bash: line 128: export: `arr[0]': not a valid identifier
Is there any way to use array variable in .gitlab-ci.yml file?
According to the docs, this is what you should be doing:
It is not possible to create a CI/CD variable that is an array of values, but you can use shell scripting techniques for similar behavior.
For example, you can store multiple variables separated by a space in a variable, then loop through the values with a script:
job1:
variables:
FOLDERS: src test docs
script:
- |
for FOLDER in $FOLDERS
do
echo "The path is root/${FOLDER}"
done
After some investigations I found some surrogate solution. Perhaps It may be useful for somebody:
variables:
# Name of using set
targetType: 'test'
# Variables set test
X_test: 'TestValue'
# Variables set dev
X_dev: 'DevValue'
# Name of variable from effective set
X_curName: 'X_$targetType'
.....
script: |
echo Variable X_ is ${!X_curName} # prints TestValue
Another approach you could follow is two use a matrix of jobs that will create a job per array entry.
deploystacks:
stage: deploy
parallel:
matrix:
- PROVIDER: aws
STACK: [monitoring, app1]
- PROVIDER: gcp
STACK: [data]
tags:
- ${PROVIDER}-${STACK}
Here is the Gitlab docs regarding matrix
https://docs.gitlab.com/ee/ci/jobs/job_control.html#run-a-one-dimensional-matrix-of-parallel-jobs
Is there any way to use an export variable, defined in the generic before_script:
before_script:
- export UPPERHASH=$(echo $CI_COMMIT_REF_SLUG | md5sum | tr [a-z] [A-Z])
into another job as a variable, because I am gonna use trigger but trigger does not allow to have any script, ex:
test variables:
stage: test-variables
variables:
UPPERHASH_TEST1: $UPPERHASH
trigger:
project: "...\..."
I have tried multiple options but none of them is working.
It will not work this way because "test variables".variables is processed before before_script
You only can refer to this variable in a script:
test variables:
stage: test-variables
script:
UPPERHASH_TEST1=$UPPERHASH
... trigger other project from command line ...
Read here on how to trigger other project from command line
https://docs.gitlab.com/ee/ci/triggers/README.html