is there any chanche to use the TEST_VAR variable in the artifacts path?
inject:
stage: inject
script:
- echo "testDownload.zip" > varName.txt
- export TEST_VAR=$(cat varName.txt)
- echo $TEST_VAR #This is working
- wget http://some.url.com/download/testDownload.zip
artifacts:
name:
paths:
- $TEST_VAR
expire_in: 1h
In production the file testDownload.zip will have a variable name (-.zip) and I'd like to make it available to all later stages with its original name.
Thank you
Andrew is correct (unfortunately) - somewhere I found the following "workaround" (not great, not terrible):
variables:
INJECTION_PATH: "./Test/"
inject:
stage: inject
before_script:
- sudo apt-get -qq install unzip
script:
- wget hhttp://some.url.com/download/7d8e6751-2ca3-477c-9185-7097932c3043.zip
- unzip ./7d8e6751-2ca3-477c-9185-7097932c3043.zip -d $INJECTION_PATH
artifacts:
paths:
- $INJECTION_PATH
expire_in: "600"
This will make the $INJECTION_PATH folder and its contant available in every stage independent of the filenames in the folder.
Unfortunately, it's not possible, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#gitlab-runner-internal-variable-expansion-mechanism :
GitLab Runner internal variable expansion mechanism
Supported: project/group variables, .gitlab-ci.yml variables, config.toml variables, and variables from triggers, pipeline
schedules, and manual pipelines.
Not supported: variables defined inside of
scripts (e.g., export MY_VARIABLE="test").
Exported variables are available only in "*script" section, with some limitations regarding after_script, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#execution-shell-environment
My understanding is that everything outside *script sections is initialized when job is started, and can't be redefined at runtime. In your case you may set a predefined path (i.e. downloaded_artifact.zip), and then wget should save downloaded content into that file. To preserve filename you can echo filename in downloaded_artifact.txt, and pass it to the next stage. It's hacky, but unfortunately you can't pass context between stages in anything other that files (or some shared db, which can be a more hacky solution).
Related ticket: https://gitlab.com/gitlab-org/gitlab/-/issues/16765
Related
I've got some trouble defining a job. I'm in an early stage of defining my pipeline and I'm trying to solve my problems with variables :). I've got a few subfolder under in gcp/live/development and I'm trying to cd them and then run terraform stuff:
.plan-development:
variables:
ENVIRONMENT: "development"
TERRAFORM_DEVELOPMENT_PATH: "gcp/live/development"
TF_ROOT: ${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development}
stage: plan-development
script:
- cd ${TF_ROOT}
- gitlab-terraform plan
- gitlab-terraform plan-json
artifacts:
name: plan-${ENVIRONMENT}-${CI_JOB_NAME%-plan-development}
paths:
- ${TF_ROOT}/plan.cache
reports:
terraform: ${TF_ROOT}/plan.json
But according to my failing pipeline, the first script part cd ${TF_ROOT} doesn't enter the expected folder at gcp/live/development/cloud-nat.
I've got some debug outout, which tells me following:
$ cd ${TF_ROOT}
$ pwd && ls
/builds/b79ya_1j/0/devops/terraform/gcp/live/development
cloud-nat
firewall
gke
vpc
$ echo "${CI_JOB_NAME%-plan-development}"
cloud-nat
Am I missing something combing two variables (TF_ROOT: ${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development})?
Hope you can help me out.
Am I missing something combing two variables (TF_ROOT:
${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development})?
The issue is that Gitlab in variables section, doesn't perform bash operations, like you attempt here
${CI_JOB_NAME%-plan-development}
Gitlab literally searches for the variable named CI_JOB_NAME%-plan-development
That's why when it creates the TF_ROOT variable its value is gcp/live/development/ since it could not find the variable named CI_JOB_NAME%-plan-development
I am running the below code section in gitlab-ci.yml file:
script:
- pip install --upgrade pip
- cd ./TestAutomation
- pip install -r ./requirements.txt
Below are the keys and values. So I have to pass any values to the pipeline with key as a variable
ENV : dev
I have added all the above three variables in the GitLab CI CD variables sections by expanding them. just added a single value along with key
I also found like we can add variables in the .yml file itself as below. I am not sure how we can add multiple values for one key
variables:
TEST:
value: "some value" # this would be the default value
description: "This variable makes cakes delicious"
When I run the pipeline I am getting errors as looks like these variables and values are not injected properly.
More details:
And the same error I am getting while running the pipeline. Hence my suspect is like Category variable is not injected properly when I am running through the pipeline
If needed I will show it on the share screen
please find attached an image snippet of my gitlab-ci.yml file- [![enter image description here][1]][1]
I am passing the below parameter while running pipeline -
[![enter image description here][2]][2]
What I have observed is --the values associated with keys which I am passing as parameter or variables , those are not injected or replaced instead of key. So ideally ${Category} should be replaced with value smoke etc
Variables set in the GitLab UI are not passed down to service containers. To set them, assign them to variables in the UI, then re-assign them in your .gitlab-ci.yml:
stages:
- Test
# Added this to your yml file
variables:
ENV: $ENV
BROWSER: $BROWSER
Category: $Category
ui_tests:
stage: Test
image:
name: joyzourky/python-chromedriver:3.8
entrypoint: [""]
tags:
- micro
only:
- develop
when: manual
script:
- pip install --upgrade pip
- cd ./src/Tests/UIAutomation
- pip install -r ./requirements.txt
- pytest -s -v --env=${ENV} --browser=${BROWSER} --alluredir=./reports ./tests -m ${Category}
artifacts:
when: always
path:
- ./src/Tests/UIAutomation/reports/
- ./src/Tests/UIAutomation/logs/
expire_in: 1 day
Please refer attachment it's working with any issue.
When Gitlab CI CD variables are not getting injected into your pipelines as environment variables, please follow the following steps to verify.
Check whether the variable is defined. You need to have at least the Maintainer role setup for your user. Go to Settings --> CI/CD --> Variables. You can see all project variables, and group variables (inherited).
Next, check whether these variables are defined as Protected variables. If they are marked as Protected, then they are only exposed to protected branches or protected tags. I would suggest to uncheck this, if your current branch is not a protected branch. If not you can always make your current branch a protected one.
Next, check whether your code is accessing the environment variables correctly. Based on your scripting language, just access as if you are accessing a regular environment variable.
You don't really need to define these variables in the .gitlab-ci.yaml file. (Even though their documentation says so)
Hope this helps.
As #Keet Sugathadasa mentioned, the branch that triggers the CI must be protected; this was my case so I have to protect it by going to Settings > Repository > Protected branch and then protect the branch from there
I need to pass a file path to a trigger job where the file path is found within a specified json file in a separate job. Something along the lines of this...
stages:
- run_downstream_pipeline
variables:
- FILE_NAME: default_file.json
.get_path:
stage: run_downstream_pipeline
needs: []
only:
- schedules
- triggers
- web
script:
- apt-get install jq
- FILE_PATH=$(jq '.file_path' $FILE_NAME)
run_pipeline:
extends: .get_path
variables:
PATH: $FILE_PATH
trigger:
project: my/project
branch: staging
strategy: depend
I can't seem to find any workaround to do this, as using extends won't work since Gitlab wont allow for a script section in a trigger job.
I thought about trying to use the Gitlab API trigger method, but I want the status of the downstream pipeline to actually show up in the pipeline UI and I want the upstream pipeline to depend on the status of the downstream pipeline, which from my understanding is not possible when triggering via the API.
Any advice would be appreciated. Thanks!
You can use artifacts:reports:dotenv for setting variables dynamically for subsequent jobs.
stages:
- one
- two
my_job:
stage: "one"
script:
- FILE_PATH=$(jq '.file_path' $FILE_NAME) # In script, get the environment URL.
- echo "FILE_PATH=${FILE_PATH}" >> variables.env # Add the value to a dotenv file.
artifacts:
reports:
dotenv: "variables.env"
example:
stage: two
script: "echo $FILE_PATH"
another_job:
stage: two
trigger:
project: my/project
branch: staging
strategy: depend
Variables in the dotenv file will automatically be present for jobs in subsequent stages (or that declare needs: for the job).
You can also pull artifacts into child pipelines, in general.
But be warned you probably don't want to override the PATH variable, since that's a special variable used to help you find builtin binaries.
I am using below two gitlab repository
Parent Gitlab repo - Application code, for example - Angular application
Child Gitlab repo - For Gitlab Pipeline, has only gitlab-ci.yml file which contain script to run pipeline
I am calling pipeline/child-project gitlab-ci.yml file form parent using below steps
Parent Gitlab repo - gitlab-ci.yml file
include:
- project: 'my-group/child-project'
ref: master
file: '/templates/.gitlab-ci-template.yml'
Child-project - gitlab-ci.yml file
stages:
- test
- build
before_script:
- export PARENT_PROJECT_NAME = ?
- export PARENT_PROJECT_PIPELINE_ID = ?
- export PARENT_PROJECT_BRANCH_NAME = ?
job 1:
stage: test
script:
- echo "Runnig test for project ${PARENT_PROJECT_NAME}"
- node_modules/.bin/ng test
release_job:
stage: build
script: node_modules/.bin/ng build --prod
artifacts:
name: "project-$CI_COMMIT_REF_NAME"
paths:
- dist/
only:
- tags
How can I get the parent-repo details like parent-project name, pipeline-id & branch name in child-project which is running the pipeline?
One way is to define the variables in parent-project and use in child project, but is there any other way where we can directly access the parent-project detail in the child-project?
In your example, include is not the same as trigger. Include just merges all the files together into one giant pipeline so you should be able to access any variables you want from the included files, as long as the variable's scope is correct.
If you are actually looking to pass details to a child pipeline from a parent pipeline you could add a job that exports the variables and details you want to the dotenv, then have the child pipeline access that dotenv. This would allow the code to be dynamic inside of hard coding in the variables and directly passing them to the child pipelines
export-parent-details:
script:
- echo "PARENT_PROJECT_NAME=?" >> build.env
- echo "PARENT_PROJECT_PIPELINE_ID=?" >> build.env
- echo "PARENT_PROJECT_BRANCH_NAME=?" >> build.env
artifacts:
reports:
dotenv: build.env
trigger-child:
stage: docker_hub
trigger:
include:
- project: 'my-group/child-project'
ref: master
file: '/templates/.gitlab-ci-template.yml'
# use this variable in child pipeline to download artifacts from parent pipeline
variables:
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
Then inside the child jobs, you should be able to access the parent artifacts from the parent
child-job:
needs:
- pipeline: $PARENT_PIPELINE_ID
job: export-parent-details
script:
- echo $PARENT_PROJECT_NAME
See
https://docs.gitlab.com/ee/ci/yaml/README.html#artifact-downloads-to-child-pipelines
https://docs.gitlab.com/ee/ci/multi_project_pipelines.html#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance
Another option could be to make an API call to the get the parent project's details since the runners have a read-only token under $CI_JOB_TOKEN, this method is dependent on repo access privileges and the details you want
curl -H "JOB_TOKEN: $CI_JOB_TOKEN" "https://gitlab.com/api/v4/{whatever the api call is}"
Since you’re including the child project’s configuration instead of triggering it, the two pipeline definition files are merged and become one before the pipeline starts, so there’s be no practical difference between this method and having the content of the child project’s definition in the parent’s.
Because of this, all of the predefined variables will be based on the parent project if the pipeline runs there. For example, variables like $CI_COMMIT_REF_NAME, $CI_PROJECT_NAME will point to the parent project and the parent project's branches.
I have the following gitlab-ci.yml file that reads the package.json using the jq processor to dynamically set the variable name of the artifact folder, something along the lines of
image: node:latest
stages:
- build
before_script:
## steps ignored for purpose of question
- export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
prod_build:
stage: build
script:
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
- yarn run build
artifacts:
paths:
- dist/$NAME_OF_ARTIFACT_FOLDER ## this does not work
expire_in: 2 hrs
The issue here is - dist/$NAME_OF_ARTIFACT_FOLDER does not work, not sure if am missing something here.
EDIT
Upon hard coding the expected path such as the following, it works fine, which would mean that the folder name is valid and that the artifact is indeed identified appropriately, but does NOT work when coming from $NAME_OF_ARTIFACT_FOLDER
artifacts:
paths:
- dist/myApp_1.0.0.zip ##hardcoding the expected works just fine
expire_in: 2 hrs
Well, that is not possible currently. Manual says as follows:
The artifacts:name variable can make use of any of the predefined variables.
That is no variables set in your script part of the job can be used.
This is an open issue at GitLab
Artifacts Filename Cannot Be Set with Dynamic Variables
I had a project variable defining the path to a zip file in a script which I reused at artifacts:paths level. The linked issue would have been more obvious had the artifacts:paths instance completely failed to get assigned but in my case it inherited a different value from that a mere two lines above in my job!