I have the following gitlab-ci.yml file that reads the package.json using the jq processor to dynamically set the variable name of the artifact folder, something along the lines of
image: node:latest
stages:
- build
before_script:
## steps ignored for purpose of question
- export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
prod_build:
stage: build
script:
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
- yarn run build
artifacts:
paths:
- dist/$NAME_OF_ARTIFACT_FOLDER ## this does not work
expire_in: 2 hrs
The issue here is - dist/$NAME_OF_ARTIFACT_FOLDER does not work, not sure if am missing something here.
EDIT
Upon hard coding the expected path such as the following, it works fine, which would mean that the folder name is valid and that the artifact is indeed identified appropriately, but does NOT work when coming from $NAME_OF_ARTIFACT_FOLDER
artifacts:
paths:
- dist/myApp_1.0.0.zip ##hardcoding the expected works just fine
expire_in: 2 hrs
Well, that is not possible currently. Manual says as follows:
The artifacts:name variable can make use of any of the predefined variables.
That is no variables set in your script part of the job can be used.
This is an open issue at GitLab
Artifacts Filename Cannot Be Set with Dynamic Variables
I had a project variable defining the path to a zip file in a script which I reused at artifacts:paths level. The linked issue would have been more obvious had the artifacts:paths instance completely failed to get assigned but in my case it inherited a different value from that a mere two lines above in my job!
Related
I've got some trouble defining a job. I'm in an early stage of defining my pipeline and I'm trying to solve my problems with variables :). I've got a few subfolder under in gcp/live/development and I'm trying to cd them and then run terraform stuff:
.plan-development:
variables:
ENVIRONMENT: "development"
TERRAFORM_DEVELOPMENT_PATH: "gcp/live/development"
TF_ROOT: ${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development}
stage: plan-development
script:
- cd ${TF_ROOT}
- gitlab-terraform plan
- gitlab-terraform plan-json
artifacts:
name: plan-${ENVIRONMENT}-${CI_JOB_NAME%-plan-development}
paths:
- ${TF_ROOT}/plan.cache
reports:
terraform: ${TF_ROOT}/plan.json
But according to my failing pipeline, the first script part cd ${TF_ROOT} doesn't enter the expected folder at gcp/live/development/cloud-nat.
I've got some debug outout, which tells me following:
$ cd ${TF_ROOT}
$ pwd && ls
/builds/b79ya_1j/0/devops/terraform/gcp/live/development
cloud-nat
firewall
gke
vpc
$ echo "${CI_JOB_NAME%-plan-development}"
cloud-nat
Am I missing something combing two variables (TF_ROOT: ${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development})?
Hope you can help me out.
Am I missing something combing two variables (TF_ROOT:
${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development})?
The issue is that Gitlab in variables section, doesn't perform bash operations, like you attempt here
${CI_JOB_NAME%-plan-development}
Gitlab literally searches for the variable named CI_JOB_NAME%-plan-development
That's why when it creates the TF_ROOT variable its value is gcp/live/development/ since it could not find the variable named CI_JOB_NAME%-plan-development
I am running the below code section in gitlab-ci.yml file:
script:
- pip install --upgrade pip
- cd ./TestAutomation
- pip install -r ./requirements.txt
Below are the keys and values. So I have to pass any values to the pipeline with key as a variable
ENV : dev
I have added all the above three variables in the GitLab CI CD variables sections by expanding them. just added a single value along with key
I also found like we can add variables in the .yml file itself as below. I am not sure how we can add multiple values for one key
variables:
TEST:
value: "some value" # this would be the default value
description: "This variable makes cakes delicious"
When I run the pipeline I am getting errors as looks like these variables and values are not injected properly.
More details:
And the same error I am getting while running the pipeline. Hence my suspect is like Category variable is not injected properly when I am running through the pipeline
If needed I will show it on the share screen
please find attached an image snippet of my gitlab-ci.yml file- [![enter image description here][1]][1]
I am passing the below parameter while running pipeline -
[![enter image description here][2]][2]
What I have observed is --the values associated with keys which I am passing as parameter or variables , those are not injected or replaced instead of key. So ideally ${Category} should be replaced with value smoke etc
Variables set in the GitLab UI are not passed down to service containers. To set them, assign them to variables in the UI, then re-assign them in your .gitlab-ci.yml:
stages:
- Test
# Added this to your yml file
variables:
ENV: $ENV
BROWSER: $BROWSER
Category: $Category
ui_tests:
stage: Test
image:
name: joyzourky/python-chromedriver:3.8
entrypoint: [""]
tags:
- micro
only:
- develop
when: manual
script:
- pip install --upgrade pip
- cd ./src/Tests/UIAutomation
- pip install -r ./requirements.txt
- pytest -s -v --env=${ENV} --browser=${BROWSER} --alluredir=./reports ./tests -m ${Category}
artifacts:
when: always
path:
- ./src/Tests/UIAutomation/reports/
- ./src/Tests/UIAutomation/logs/
expire_in: 1 day
Please refer attachment it's working with any issue.
When Gitlab CI CD variables are not getting injected into your pipelines as environment variables, please follow the following steps to verify.
Check whether the variable is defined. You need to have at least the Maintainer role setup for your user. Go to Settings --> CI/CD --> Variables. You can see all project variables, and group variables (inherited).
Next, check whether these variables are defined as Protected variables. If they are marked as Protected, then they are only exposed to protected branches or protected tags. I would suggest to uncheck this, if your current branch is not a protected branch. If not you can always make your current branch a protected one.
Next, check whether your code is accessing the environment variables correctly. Based on your scripting language, just access as if you are accessing a regular environment variable.
You don't really need to define these variables in the .gitlab-ci.yaml file. (Even though their documentation says so)
Hope this helps.
As #Keet Sugathadasa mentioned, the branch that triggers the CI must be protected; this was my case so I have to protect it by going to Settings > Repository > Protected branch and then protect the branch from there
is there any chanche to use the TEST_VAR variable in the artifacts path?
inject:
stage: inject
script:
- echo "testDownload.zip" > varName.txt
- export TEST_VAR=$(cat varName.txt)
- echo $TEST_VAR #This is working
- wget http://some.url.com/download/testDownload.zip
artifacts:
name:
paths:
- $TEST_VAR
expire_in: 1h
In production the file testDownload.zip will have a variable name (-.zip) and I'd like to make it available to all later stages with its original name.
Thank you
Andrew is correct (unfortunately) - somewhere I found the following "workaround" (not great, not terrible):
variables:
INJECTION_PATH: "./Test/"
inject:
stage: inject
before_script:
- sudo apt-get -qq install unzip
script:
- wget hhttp://some.url.com/download/7d8e6751-2ca3-477c-9185-7097932c3043.zip
- unzip ./7d8e6751-2ca3-477c-9185-7097932c3043.zip -d $INJECTION_PATH
artifacts:
paths:
- $INJECTION_PATH
expire_in: "600"
This will make the $INJECTION_PATH folder and its contant available in every stage independent of the filenames in the folder.
Unfortunately, it's not possible, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#gitlab-runner-internal-variable-expansion-mechanism :
GitLab Runner internal variable expansion mechanism
Supported: project/group variables, .gitlab-ci.yml variables, config.toml variables, and variables from triggers, pipeline
schedules, and manual pipelines.
Not supported: variables defined inside of
scripts (e.g., export MY_VARIABLE="test").
Exported variables are available only in "*script" section, with some limitations regarding after_script, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#execution-shell-environment
My understanding is that everything outside *script sections is initialized when job is started, and can't be redefined at runtime. In your case you may set a predefined path (i.e. downloaded_artifact.zip), and then wget should save downloaded content into that file. To preserve filename you can echo filename in downloaded_artifact.txt, and pass it to the next stage. It's hacky, but unfortunately you can't pass context between stages in anything other that files (or some shared db, which can be a more hacky solution).
Related ticket: https://gitlab.com/gitlab-org/gitlab/-/issues/16765
Is the list value build:image just a name like build_image? Or does it have special usage in either the yaml file or the .gitlab-ci.yml file? If there isn't a special usage, what is the value of using name1:name2 instead of name1_name2?
The :image doesn't seem to be put into a variable. When I run this through the gitlab pipeline, the output is
Skipping Git submodules setup
Restoring cache
Downloading artifacts
Running before_script and script
$ echo image is $image
image
is
.gitlab-ci.yml
stages:
- build:image
- tag:image
- deploy
build:
stage: build:image
script:
- echo image is $image
I don't see anything like this in the GitLab CI/CD Pipeline Configuration Reference
Where did you see this .gitlab-ci.yml file?
I ran the .gitlab-ci.yml you provided and it seems to work fine, apparently GitLab CI doesn't treat the colon in any special way -- and I wouldn't expect it to, as there is no mention of it in the documentation.
i have the following .gitlab.yaml file
stages:
- build
- test
compile:
stage:build
script:
- stuff_happening
test_1:
stage: test
script:
- do_something_1
artifacts:
when: on_failure
name: "$CI_COMMIT_REF_NAME"
paths:
- /root/dir
When test_1 is executed , it creates a folder dir inside root
I want to add it to artifacts but , i get an error saying:
no matching files. If I add - ls root in the job , I can see the folder.
There is an open question in support forum, but still no response there.
Can anyone help? Thanks
The issue here is that you are trying to upload files that are outside of the project's scope.
From the official documentation on artifacts:
You can only use paths that are within the project workspace.
The reason is obvious; if a git runner were allowed to upload anything outside its workspace, it would cause a serious security issues.
However, if you really wanted to upload anything outside the runner's workspace, you might want to try copying the files outside into the project's root folder, and uploading it from there.
The git runners are usually registered as user:group git-runner:git-runner and so it will likely require sudo privilege if you wanted it to copy files from /root.
According to https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts
You can try adding an additional slash after your path.
paths:
- /root/dir/
Examples from the link above:
Send all files in binaries and .config:
artifacts:
paths:
- binaries/
- .config