I've got some trouble defining a job. I'm in an early stage of defining my pipeline and I'm trying to solve my problems with variables :). I've got a few subfolder under in gcp/live/development and I'm trying to cd them and then run terraform stuff:
.plan-development:
variables:
ENVIRONMENT: "development"
TERRAFORM_DEVELOPMENT_PATH: "gcp/live/development"
TF_ROOT: ${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development}
stage: plan-development
script:
- cd ${TF_ROOT}
- gitlab-terraform plan
- gitlab-terraform plan-json
artifacts:
name: plan-${ENVIRONMENT}-${CI_JOB_NAME%-plan-development}
paths:
- ${TF_ROOT}/plan.cache
reports:
terraform: ${TF_ROOT}/plan.json
But according to my failing pipeline, the first script part cd ${TF_ROOT} doesn't enter the expected folder at gcp/live/development/cloud-nat.
I've got some debug outout, which tells me following:
$ cd ${TF_ROOT}
$ pwd && ls
/builds/b79ya_1j/0/devops/terraform/gcp/live/development
cloud-nat
firewall
gke
vpc
$ echo "${CI_JOB_NAME%-plan-development}"
cloud-nat
Am I missing something combing two variables (TF_ROOT: ${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development})?
Hope you can help me out.
Am I missing something combing two variables (TF_ROOT:
${TERRAFORM_DEVELOPMENT_PATH}/${CI_JOB_NAME%-plan-development})?
The issue is that Gitlab in variables section, doesn't perform bash operations, like you attempt here
${CI_JOB_NAME%-plan-development}
Gitlab literally searches for the variable named CI_JOB_NAME%-plan-development
That's why when it creates the TF_ROOT variable its value is gcp/live/development/ since it could not find the variable named CI_JOB_NAME%-plan-development
Related
I'd like to parameterize the environment URL of one of my GitLab CI job in a project which belongs to a subgroup.
If for example I have:
CI_PROJECT_PATH = mygroup/mysubgroup/myproject
CI_PROJECT_NAMESPACE = mygroup/mysubgroup
CI_PROJECT_NAME = myproject
I'd like to have the URL to be something like https://mygroup.gitlab.com/-/mysubgroup/myproject/-/jobs/12345/artifacts/public/index.html.
But I cannot find a way to do this since there is no predefined variable for the "sub-namespace" (here mysubgroup) and there is no variable substitution in environment.url as far as I can see.
I tried in my gitlab-ci.yml things like that:
build:
stage: build
image: bash:latest
script:
- export # print the available variables
environment:
name: test
url: ${CI_SERVER_PROTOCOL}://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_PATH#${CI_PROJECT_ROOT_NAMESPACE}/}/-/jobs/$CI_JOB_ID/artifacts/public/index.html
but the result is https://mygroup.gitlab.com/-/${CI_PROJECT_PATH#myproject/}/-/jobs/12345/artifacts/public/index.html.
References:
https://stackoverflow.com/a/58402821/1064669
https://gitlab.com/gitlab-org/gitlab/-/issues/350902
job:environment:url is directly processed by gitlab, which does not support this kind of parameter expansion.
You'll need to use bash or a similar shell for this to work as intended:
build:
stage: build
image: bash:latest
script:
- echo "DEPLOY_URL=${CI_SERVER_PROTOCOL}://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_PATH#${CI_PROJECT_ROOT_NAMESPACE}/}/-/jobs/$CI_JOB_ID/artifacts/public/index.html" > deploy.env
- cat deploy.env
artifacts:
reports:
dotenv: deploy.env
environment:
name: test
url: $DEPLOY_URL
Based on the example given here: https://docs.gitlab.com/ee/ci/environments/#example-of-setting-dynamic-environment-urls
I think you need to escape the inner curly braces like this: (${CI_PROJECT_ROOT_NAMESPACE} should become $`{CI_PROJECT_ROOT_NAMESPACE`})
So the whole thing becomes:
url: ${CI_SERVER_PROTOCOL}://${CI_PROJECT_ROOT_NAMESPACE}.${CI_PAGES_DOMAIN}/-/${CI_PROJECT_PATH#$`{CI_PROJECT_ROOT_NAMESPACE`}/}/-/jobs/$CI_JOB_ID/artifacts/public/index.html
is there any chanche to use the TEST_VAR variable in the artifacts path?
inject:
stage: inject
script:
- echo "testDownload.zip" > varName.txt
- export TEST_VAR=$(cat varName.txt)
- echo $TEST_VAR #This is working
- wget http://some.url.com/download/testDownload.zip
artifacts:
name:
paths:
- $TEST_VAR
expire_in: 1h
In production the file testDownload.zip will have a variable name (-.zip) and I'd like to make it available to all later stages with its original name.
Thank you
Andrew is correct (unfortunately) - somewhere I found the following "workaround" (not great, not terrible):
variables:
INJECTION_PATH: "./Test/"
inject:
stage: inject
before_script:
- sudo apt-get -qq install unzip
script:
- wget hhttp://some.url.com/download/7d8e6751-2ca3-477c-9185-7097932c3043.zip
- unzip ./7d8e6751-2ca3-477c-9185-7097932c3043.zip -d $INJECTION_PATH
artifacts:
paths:
- $INJECTION_PATH
expire_in: "600"
This will make the $INJECTION_PATH folder and its contant available in every stage independent of the filenames in the folder.
Unfortunately, it's not possible, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#gitlab-runner-internal-variable-expansion-mechanism :
GitLab Runner internal variable expansion mechanism
Supported: project/group variables, .gitlab-ci.yml variables, config.toml variables, and variables from triggers, pipeline
schedules, and manual pipelines.
Not supported: variables defined inside of
scripts (e.g., export MY_VARIABLE="test").
Exported variables are available only in "*script" section, with some limitations regarding after_script, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#execution-shell-environment
My understanding is that everything outside *script sections is initialized when job is started, and can't be redefined at runtime. In your case you may set a predefined path (i.e. downloaded_artifact.zip), and then wget should save downloaded content into that file. To preserve filename you can echo filename in downloaded_artifact.txt, and pass it to the next stage. It's hacky, but unfortunately you can't pass context between stages in anything other that files (or some shared db, which can be a more hacky solution).
Related ticket: https://gitlab.com/gitlab-org/gitlab/-/issues/16765
I am writing a CI/CD pipeline for terraform to deploy GCP resources.
In terraform code, I got many functionality and Network is one of them. The folder structure of the Network is
Network
VPC
LoadBalancer
DNS
VPN
So, I want to loop terraform init, plan and apply commands for all the sub-folder of Network folder. The yml file looks like
image:
name: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
variables:
TF_ROOT: ${CI_PROJECT_DIR}
env: 'prod'
network_services: ""
stages:
- init
init:
stage: init
script:
- |
network_services = ['vpc' 'vpn']
for service in $network_services[#]
do
echo The path is var/$env/terraform.tfvars
done
The above gives me the error:
$ network_services = ['vpc' 'vpn'] # collapsed multi-line command
/bin/sh: eval: line 103: network_services: not found
Please suggest a way to declare array variable in gitlab CI/CD yml.
Try changing your job to something like this:
init:
stage: init
script:
- network_services = ('vpc' 'vpn')
- for service in $network_services[#]
do
echo "The path is var/$env/terraform.tfvars
done
I don't believe you can do multi-line commands like that due to the way the the gitlab-runner eval's the entries in the script array, though combining them with ; has worked for me if I want them on one line.
I have the following gitlab-ci.yml file that reads the package.json using the jq processor to dynamically set the variable name of the artifact folder, something along the lines of
image: node:latest
stages:
- build
before_script:
## steps ignored for purpose of question
- export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
prod_build:
stage: build
script:
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
- yarn run build
artifacts:
paths:
- dist/$NAME_OF_ARTIFACT_FOLDER ## this does not work
expire_in: 2 hrs
The issue here is - dist/$NAME_OF_ARTIFACT_FOLDER does not work, not sure if am missing something here.
EDIT
Upon hard coding the expected path such as the following, it works fine, which would mean that the folder name is valid and that the artifact is indeed identified appropriately, but does NOT work when coming from $NAME_OF_ARTIFACT_FOLDER
artifacts:
paths:
- dist/myApp_1.0.0.zip ##hardcoding the expected works just fine
expire_in: 2 hrs
Well, that is not possible currently. Manual says as follows:
The artifacts:name variable can make use of any of the predefined variables.
That is no variables set in your script part of the job can be used.
This is an open issue at GitLab
Artifacts Filename Cannot Be Set with Dynamic Variables
I had a project variable defining the path to a zip file in a script which I reused at artifacts:paths level. The linked issue would have been more obvious had the artifacts:paths instance completely failed to get assigned but in my case it inherited a different value from that a mere two lines above in my job!
I'd like to add a build signature by the end of a file name in artifacts. I could be job id or a combination of job id and commit reference.
At the moment I get image.slp but I prefer to get something like image.1.slp or image.1.e8f8c4ed.slp . Here is my gitlab-ci.yml:
build-runner:
stage: build
script:
- ./build.sh
- cp ../output/image.slp .
artifacts:
paths:
- image.slp
You should be be able to accomplish that by means of CI_JOB_ID Environment variable.
Refer to docs for a comprehensive list of available variables that you can use.
Probably something like this could solve your problem:
build-runner:
stage: build
script:
- ./build.sh
- cp ../output/image.slp image.$CI_JOB_ID.slp
artifacts:
paths:
- image.$CI_JOB_ID.slp