Gitlab CI CD variable are not getting injected while running gitlab pipeline - gitlab

I am running the below code section in gitlab-ci.yml file:
script:
- pip install --upgrade pip
- cd ./TestAutomation
- pip install -r ./requirements.txt
Below are the keys and values. So I have to pass any values to the pipeline with key as a variable
ENV : dev
I have added all the above three variables in the GitLab CI CD variables sections by expanding them. just added a single value along with key
I also found like we can add variables in the .yml file itself as below. I am not sure how we can add multiple values for one key
variables:
TEST:
value: "some value" # this would be the default value
description: "This variable makes cakes delicious"
When I run the pipeline I am getting errors as looks like these variables and values are not injected properly.
More details:
And the same error I am getting while running the pipeline. Hence my suspect is like Category variable is not injected properly when I am running through the pipeline
If needed I will show it on the share screen
please find attached an image snippet of my gitlab-ci.yml file- [![enter image description here][1]][1]
I am passing the below parameter while running pipeline -
[![enter image description here][2]][2]
What I have observed is --the values associated with keys which I am passing as parameter or variables , those are not injected or replaced instead of key. So ideally ${Category} should be replaced with value smoke etc

Variables set in the GitLab UI are not passed down to service containers. To set them, assign them to variables in the UI, then re-assign them in your .gitlab-ci.yml:
stages:
- Test
# Added this to your yml file
variables:
ENV: $ENV
BROWSER: $BROWSER
Category: $Category
ui_tests:
stage: Test
image:
name: joyzourky/python-chromedriver:3.8
entrypoint: [""]
tags:
- micro
only:
- develop
when: manual
script:
- pip install --upgrade pip
- cd ./src/Tests/UIAutomation
- pip install -r ./requirements.txt
- pytest -s -v --env=${ENV} --browser=${BROWSER} --alluredir=./reports ./tests -m ${Category}
artifacts:
when: always
path:
- ./src/Tests/UIAutomation/reports/
- ./src/Tests/UIAutomation/logs/
expire_in: 1 day
Please refer attachment it's working with any issue.

When Gitlab CI CD variables are not getting injected into your pipelines as environment variables, please follow the following steps to verify.
Check whether the variable is defined. You need to have at least the Maintainer role setup for your user. Go to Settings --> CI/CD --> Variables. You can see all project variables, and group variables (inherited).
Next, check whether these variables are defined as Protected variables. If they are marked as Protected, then they are only exposed to protected branches or protected tags. I would suggest to uncheck this, if your current branch is not a protected branch. If not you can always make your current branch a protected one.
Next, check whether your code is accessing the environment variables correctly. Based on your scripting language, just access as if you are accessing a regular environment variable.
You don't really need to define these variables in the .gitlab-ci.yaml file. (Even though their documentation says so)
Hope this helps.

As #Keet Sugathadasa mentioned, the branch that triggers the CI must be protected; this was my case so I have to protect it by going to Settings > Repository > Protected branch and then protect the branch from there

Related

How do I configure different env variables values when using the Deploy-ECS.gitlab-ci template in Gitlab CI and ECS deployment?

Going over this tutorial:
https://docs.gitlab.com/ee/ci/cloud_deployment/ecs/deploy_to_aws_ecs.html
I couldn't find anywhere an explanation how to configure different env variable values for different branches.
Specifically, I am trying to have different ECS cluster names for main and dev. I tried to create these from the CI/CD menu with different values for the main/dev environments but it doesn't work.
This is my gitlab-ci.yml:
include:
- template: AWS/Deploy-ECS.gitlab-ci.yml
When I try printing the env variables with this yml, the variables are printed as expected:
`stages:
debug
print-all-env-vars-job:
stage: debug
script:
- echo "GitLab CI/CD | Print all environment variables"
- env
`
I tried passing variables through the CI/CD menu on Gitlab, and I also tried passing variables to the gitlab-ci.yml like this:
variables: CI_AWS_ECS_CLUSTER: ${CI_AWS_ECS_CLUSTER} CI_AWS_ECS_TASK_DEFINITION: ${CI_AWS_ECS_TASK_DEFINITION}
Any idea how to solve this?

Use script variable in artifact path

is there any chanche to use the TEST_VAR variable in the artifacts path?
inject:
stage: inject
script:
- echo "testDownload.zip" > varName.txt
- export TEST_VAR=$(cat varName.txt)
- echo $TEST_VAR #This is working
- wget http://some.url.com/download/testDownload.zip
artifacts:
name:
paths:
- $TEST_VAR
expire_in: 1h
In production the file testDownload.zip will have a variable name (-.zip) and I'd like to make it available to all later stages with its original name.
Thank you
Andrew is correct (unfortunately) - somewhere I found the following "workaround" (not great, not terrible):
variables:
INJECTION_PATH: "./Test/"
inject:
stage: inject
before_script:
- sudo apt-get -qq install unzip
script:
- wget hhttp://some.url.com/download/7d8e6751-2ca3-477c-9185-7097932c3043.zip
- unzip ./7d8e6751-2ca3-477c-9185-7097932c3043.zip -d $INJECTION_PATH
artifacts:
paths:
- $INJECTION_PATH
expire_in: "600"
This will make the $INJECTION_PATH folder and its contant available in every stage independent of the filenames in the folder.
Unfortunately, it's not possible, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#gitlab-runner-internal-variable-expansion-mechanism :
GitLab Runner internal variable expansion mechanism
Supported: project/group variables, .gitlab-ci.yml variables, config.toml variables, and variables from triggers, pipeline
schedules, and manual pipelines.
Not supported: variables defined inside of
scripts (e.g., export MY_VARIABLE="test").
Exported variables are available only in "*script" section, with some limitations regarding after_script, see https://docs.gitlab.com/ee/ci/variables/where_variables_can_be_used.html#execution-shell-environment
My understanding is that everything outside *script sections is initialized when job is started, and can't be redefined at runtime. In your case you may set a predefined path (i.e. downloaded_artifact.zip), and then wget should save downloaded content into that file. To preserve filename you can echo filename in downloaded_artifact.txt, and pass it to the next stage. It's hacky, but unfortunately you can't pass context between stages in anything other that files (or some shared db, which can be a more hacky solution).
Related ticket: https://gitlab.com/gitlab-org/gitlab/-/issues/16765

Gitlab - How to add badge based on jobs pipeline

My goal is to show badges (ex : ) based on pipeline results.
I have a private gitlab ce omnibus instance with the following .gitlab-ci.yml :
image: python:3.6
stages:
- lint
- test
before_script:
- python -V
- pip install pipenv
- pipenv install --dev
lint:
stage: lint
script:
- pipenv run pylint --output-format=text --load-plugins pylint_django project/ | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- echo "Pylint score was $score"
- ls
- pwd
- pipenv run anybadge --value=$score --file=pylint.svg pylint
artifacts:
paths:
- pylint.svg
test:
stage: test
script:
- pipenv run python manage.py test
So I thought that I would store the image in the artifacts of the lint job and display it via the badge feature.
But I encounter the following issue : when I browse https://example.com/[group]/[project]/-/jobs/[ID]/artifacts/file/pylint.svg, instead of seeing the badge I have the following message :
The image could not be displayed because it is stored as a job artifact. You can download it instead.
And anyways, I feel like this is the wrong way, because even if I could get the image, there don't seems to be a way to get the image from the last job since gitlab URL for badges images only supports %{project_path}, %{project_id}, %{default_branch}, %{commit_sha}
So how one would add badge to a gitlab project based on a svg generated from results in a gitlab pipeline ?
My guess is that I could push to a .badge folder but that doesn't sound like a clean solution.
You can indeed get the artifact(s) for the latest job (see documentation here), but the trick is that you need to use a slightly different URL:
https://example.com/[group]/[project]/-/jobs/artifacts/[ref]/raw/pylint.svg?job=lint
where [ref] is the reference to your branch/commit/tag.
Speaking of badge placeholders available in Gitlab, you can potentially put %{default_branch} or %{commit_sha} into [ref]. This won't allow you to get the correct badge for every branch, but at least your default branch will get one.
Please also note that ?job=lint query parameter is required, without it the URL won't work.

dynamically setting artifact path/folder structure on gitlab-ci.yml

I have the following gitlab-ci.yml file that reads the package.json using the jq processor to dynamically set the variable name of the artifact folder, something along the lines of
image: node:latest
stages:
- build
before_script:
## steps ignored for purpose of question
- export NAME_OF_ARTIFACT_FOLDER=$(cat package.json | jq -r .name)"_"$(cat package.json | jq -r .version)".zip"
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
prod_build:
stage: build
script:
- echo $NAME_OF_ARTIFACT_FOLDER ##prints the expected name here eg. myApp_1.0.0.zip
- yarn run build
artifacts:
paths:
- dist/$NAME_OF_ARTIFACT_FOLDER ## this does not work
expire_in: 2 hrs
The issue here is - dist/$NAME_OF_ARTIFACT_FOLDER does not work, not sure if am missing something here.
EDIT
Upon hard coding the expected path such as the following, it works fine, which would mean that the folder name is valid and that the artifact is indeed identified appropriately, but does NOT work when coming from $NAME_OF_ARTIFACT_FOLDER
artifacts:
paths:
- dist/myApp_1.0.0.zip ##hardcoding the expected works just fine
expire_in: 2 hrs
Well, that is not possible currently. Manual says as follows:
The artifacts:name variable can make use of any of the predefined variables.
That is no variables set in your script part of the job can be used.
This is an open issue at GitLab
Artifacts Filename Cannot Be Set with Dynamic Variables
I had a project variable defining the path to a zip file in a script which I reused at artifacts:paths level. The linked issue would have been more obvious had the artifacts:paths instance completely failed to get assigned but in my case it inherited a different value from that a mere two lines above in my job!

Execute a script before the branch is deleted in GitLab-CI

GitLab-CI executes the stop-environment script in dynamic environments after the branch has been deleted. This effectively forces you to put all the teardown logic into the .gitlab-ci.yml instead of a script that .gitlab-ci.yml just calls.
Does anyone know a workaround for this? I have a shell script that removes the deployment. This script is part of the repository and can also be called locally (i.e. not onli in an CI environment). I want GitLab-CI to call this script when removing a dynamic environment but it's obviously not there anymore when the branch has been deleted. I also cannot put this script to the artifacts as it is generated before the build by a configure script and contains secrets. It would be great if one could execute the teardown script before the branch is deleted.
Here's a relevant excerpt from the .gitlab-ci.yml
deploy_dynamic_staging:
stage: deploy
variables:
SERVICE_NAME: foo-service-$CI_BUILD_REF_SLUG
script:
- ./configure
- make deploy.staging
environment:
name: staging/$CI_BUILD_REF_SLUG
on_stop: stop_dynamic_staging
except:
- master
stop_dynamic_staging:
stage: deploy
variables:
GIT_STRATEGY: none
script:
- make teardown # <- this fails
when: manual
environment:
name: staging/$CI_BUILD_REF_SLUG
action: stop
Probably not ideal, but you can curl the script using the gitlab API before running it:
curl \
-X GET https://gitlab.example. com/raw/master/script.sh\
-H 'PRIVATE-TOKEN: ${GITLAB_TOKEN}' > script.sh
GitLab-CI executes the stop-environment script in dynamic environments after the branch has been deleted.
That includes:
An on_stop action, if defined, is executed.
With GitLab 15.1 (June 2022), you can skip that on_top action:
Force stop an environment
In 15.1, we added a force option to the stop environment API call.
This allows you to delete an active environment without running the specified on_stop jobs in cases where running these defined actions is not desired.
See Documentation and Issue.

Resources