I have set up my GitLab pipeline and I'm using GitLab CI variables to generate my configuration file during the build phase. Now we've set up a couple new environments, with each having its own database and other credentials, so I need to generate my configuration file using each environment's variables based on branch. I've already seen:
https://gitlab.com/gitlab-org/gitlab/-/issues/14223
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13379
https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8858
https://medium.com/spacepilots/sourcing-environment-variables-in-gitlab-ci-or-a-poor-mans-dotenv-dfc33ca231df
Some users suggested sourcing environment variables from files but that isn't really a solution because we want to limit access to these variables.
Is there a solution or workaround to this problem in .gitlab-ci.yaml?
Yes. The following example will echo ${TEST_VAR} depending on the branch (dev or main).
.test-job:
stage: test
script:
- echo "${TEST_VAR}"
test-dev-job:
variables:
TEST_VAR: "dev-value"
rules:
- if: $CI_COMMIT_BRANCH == "dev"
extends: .test-job
test-main-job:
variables:
TEST_VAR: "main-value"
rules:
- if: $CI_COMMIT_BRANCH == "main"
extends: .test-job
Related
Context
I am currently trying to implement a homolog/production environment with Gitlab, and have installations of Gitlab Runners on two different servers, both with the same configurations and dependencies, in order to run the build project.
I've written the .gitlab-ci.yml and tried on both runners, using the tag: runner_name and it worked just fine. The problem is that I cannot assign the executor when the pipeline runs.
A example of what I wrote:
job1:
stage: setup
variables:
RUNNER_TAG: "executor-default"
rules:
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME =~ /development/
variables:
RUNNER_TAG: "executor-default"
when: always
allow_failure: false
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME =~ /production/
variables:
RUNNER_TAG: "executor-production"
when: always
allow_failure: false
tags:
- $RUNNER_TAG
script:
- npm install
This job should run on the executor-default if the target is the development branch and on the executor-production if the target branch is the production. On every execution I've tried it ran on the executor-default. I have checked the documentation and forum, but no clue on how to implement or fix this behavior. How to dynamically set a gitlab executor via tags on the job?
You can use "tags" keyword with 2 values (executor-default & executor-production) and used "CI_RUNNER_TAGS" predefined variable. Try below code.
job1:
stage: setup
tags:
- "executor-default"
- "exector-production"
rules:
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME =~ /development/
- CI_RUNNER_TAGS == "executor-default"
when: always
allow_failure: false
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME =~ /production/
- CI_RUNNER_TAGS == "executor-production"
when: always
allow_failure: false
I try to trigger another repo only if in current repo I add TAG to commit. Unfortunately when I want to pass variable as well, this job not starting, if i comment out variable key - its working.
test:
stage: build
trigger:
project: php/project-to-run
branch: develop
rules:
- if: $CI_COMMIT_TAG == 'tst'
variables:
CI_COMMIT_TAG: "$CI_COMMIT_TAG"
Instead of rules i also tried "only" and it works ONLY without passing variable
only:
refs:
- tags
variables:
- $CI_COMMIT_TAG =~ /^stage_[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$/
We have 2 environments (development, production). Per default we deploy to production but if the branch name or the commit message start with dev we deploy to development environment.
I create variables (app_url and app_url_dev) via gitlab GUI : (Project => Settings => CI/CD => Variables)
I use gitlab rule to change the app_url variable of deployement. The script deploy.py (and many other scripts) are using app_url variable.
The gitlab-ci code is:
Tags:
stage: Deploy code
rules:
# - if: $CI_COMMIT_REF_NAME == "main" && $CI_COMMIT_MESSAGE =~ /^dev/
- if: $CI_COMMIT_REF_NAME == "main" && $CI_COMMIT_BRANCH =~ /^dev/
variables:
app_url: $app_url_dev
script:
- Python deploy.py
The problem is that i never get the value of app_url_dev in app_url. I tried other combinaison but i always get the production url and not the dev one:
app_url: ${app_url_dev}
app_url: app_url_dev
But i get always the app_url and not the app_url_dev.
Anyone can help please ?
There is no variable expansion made in rules:variables: section. You can only set a simple text value for a variable.
For an app_url, you may consider using the environment:url: section, which is where variable expansion works. But you can't use it inside of rules:.
Or have some conditional logic in the script: section.
I am trying to include an external GitLab CI YAML file into my project local .gitlab-ci.yml file. The external YAML, which is in my other GitLab project ci/cd > templates contains some workflow rules:
# ci-cd.yml
workflow:
rules:
- if: '$TRACK != null' # TRACK is the environment type (staging/production)
when: always
- if: '$CI_PIPELINE_SOURCE =~ /^trigger|pipeline|web|api$/'
when: always
Below is my project local .gitlab-ci.yml:
include:
- '/.gitlab-ci-test.yml'
- project: 'ci-cd/templates'
file: 'ci-cd.yml'
...
The problem is - none of the jobs I have defined inside locally included .gitlab-ci-test.yml get triggered when I push the changes to GitLab, even when the job has when: always set. Seems like the workflow rules in external ci-cd.yml are not letting the jobs run.
I've also tried to locally add a workflow rule to .gitlab-ci.yml that evaluates to true, because the GitLab workflow keyword docs say
When no rules evaluate to true, the pipeline does not run.
That means if any one of the rules evaluates to true, pipeline should have run, which did not happen when I added a local workflow rule.
EDIT - the external file which has workflow rules is used by many projects so this can't be modified to have "push" in $CI_PIPELINE_SOURCE. My intention is to let it be as a global rule and try to 'override' locally in my project.
I hope I was clear in issue description. Would appreciate any help!
You are missing push event for $CI_PIPELINE_SOURCE in you workflow rule.
workflow:
rules:
- if: '$TRACK != null' # TRACK is the environment type (staging/production)
when: always
- if: '$CI_PIPELINE_SOURCE =~ /^trigger|pipeline|push|web|api$/'
when: always
EDIT: if you are not able to change the workflow rule in the included file, the only option I see is to duplicate the workflow in your gitlab-ci.yml and add the missed push there.
workflow rules take precedence before all other rules and it is not possible to merge two workflow blocks. If a worfklow block is used in an included yml file and the gitlab-ci.yml itself, the workflow from the gitlab-ci.yml is used. You can check this in CI/CD -> Editor -> View merged YAML in gitlab.
include:
- '/.gitlab-ci-test.yml'
- project: 'ci-cd/templates'
file: 'ci-cd.yml'
workflow:
rules:
- if: '$TRACK != null' # TRACK is the environment type (staging/production)
when: always
- if: '$CI_PIPELINE_SOURCE =~ /^trigger|pipeline|push|web|api$/'
when: always
...
As you see below I have two stages.
1- Build
2- Deploy
And I have 3 kind of branches
1- master
2- test
3- Other branches which created by developers. (Development)
I want to run build for all branches but I have a condition for deploy
1- If branch is test run deploy.
2- If branch is other branches don't run deploy.
3- if branch is master run deploy only manual.
But this three condition doesn't works properly with my gitlab-ci.yml
stages:
- build
- deploy
default:
image: node:latest
before_script:
- |
if [ "$CI_BUILD_REF_NAME" == "master" ]; then
ENV="prod"
elif [ "$CI_BUILD_REF_NAME" == "test" ]; then
ENV="test"
else
ENV="dev"
fi
- npm install
build:
stage: build
script:
- cp ./src/env-${ENV}.js ./src/env.js
- npm run build-compressed
deploy:
stage: deploy
rules: //this section doesn't work properly
- if: '$ENV == "test"'
when: always
- if: '$ENV == "prod"'
when: manual
- when: never
script:
- cp ./src/env-${ENV}.js ./src/env.js
- npm run build-compressed
- npm run publish:${ENV}
I recommend doing such a differentiation via Rules. As you can easily override variables with rules, see GitLab CI Doc.
Below you can find a short example
rules
- if: $CI_COMMIT_BRANCH =~ /test/
variables: # Override ENV defined
ENV: "test" # at the job level.
- if: $CI_COMMIT_BRANCH =~ /master/
variables: # Override ENV defined
ENV: "prod" # at the job level.
- variables: # not 100% sure about this one, as when can stand without a if clause, i assume variables can too :D
ENV: "dev"
Furthermore i would also do the rule check based on the branch within the deploy stage. - as it is easier to verify the branch, than to ensure at a later stage, that the Env variable is still set properly.
Also important to notice is the order of evaluation. Just because there is a before_script in the default section, does not mean it is evaluated just once for the whole. The before_script is evaluated once per Job.
Additionally the rules are evaluated, before the JOB runs, as it needs to determine if the job should be executed or not. This means you are not able to reference the variable from the before_script.
If you want to pass on the variable from the first job to the second job, you need to persist it in a dotenv-artifact - this way, the variables are available before the second stage start. (I am still not sure if you can use them for evaluation in rules, that is something you need to check)
I hope my additional clarifications, based on your comments, help you to find the best way.