Gitlab-CI run Stage conditionally - gitlab

As you see below I have two stages.
1- Build
2- Deploy
And I have 3 kind of branches
1- master
2- test
3- Other branches which created by developers. (Development)
I want to run build for all branches but I have a condition for deploy
1- If branch is test run deploy.
2- If branch is other branches don't run deploy.
3- if branch is master run deploy only manual.
But this three condition doesn't works properly with my gitlab-ci.yml
stages:
- build
- deploy
default:
image: node:latest
before_script:
- |
if [ "$CI_BUILD_REF_NAME" == "master" ]; then
ENV="prod"
elif [ "$CI_BUILD_REF_NAME" == "test" ]; then
ENV="test"
else
ENV="dev"
fi
- npm install
build:
stage: build
script:
- cp ./src/env-${ENV}.js ./src/env.js
- npm run build-compressed
deploy:
stage: deploy
rules: //this section doesn't work properly
- if: '$ENV == "test"'
when: always
- if: '$ENV == "prod"'
when: manual
- when: never
script:
- cp ./src/env-${ENV}.js ./src/env.js
- npm run build-compressed
- npm run publish:${ENV}

I recommend doing such a differentiation via Rules. As you can easily override variables with rules, see GitLab CI Doc.
Below you can find a short example
rules
- if: $CI_COMMIT_BRANCH =~ /test/
variables: # Override ENV defined
ENV: "test" # at the job level.
- if: $CI_COMMIT_BRANCH =~ /master/
variables: # Override ENV defined
ENV: "prod" # at the job level.
- variables: # not 100% sure about this one, as when can stand without a if clause, i assume variables can too :D
ENV: "dev"
Furthermore i would also do the rule check based on the branch within the deploy stage. - as it is easier to verify the branch, than to ensure at a later stage, that the Env variable is still set properly.
Also important to notice is the order of evaluation. Just because there is a before_script in the default section, does not mean it is evaluated just once for the whole. The before_script is evaluated once per Job.
Additionally the rules are evaluated, before the JOB runs, as it needs to determine if the job should be executed or not. This means you are not able to reference the variable from the before_script.
If you want to pass on the variable from the first job to the second job, you need to persist it in a dotenv-artifact - this way, the variables are available before the second stage start. (I am still not sure if you can use them for evaluation in rules, that is something you need to check)
I hope my additional clarifications, based on your comments, help you to find the best way.

Related

CI_PROJECT_DIR not respected in gitlab CI rules

Consider the following pipeline:
image: alpine
stages:
- test
this-does-not-run:
stage: test
rules:
- if: $CI_PROJECT_DIR == "/builds/test"
script:
- echo "It works!"
this-runs:
stage: test
rules:
- if: $CI_PROJECT_DIR != "/builds/test"
script:
- echo "It doesn't work! CI_PROJECT_DIR- $CI_PROJECT_DIR"
And assume my repo is called test. The output of this CI will always be:
It doesn't work! CI_PROJECT_DIR- /builds/test
Obviously $CI_PROJECT_DIR == /builds/test, so I would expect the output to be It works!. Am I missing something, or is $CI_PROJECT_DIR not respected in rules?
The problem here is that variables in rules: must be expanded (evaluated by) the GitLab server at pipeline creation time, but $CI_PROJECT_DIR is determined by the runner's builds_dir setting, which is only known by the runner at runtime when the job actually runs:
References to unavailable variables are left intact. In this case, the runner attempts to expand the variable value at runtime. For example, a variable like CI_BUILDS_DIR is known by the runner only at runtime.
That is to say, like CI_BUILDS_DIR, you can't use CI_PROJECT_DIR or other runtime-only variables in rules:if:. Instead, you should use a variable that is known to the gitlab server at pipeline creation time, such as CI_PROJECT_NAME CI_PROJECT_ID, CI_PROJECT_URL, CI_PROJECT_PATH_SLUG or similar.
Try to wrap the expression in quotes. Afaik this is an expression and you would need to wrap it.
Eg:
rules:
- if: '$CI_PROJECT_DIR == "/builds/test"'
If you only have variables, then from experience you don't need to wrap it.
Eg:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Unexpected behaviour of "rules" in GitLab CI

I have some problems with understanding how and why "rules" in GitLab CI work.
I have written a minimal code showing the issue in a GitLab project: https://gitlab.com/daniel.grabowski/gitlab-ci-rules-problems It contains two directories ("files1/" and "files2/") with some files in them and a .gitlab-ci.yml file.
My configuration
Here's the CI configuration:
stages:
- build
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
.job_tpl:
image: alpine:latest
stage: build
variables:
TARGET_BRANCH: $CI_DEFAULT_BRANCH
rules:
- if: $CI_COMMIT_BRANCH == $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
variables:
JOB_ENV: "prod"
- if: $CI_COMMIT_BRANCH != $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
when: manual
allow_failure: true
variables:
JOB_ENV: "dev"
script:
- echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
- echo "TARGET_BRANCH=$TARGET_BRANCH"
- echo "JOB_ENV=$JOB_ENV"
files1 job:
extends: .job_tpl
variables:
FILES_DIR: files1
files2 job:
extends: .job_tpl
variables:
FILES_DIR: files2
As you can see in the above code I'm using workflow to run only "branch pipelines" and have two "twin" jobs configured to watch for changes in one of the project's directories each. The TARGET_BRANCH variable is of course unnecessary in the demo project but i need something like this in the real one and it shows one of my problems. Additionally the jobs behave differently depending on the branch for which they are run.
My expectations
What I want to achieve is:
Each of the jobs should be added to a pipeline only when I push changes to files1/ or files2/ directory respectively.
When I push changes to a branch different then "main" a manual job responsible for the changed directory shoud be added to a pipeline.
When I merge changes to the "main" branch a job responsible for the changed directory shoud be added to a pipeline and it should be automatically started.
Test scenario
I'm creating a new branch from "main", make some change in the file1/test.txt and push the branch to GitLab.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: a pipeline with both jobs (both manual). Actually I've found explanation of such behaviour here: https://docs.gitlab.com/ee/ci/jobs/job_control.html#jobs-or-pipelines-run-unexpectedly-when-using-changes - "The changes rule always evaluates to true when pushing a new branch or a new tag to GitLab."
On the same branch I make another change in the file1/test.txt and make push.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: exactly what I expect since the branch isn't a "new" one
I create a Merge Request from my branch to main and make the merge.
what I expect: a pipeline created with only "files1 job" which starts automatically
what I get: a pipeline created with only "files1 job" but a manual one
My questions/problems
Can you suggest me any way to bypass the issue with "changes" evaluating always to "true" on new branches? Actually it behaves exactly as I want it if I don't use "rules" but let's assume I need "rules".
Why the jobs run as "manual" on the main branch in spite of the "if" condition in which both CI_COMMIT_BRANCH and TARGET_BRANCH variables are (or should be) set to "main". To debug it I'm printing those vars in job's "script" and when I run it on "main" pipeline I'm getting:
$ echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
CI_COMMIT_BRANCH=main
$ echo "TARGET_BRANCH=$TARGET_BRANCH"
TARGET_BRANCH=main
$ echo "JOB_ENV=$JOB_ENV"
JOB_ENV=dev
so theoretically CI should enter into the "automatic" job path.
Generally I find the CI "rules" quite inconvenient and confusing but as I understand it GitLab prefers them to "only/except" solution so I'm trying to refactor my CI/CD to use them which will fail if I don't find solution for the above difficulties :(

How to run Job when Pipeline was triggered manually

I setup jobs to run only when pushing/merging to branch "dev", but I also want it so I'm able to run them if I trigger that pipeline manually. Something like this:
test:
stage: test
<this step should be run always>
build:
stage: build
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
- if: <also run if the pipeline was run manually, but skip if it was triggered by something else>
This job is defined in a child "trigger" pipeline. This is how the parent looks like:
include:
- template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml'
stages:
- triggers
microservice_a:
stage: triggers
trigger:
include: microservice_a/.gitlab-ci.microservice_a.yml
strategy: depend
rules:
- changes:
- microservice_a/*
The effect I want to achieve is:
Run test in all cases
Run build in the child pipeline only when pushing/merging to "dev"
Also run the build job when the pipeline is run maually
Do not run the build job on any other cases (like a MR)
The rules examples showcase:
job:
script: echo "Hello, Rules!"
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
when: manual
allow_failure: true
- if: '$CI_PIPELINE_SOURCE == "schedule"'
The when:manual should be enough in your case: it does require that a job doesn’t run unless a user starts it.
Bonus question: This job is defined in a child "trigger" pipeline
Then it is related to gitlab-org/gitlab issue 201938, which is supposed to be fixed with GitLab 13.5 (Oct. 2020), but that only allow manual actions for parent-child pipeline (illustrated by this thread)
Double-check the environment variables as set in your child job
echo $CI_JOB_MANUAL
If true, that would indicate a job part of a manual triggered job.
While issue 22448 ("$CI_JOB_MANUAL should be set in all dependent jobs") points to this option not working, it includes a workaround.

Stuggling to Create GitLab-CI Pipeline that includes 'schedules' and 'changes' declarations

I have a GitLab-CI pipeline in place with my Katalon Studio automation tests that I would like to have the following functionality:
Various nightly schedules that run based on scheduled variables being present.
Changes declaration so pushes to the repo only trigger a pipeline run if certain files have been touched.
I have the scheduled portion running as expected, but I am struggling on pairing that with the 'changes' declaration to only run the pipeline IF someone pushes after changing certain files. Can someone help? I am guessing this is an issue with my YAML formatting.
Here is an example snippet from my current GitLab-CI.yaml
Example Tests:
stage: Example
tags:
- aws-medium-runner
script:
- MY_SCRIPT
rules:
- if: $SCHEDULE_A == "true" # tied to schedule A in scheduler tool
when: always
- if: '$CI_PIPELINE_SOURCE == "push"'
changes: # Only run on pushes if changes have been made to certain directories
- Test\ Cases/Example/*
- Object\ Repository/Example/*
- Test\ Suites/Example/*
- Scripts/Example/*
when: always
dependencies:
- Set Release Version
You are missing 2 spaces on the changes: part.
It should work when you add those. (see the example below)
And in case you want to combine the 2 rules with the same conditions, you can combine them into one rule:
Example Tests:
stage: Example
tags:
- aws-medium-runner
script:
- MY_SCRIPT
rules:
- if: '$CI_PIPELINE_SOURCE == "push" || $SCHEDULE_A == "true"'
changes: # Only run on pushes if changes have been made to certain directories
- Test Cases/Example/*
- Object Repository/Example/*
- Test Suites/Example/*
- Scripts/Example/*
when: always
dependencies:
- Set Release Version

Use variables per branch in gitlab

I have set up my GitLab pipeline and I'm using GitLab CI variables to generate my configuration file during the build phase. Now we've set up a couple new environments, with each having its own database and other credentials, so I need to generate my configuration file using each environment's variables based on branch. I've already seen:
https://gitlab.com/gitlab-org/gitlab/-/issues/14223
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13379
https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8858
https://medium.com/spacepilots/sourcing-environment-variables-in-gitlab-ci-or-a-poor-mans-dotenv-dfc33ca231df
Some users suggested sourcing environment variables from files but that isn't really a solution because we want to limit access to these variables.
Is there a solution or workaround to this problem in .gitlab-ci.yaml?
Yes. The following example will echo ${TEST_VAR} depending on the branch (dev or main).
.test-job:
stage: test
script:
- echo "${TEST_VAR}"
test-dev-job:
variables:
TEST_VAR: "dev-value"
rules:
- if: $CI_COMMIT_BRANCH == "dev"
extends: .test-job
test-main-job:
variables:
TEST_VAR: "main-value"
rules:
- if: $CI_COMMIT_BRANCH == "main"
extends: .test-job

Resources