I have a Pipeline that has a few stages: detect, test, build, deploy
The detect stage detects the type of application and the test and build stages have jobs that are included or excluded based on what is computed in detect. The detect stage writes it's value to a environment variable called BUILD_MODE.
I am using rules like so:
ng-build:
extends:
- '.ng/job/build'
stage: build
rules:
- if: $BUILD_MODE == "ANGULAR"
when: always
npm-build:
extends:
- '.npm/job/build'
stage: build
rules:
- if: $BUILD_MODE == "NPM"
when: always
The problem with this is that the BUILD_MODE variable is evaluated statically when the pipeline is created not after the detect stage runs so the above never works unless I set the variable explicitly in the top level YML file like so:
variables:
BUILD_MODE: "ANGULAR"
What is the best way to solve this problem? The summary of what I want to do is evaluate some condition, either set the stages dynamically or set the variable itself before the stages in the Pipleline are created so they will be created with the rules evaluated correctly.
You could take a look at dynamic child-pipelines. Maybe you could solve your problem by dynamically creating your npm/ng build jobs.
Related
I have a yaml pipeline that builds and deploys a project. it deploys to dev & tst. But i dont always want to deploy to tst. at the moment i have solved this with approvals. when the dev stage finishes the pipeline waits for approval.
this introduces some problems:
all developers get an email every time, which can be annoying.
the pipeline waits for deployment to tst, even when deploymen to tst isnt needed. it keeps
waiting
new ci builds dont start when the pipeline is waiting
If there is a well defined way how to determine whether to run the tst stage or not, you can either specify a custom condition on the stage or conditionally include the stage using an expression
Specify a condition
From the docs:
You can specify the conditions under which each stage, job, or step
runs. By default, a job or stage runs if it does not depend on any
other job or stage, or if all of the jobs or stages that it depends on
have completed and succeeded.
There are a number of ways you can customize this behavior, either by built in or custom defined conditions, for example, to run tun the tst stage only if the branch is main:
stages:
- stage: dev
jobs:
- job: dev1
steps:
- script: echo Hello from dev!
- stage: tst
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
jobs:
- job: tst1
steps:
- script: echo Hello From tst!
The tst stage will still be visible when you open the pipeline run, it will be marked as skipped
The condition under the tst stage can be altered to fit your need (it can include both variables, parameters and even output from previous stages.
Conditional expression
From the docs:
You can use if, elseif, and else clauses to conditionally assign
variable values or set inputs for tasks. You can also conditionally
run a step when a condition is met.
Conditional insertion only works with template syntax, which is evaluated when the pipeline yaml is compiled (i.e before the pipeline starts). Thus it cannot evaluate output of previous steps. In the below example the tst stage is completely removed from the pipeline run when the pipeline is instantiated (and the template compiled) if the branch is not main
stages:
- stage: dev
jobs:
- job: dev1
steps:
- script: echo Hello from dev!
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- stage: tst
jobs:
- job: tst1
steps:
- script: echo Hello From tst!
How can I run a gitlab stage periodically? I am aware of pipeline schedules documented here: https://docs.gitlab.com/ee/ci/pipelines/schedules.html. I am only interested in running a particular test stage of a branch. The issue is the branch I want to target has a lot of other stages involved with it. One option might be introducing a variable in the script portion that says if true then execute the script. However, this can be cumbersome and require a lot of changes to all the stages of the ci file for that particular branch under consideration.
You can control when your jobs run using the only/except keywords, or the more advanced rules keyword along with the pipeline source variable.
Example with only:
scheduled_test:
stage: tests
image: my_image:latest
only:
- schedules
script:
- ./run_some_things.sh
The only keyword lets you define some conditions that mean "this job only runs if these conditions are true", and offers a shorthand to check the source of the pipeline. schedules means that the pipeline was started from a schedule rather than a push, trigger, etc. The except keyword is just the opposite. If it had except: schedules the job would always run except if it was scheduled. You can see the full documentation for the only/except keywords here: https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic
As of Gitlab version 12.3, the rules keyword extends the possibilities of the only/except options. We could get the same result from the example above like this:
scheduled_test:
stage: tests
image: my_image:latest
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
when: always
- if: '$CI_PIPELINE_SOURCE == "push"'
when: never
In this example, we check the predefined variable $CI_PIPELINE_SOURCE to see what started this pipeline. If it's "schedule", we always run this job. As an example, if the source is "push" (so the pipeline was started by a git push command), this job will never run. With the rules keyword, all if statements are OR'ed together, so the example above reads, if the source is schedule, always run OR if the source is a push, never run. However, you can AND multiple conditionals together in the same if:
rules:
- if: '$CI_PIPELINE_SOURCE == "push" && $MY_CUSTOM_VARIABLE == true'
when: manual
You can read the full documentation for the rules keyword here: https://docs.gitlab.com/ee/ci/yaml/#rules
While there are many questions and answers around conditional job execution of GitLab CI pipelines I could not find a solution for my problem: To trigger a particular job from pipeline when there are changes in some files OR a env variable is set to a specific value. Something like this:
job_build:
tags:
- executor-shell
stage:
build
script:
- ./gitlab_ci_helper.sh build
only:
changes:
- /path/to/files/with/changes/*
variables:
- $BUILD_IS_A_MUST == "TRUE"
As per the documentation from GitLab itself, we have the following:
Now the question I have is how to make the condition NOT(any of variables) AND (any of changes)) which is same as (any of variables) OR (any of changes) but applied for the only parameter?
Well, in the end, looks like found myself the answer to the question. It turns out that starting from GitLab version 12.3 a new configuration parameter was introduced called rules:
Looks like this parameter is exactly addressing the problem I was looking for an answer. While I don't have yet GitLab 12.3 version to test it, the above ci job would be changed into something like this:
job_build:
tags:
- executor-shell
stage:
build
script:
- ./gitlab_ci_helper.sh build
rules:
- changes:
- /path/to/files/with/changes/*
when: on_success
- if: $BUILD_IS_A_MUST == "TRUE"
when: on_success
In this case, the default rule will get resolved to when: never which is what I'm after thus no need to specify it.
We have a job (deploy to production) that we generally manually click after checking that build on staging. However, very occasionally we have an issue that we've accidentally deployed and want to get a fix out ASAP. In those case we run the tests locally (much faster) and put [urgent-fix] in our commit message to stop the tests running in CI (skipping straight to Docker image build and staging deploy).
What we'd like to do is if we put [urgent-fix] it automatically triggers the production deploy (usually a when: manual step). Can we achieve this somehow?
Sounds like you can use a combination of the only:variables syntax and $CI_COMMIT_MESSAGE predefined variable.
A rough idea (untested):
.deploy_production: &deploy_production
stage: deploy production
script:
- echo "I'm deploy production here"
tags:
- some special tag
deploy::manual:
<< *deploy_production
when: manual
allow_failure: false
deploy:urgent_fix:
<< *deploy_production
only:
variables:
- $CI_COMMIT_MESSAGE =~/[urgent-fix]/
As of GitLab v12.3 (~September 2019) GitLab comes with "Flexible Rules for CI Build config". The feature is intended to replace the only/except functionality and is fully documented here.
With rules: you can now fully influence the when: behaviour of your job based on various conditions (in contrast to only/except: which forced you to create separate jobs for situations like the one described in the OP; see accepted answer).
For example you can do:
deploy:
rules:
- if: '$CI_COMMIT_TITLE =~ /urgent-fix/'
when: on_success
- when: manual # default fallback
script:
- sh deploy.sh
One thing to highlight is that in the above example I used $CI_COMMIT_TITLE instead of $CI_COMMIT_MESSAGE (see gitlab docs) to avoid the string "urgent-fix" reoccuring in a commit message automatically assembled in the course of a git merge, which would then accidentally retrigger the job.
Disclaimer: Please be aware that the new rules: feature stands in conflict with only/except: and thus requires you to remove only/except: occurences. Also, please note that rules: should only be used in combination with workflow: (read the docs) to avoid unwanted "detached" pipelines being triggered.
I'm trying to setup Gitlab CI for condition:
files modified by git push event on directory "spark/**/*"
OR there is a variable $PROJECT == 'spark'
The second condition is for pipelines created by api call (trigger token) which sets the variable.
With AND expression it looks like:
only:
variables:
- $PROJECT == 'spark'
changes:
- spark/**/*
So far my best try is:
deploy-spark: &deploy-spark
# job definition
only:
changes:
- spark/**/*
except:
- triggers
deploy-spark-token:
<<: *deploy-spark
only:
variables:
- $PROJECT == 'spark'
refs:
- triggers
except:
But its a lot of boilerplate
I'm not sure there's an elegant solution to this that significantly reduces the boilerplate to implement OR logic. Whenever I've had to do this I try to separate the "work" from the CI mechanisms to help with maintainability.
Pulling your job definition out into a template, and including that template into each of the deploy jobs helps to keep the interfaces clean. It means you have one obvious place to change the code that runs, and two places that use that code. The two deploy jobs don't depend on each other and any changes to one won't inadvertently affect the other.
.job-definition: &job-definition
script:
- echo "Some script
deploy-spark
<<: *job-definition
only:
changes:
- spark/**/*
except:
- triggers
deploy-spark-token:
<<: *job-definition
only:
variables:
- $PROJECT == 'spark'
refs:
- triggers
I know this probably doesn't answer your question, but I hope this helps.
Note that the . in .job-definition means that the job doesn't run (I like to think of it like hidden files in Linux, adding the dot prefix hides it from CI).