I have the following stages defined in my project's gitlab-config.yaml:
stages:
- Prepare
- Report
- Static Analysis
- Unit Test
- Integration Test Prep
- Integration Tests
- Deploy
The stage before Deploy is Integration Tests, and all jobs within this stage are not allowed to fail (which is the default according to the docs).
I have a number of deploy jobs that deploy to different environments. My production deploy job uses the following logic:
rules:
- if: $DEPLOY_ENV == "production" && $CI_COMMIT_BRANCH == "production"
when: always
My problem with the current setup is that even though the Integration Tests jobs are not allowed to fail, even if they do, the production deploy stage is still reached.
It appears that the use of always overrides the fact that the previous stage's jobs are not allowed to fail.
How can I prevent the production deploy job from running if any of the previous Integration Tests jobs fail?
The solution is to use on_success instead of always (docs):
rules:
- if: $DEPLOY_ENV == "production" && $CI_COMMIT_BRANCH == "production"
when: on_success
You can also use the needs keyword (https://docs.gitlab.com/ee/ci/yaml/index.html#needs) to specify more complex dependencies between jobs
jobA:
stage: build
script: echo "Building"
jobB:
stage: build
needs: ["jobA"]
script: echo "Another Build"
Related
I'm new to this Gitlab CI/CD features, and I encountered the following issues.
I have these 2 jobs in my gitlab-ci.yml, the automation test and my deployment job.
automation_test_scheduled:
stage: test
script:
- yarn test:cypress
only:
- schedules
deploy_to_staging:
stage: deploy-staging
environment: staging
only:
- staging
I want to run my automation test automatically on a daily basis and I have created a new pipeline schedule against my staging branch.
however, when the scheduler is triggered, it also runs my deployment job, which is not needed because I only want my automation test to run in the scheduled pipeline.
Does this happen because my deploy_to_staging job has only: - staging rules? if so, how can I set my scheduled pipeline to only run the automation test without triggering another job?
If you wanted to do this with only/except, it would probably be sufficient to add
except:
- schedules
to your deployment job.
Though as
Though notably, the rules based system is preferred at this point.
This also allows for more expressive and detailed decisions when it comes to running jobs.
The simplest way to set the rules for the two jobs would be:
automation_test_scheduled:
stage: test
script:
- yarn test:cypress
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
deploy_to_staging:
stage: deploy-staging
environment: staging
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: never
- if: $CI_COMMIT_REF_SLUG == "staging"
And that might be all you need.
Though when it comes to rules, a particularly convenient way of handling them is defining some common rules for the configuration, and reusing these through yaml anchors. The following are some reusable definitions for your case:
.definitions:
rules:
- &if-scheduled
if: $CI_PIPELINE_SOURCE == "schedule"
- ¬-scheduled
if: $CI_PIPELINE_SOURCE == "schedule"
when: never
- &if-staging
if: $CI_COMMIT_REF_SLUG == "staging"
And after that you could use them in your jobs like this:
automation_test_scheduled:
stage: test
script:
- yarn test:cypress
rules:
- *if-scheduled
deploy_to_staging:
stage: deploy-staging
environment: staging
rules:
- *not-scheduled
- *if-staging
This way of handling the rules makes it a bit easier to have a overview, and to reuse rules, which in large configurations definitely makes sense
You should use rules instead of only as the latter is not in active development any more.
With that in mind you can change to the following rules clause using the predefined variables CI_COMMIT_REF_SLUG and CI_PIPELINE_SOURCE. The automation_test_scheduled is only run on the branch staging if triggered by a schedule and the deploy_to_staging job is run on any change on the staging branch.
automation_test_scheduled:
stage: test
script:
- yarn test:cypress
rules:
- if: '$CI_COMMIT_REF_SLUG == "staging" && $CI_PIPELINE_SOURCE == "schedule"'
deploy_to_staging:
stage: deploy-staging
environment: staging
rules:
- if: '$CI_COMMIT_REF_SLUG == "staging"'
Lets assume I have few stages
stages:
- mr:stage1
- mr:stage2
- mr:stage3
On all jobs I have rule:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
So I am getting pipeline like that
How can I split my 3 stages into 2 pipelines? For example I would like to have mr:stage1 and mr:stage2 in one pipeline and if this pipeline is successful, mr:stage3 will invoke in separate pipeline.
Thx for help
Each project or repository in GitLab has a single .gitlab-ci.yml file, and therefore a single pipeline. There is no way to have multiple pipelines like this.
You can include other yml files in your base .gitlab-ci.yml file, but this is solely for your convenience as the pipeline author. At runtime, the included files are copy and pasted into a single yml file, so there is still only ever a single pipeline.
However, that single pipeline can look drastically different depending on your use cases.
Let's say you have 3 jobs that only run on push events to a non-default branch. When you push to the default branch, these 3 jobs will not run. Let's say you have another 2 jobs that only run when there's a push event to the default branch. When you push here, these jobs will run, but if you push to a feature branch, they will not.
This scenario might look like this:
stages:
- build
- test
- deploy
Pull in Backend Dependencies nonprod:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME !== $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # run your dependency manager to pull in backend dependencies, ie Composer for PHP, including dependencies only used in lower environments, like Unit Testing libraries, etc.
# ...
Pull in Frontend Dependencies nonprod:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME !== $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # Run npm install, etc., including those only needed in lower environments
# ...
Pull in Backend Dependencies Prod:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && ($CI_COMMIT_REF_NAME === $CI_DEFAULT_BRANCH || $CI_COMMIT_TAG !== '')
when: always
- when: never
script:
- # run your dependency manager to pull in backend dependencies, ie Composer for PHP, without dev dependencies
# ...
Run Unit Tests:
stage: tests
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME !== $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # run our unit tests
Deploy to production:
stage: deploy
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME === $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # deploy steps to prod
etc.
For the first two jobs, the rules say "If the event is a Push AND the branch isn't the default branch, always run this job. Otherwise, if any of these conditions isn't true, never run this job".
The Prod job's rules say "If the event is a Push AND the branch IS the default branch, OR it's a tag, run this job. Otherwise never run it."
Then we only run our unit test job for feature branches, and we only deploy to production for the default branch.
Therefore, depending on which branch/tag is pushed to, the pipeline instance will look very different. For a feature branch we'll have 2 build jobs, and a test job (2 stages). For the default branch, we'll have a single build job and a deploy job.
The same is true if you need to handle sources other than 'push'. For example, if you have a job that only runs when triggered via the API (or from another pipeline instance or another project's pipeline), you'd look to see if the $CI_PIPELINE_SOURCE variable holds the string trigger.
I setup jobs to run only when pushing/merging to branch "dev", but I also want it so I'm able to run them if I trigger that pipeline manually. Something like this:
test:
stage: test
<this step should be run always>
build:
stage: build
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
- if: <also run if the pipeline was run manually, but skip if it was triggered by something else>
This job is defined in a child "trigger" pipeline. This is how the parent looks like:
include:
- template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml'
stages:
- triggers
microservice_a:
stage: triggers
trigger:
include: microservice_a/.gitlab-ci.microservice_a.yml
strategy: depend
rules:
- changes:
- microservice_a/*
The effect I want to achieve is:
Run test in all cases
Run build in the child pipeline only when pushing/merging to "dev"
Also run the build job when the pipeline is run maually
Do not run the build job on any other cases (like a MR)
The rules examples showcase:
job:
script: echo "Hello, Rules!"
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
when: manual
allow_failure: true
- if: '$CI_PIPELINE_SOURCE == "schedule"'
The when:manual should be enough in your case: it does require that a job doesn’t run unless a user starts it.
Bonus question: This job is defined in a child "trigger" pipeline
Then it is related to gitlab-org/gitlab issue 201938, which is supposed to be fixed with GitLab 13.5 (Oct. 2020), but that only allow manual actions for parent-child pipeline (illustrated by this thread)
Double-check the environment variables as set in your child job
echo $CI_JOB_MANUAL
If true, that would indicate a job part of a manual triggered job.
While issue 22448 ("$CI_JOB_MANUAL should be set in all dependent jobs") points to this option not working, it includes a workaround.
cleanup job are defined as:
cleanup:
stage:e2e
needs:
- job: deploy
stage: e2e
script:
- make clean
resource_group: development
when: always
Main goal is to run cleanup job when deploy job successes or failed (whatever) why i use when:always and added needs:- job: deploy.
But problem is if any previous job failed it also trigger to run cleanup job, even if deploy job does not run.
I think you need to remove the when: always definition from you jobs config.
Instead you should adjust your needs to be optional like given below - that way pipeline will run
cleanup:
stage: e2e
needs:
- job: deploy
optional: true
script:
- make clean
resource_group: development
Another thing you could try would be to change your rules to say run this job on_success or on_failure
cleanup:
stage: e2e
needs:
- job: deploy
script:
- make clean
resource_group: development
rules:
# Dummy if rule to showcase multiple rules
- if: $CI_COMMIT_TAG
when: manual
- when: on_success
- when: on_failure
That second way the job would run if previous job succeeds and also if not, but would be limited to execution after deploy job is run.
Did not test this but I think this might work.
I have an A project and an E2E project. I want to deploy A project trigger E2E pipeline run test but I just want the trigger test stage. we don't need trigger E2E to build deploy ...etc
e2e_tests:
stage: test
trigger:
project: project/E2E
branch: master
strategy: depend
stage: test
I have tried to use the stage in config. but got error unknown keys: stage
have any suggestions?
In your E2E project, the one that receives the trigger, you can tell a job to only run when the pipeline source is a trigger using the rules syntax:
build-from-trigger:
stage: build
when: never
rules:
- if: "$CI_COMMIT_REF_NAME == 'master' && $CI_PIPELINE_SOURCE == 'trigger'
when: always
script:
- ./build.sh # this is just an example, you'd run whatever you normally would here
The first when statement, when: never sets the default for the job. By default, this job will never run. Then using the rule syntax, we set a condition that will allow the job to run. If the CI_COMMIT_REF_NAME variable (the branch or tag name) is master AND the CI_PIPELINE_SOURCE variable (whatever kicked off this pipeline) is a trigger, then we run this job.
You can read about the when keyword here: https://docs.gitlab.com/ee/ci/yaml/#when, and you can read the rules documentation here: https://docs.gitlab.com/ee/ci/yaml/#rules