Splitting stages into multiple pipelines - gitlab

Question

Lets assume I have few stages
stages:
- mr:stage1
- mr:stage2
- mr:stage3
On all jobs I have rule:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
So I am getting pipeline like that
How can I split my 3 stages into 2 pipelines? For example I would like to have mr:stage1 and mr:stage2 in one pipeline and if this pipeline is successful, mr:stage3 will invoke in separate pipeline.
Thx for help

Answer 1

Each project or repository in GitLab has a single .gitlab-ci.yml file, and therefore a single pipeline. There is no way to have multiple pipelines like this.
You can include other yml files in your base .gitlab-ci.yml file, but this is solely for your convenience as the pipeline author. At runtime, the included files are copy and pasted into a single yml file, so there is still only ever a single pipeline.
However, that single pipeline can look drastically different depending on your use cases.
Let's say you have 3 jobs that only run on push events to a non-default branch. When you push to the default branch, these 3 jobs will not run. Let's say you have another 2 jobs that only run when there's a push event to the default branch. When you push here, these jobs will run, but if you push to a feature branch, they will not.
This scenario might look like this:
stages:
- build
- test
- deploy
Pull in Backend Dependencies nonprod:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME !== $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # run your dependency manager to pull in backend dependencies, ie Composer for PHP, including dependencies only used in lower environments, like Unit Testing libraries, etc.
# ...
Pull in Frontend Dependencies nonprod:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME !== $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # Run npm install, etc., including those only needed in lower environments
# ...
Pull in Backend Dependencies Prod:
stage: build
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && ($CI_COMMIT_REF_NAME === $CI_DEFAULT_BRANCH || $CI_COMMIT_TAG !== '')
when: always
- when: never
script:
- # run your dependency manager to pull in backend dependencies, ie Composer for PHP, without dev dependencies
# ...
Run Unit Tests:
stage: tests
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME !== $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # run our unit tests
Deploy to production:
stage: deploy
rules:
- if: $CI_PIPELINE_SOURCE === 'push' && $CI_COMMIT_REF_NAME === $CI_DEFAULT_BRANCH
when: always
- when: never
script:
- # deploy steps to prod
etc.
For the first two jobs, the rules say "If the event is a Push AND the branch isn't the default branch, always run this job. Otherwise, if any of these conditions isn't true, never run this job".
The Prod job's rules say "If the event is a Push AND the branch IS the default branch, OR it's a tag, run this job. Otherwise never run it."
Then we only run our unit test job for feature branches, and we only deploy to production for the default branch.
Therefore, depending on which branch/tag is pushed to, the pipeline instance will look very different. For a feature branch we'll have 2 build jobs, and a test job (2 stages). For the default branch, we'll have a single build job and a deploy job.
The same is true if you need to handle sources other than 'push'. For example, if you have a job that only runs when triggered via the API (or from another pipeline instance or another project's pipeline), you'd look to see if the $CI_PIPELINE_SOURCE variable holds the string trigger.