I have two simple stages. (build and test). And I want jobs in the pipeline to run sequentially.
Actually, I want when I run the test job, it doesn't run until the build job was passed completely.
My gitlab file:
stages:
- build
- test
build:
stage: build
script:
- mvn clean package
only:
- merge_requests
test:
stage: test
services:
script:
- mvn verify
- mvn jacoco:report
artifacts:
reports:
junit:
- access/target/surefire-reports/TEST-*.xml
paths:
- access/target/site/jacoco
expire_in: 1 week
only:
- merge_requests
Can I add
needs:
- build
in the test stage?
Based on the simplicity of your build file, i do not think that you actively need the needs. Based on the documentation, all stages are executed sequentially.
The pitfall you are in right now, is the only reference. The build stage will run for any branch, and for that ignore merge requests. if you add a only directive to your build job, you might get the result you are looking for like:
build:
stage: build
script:
- mvn clean package
only:
- merge_requests
- master # might be main, develop, etc. what ever your longliving branches are
This way it will not be triggered for each branch but only for merge requests and the long living branches. see the only documentation. Now the execution is not assigned to the branch but to the merge request, and you will have your expected outcome (at least what i assume)
Related
consider the following gitlab-ci.yaml, for a mono repo with multiple microfrontends
stages:
- build
- deploy
build:app1:
stage: build
script:
- sleep 30
- mkdir dist1
- touch dist1/output1.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app1/src/*
artifacts:
paths:
- dist1
build:app2:
stage: build
script:
- sleep 30
- mkdir dist2
- touch dist2/output2.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app2/src/*
artifacts:
paths:
- dist2
deploy:all:
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
when ran the order defined in stages is ignored and both the build and deploy jobs run simultaneously
causing a failure for the "deploy:all" job (since its still "building")
if i remove the condition for the changes the stage order is respected and build runs before deploy
how can i both only act on changes and enforce the defined build order?
in the real monorepo there are 10's of micro frontends not just 2 ...
When I run your gitlab_ci.yml in Gitlab CI, it does not run the build and deploy jobs simultaneously. It runs a merge request pipeline (with the build jobs) and a branch pipeline (with the deploy job). Since the artifacts from the build jobs are created in the merge request pipeline, they aren't available to the branch pipeline, and so the deploy job fails.
It's difficult to say how to fix this without knowing what your intention is, but you need to run the build and deploy jobs in the same pipeline, so either have the deploy job run with
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
or have the build jobs run in a branch pipeline.
Additionally it's not clear to me how you're expecting the deploy job to always work, as if you only run the build jobs when there are changes sometimes not every build job will have an artifact and your deploy job will fail.
In your case, you can try to use needs keyword in order to force relationships between jobs: https://docs.gitlab.com/ee/ci/yaml/#needs
You also nee to add optional: true since you have rules conditions in your build jobs.
You can add it in your deploy:all job like this:
deploy:all:
needs:
- job: build:app1
optional: true
- job: build:app2
optional: true
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
Your deploy stage might need the condition for changes as well. Unless maybe you need the deploy:all stage to always run no matter if there is changes or not.
Also indicating that the deploy stage needs artifacts is always a good idea in your case because your deploy stage is using the artifacts anyways
needs:
- job: build:app1
artifacts: true
- job: build:app2
artifacts: true
I am building a Gitlab CI pipeline and I couldn't find how can I add an OR dependency for my job. e.g.,
stages:
- test
- release
- build
test:
stage: test
script:
- do something
bump patch version:
stage: release
when: manual
script:
- do something
bump minor version:
stage: release
when: manual
script:
- do something
build:
stage: build
dependencies:
- # I want only one of the bump stages to complete before this gets triggered
before_script: []
script:
So I have three stages - test, release, and build. Now release is a manual job that creates git tag with version number depending if it is a patch/minor/major change. So only one of this bump thing will get triggered manually. After this I want the build section to autorun.
Therfore, how can I have one of the bump stages to complete, so that the build stage can be triggered automatically?
I have two jobs in GitLab CI/CD: build-job (which takes ~10 minutes) and deploy-job (which takes ~2 minutes). In a pipeline, build-job succeeded while deploy-job failed because of a typo in the script. How can I fix this typo and run only deploy-job, instead of rerunning build-job?
My current .gitlab-ci.yml is as follows:
stages:
- build
- deploy
build-job:
image: ...
stage: build
only:
refs:
- main
changes:
- .gitlab-ci.yml
- pubspec.yaml
- test/**/*
- lib/**/*
script:
- ...
artifacts:
paths:
- ...
expire_in: 1 hour
deploy-job:
image: ...
stage: deploy
dependencies:
- build-job
only:
refs:
- main
changes:
- .gitlab-ci.yml
- pubspec.yaml
- test/**/*
- lib/**/*
script:
- ...
I imagine something like:
the triggerer (me) fixes the typo in deploy-job's script and pushes the changes
the pipeline is triggered
the runner looks at the files described by build-job/only/changes and detects a single change in the .gitlab-ci.yml file
the runner looks at the .gitlab-ci.yml file and detects there IS a change, but since this change does not belong to the build-job section AND the job previously succeeded, this job is skipped
the runner looks at the files described by deploy-job/only/changes and detects a single change in the .gitlab-ci.yml file
the runner looks at the .gitlab-ci.yml file and detects there is a change, and since this change belongs to the deploy-job section, this job is executed
This way, only deploy-job is executed. Can this be done, either using rules or only?
There is a straight foward way, you just have to have access to gitlab CI interface.
Push and let the pipeline start normally.
Cancel the fist job.
Run the desired one (deploy).
Another way is to create a new job, intended to be runned only when needed, in which you can set a non-dependency deploy. Just copy the current deploy code without the dependencies. Also add the manual mode with
job:
script: echo "Hello, Rules!"
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: manual
allow_failure: true
As docs.
I have 2 stages with multiple jobs and the jobs in the first stage have some rules that tell them if the need to run or not, so what I am trying to do is to tell some of the jobs in the second stage to execute only if the relevant job in the first stage ran.
I don't want to use the same rules I used for the first stage job to prevent conflicts.
Is there a way to do that?
stages:
- build
- deploy
Build0:
stage: build
extends:
- .Build0Rules
- .Build0Make
Build1:
stage: build
extends:
- .Build1Rules
- .Build1Make
Deploy0:
stage: deploy
dependencies:
- Build0
script:
- bash gitlab-ci/deploy0.sh
Deploy1:
stage: deploy
dependencies:
- Build1
script:
- bash gitlab-ci/deploy1.sh
Thank you in advance :)
No you cannot specify that a job should be added to the pipeline if another job was added to the pipeline. Each job can specify whether it is added to the pipeline using only/except conditions or rules, but these are not able to reference other jobs.
It is possible to generate a pipeline yaml file and then trigger it, but I think this would not be ideal because of the amount of work involved.
stages:
- Build
- Deploy
build:
stage: Build
script:
- do something...
artifacts:
paths:
- deploy-pipeline-gitlab-ci.yml
deploy:
stage: Deploy
trigger:
include:
- artifact: deploy-pipeline-gitlab-ci.yml
job: build
strategy: depend
I would recommend using similar only/except conditions or rules on each job to build the pipeline that you want.
Yes you can. You should check the keyword needs that allow to do what you want: execute a job based on the execution of other jobs, ignoring stages order.
The documentation: https://docs.gitlab.com/ee/ci/yaml/#needs
Here is also an exemple of how to build a DAG (direct acrylic graph) using needs: https://about.gitlab.com/blog/2020/12/10/basics-of-gitlab-ci-updated/#directed-acyclic-graphs-get-faster-and-more-flexible-pipelines
In your case:
Deploy0:
stage: deploy
needs: ["Build0"]
script:
- bash gitlab-ci/deploy0.sh
Deploy1:
stage: deploy
needs: ["Build1"]
script:
- bash gitlab-ci/deploy1.sh
Note you can also specify multiple jobs in the needs command:
needs: ["build0", "test0", "test1"]
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE