consider the following gitlab-ci.yaml, for a mono repo with multiple microfrontends
stages:
- build
- deploy
build:app1:
stage: build
script:
- sleep 30
- mkdir dist1
- touch dist1/output1.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app1/src/*
artifacts:
paths:
- dist1
build:app2:
stage: build
script:
- sleep 30
- mkdir dist2
- touch dist2/output2.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app2/src/*
artifacts:
paths:
- dist2
deploy:all:
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
when ran the order defined in stages is ignored and both the build and deploy jobs run simultaneously
causing a failure for the "deploy:all" job (since its still "building")
if i remove the condition for the changes the stage order is respected and build runs before deploy
how can i both only act on changes and enforce the defined build order?
in the real monorepo there are 10's of micro frontends not just 2 ...
When I run your gitlab_ci.yml in Gitlab CI, it does not run the build and deploy jobs simultaneously. It runs a merge request pipeline (with the build jobs) and a branch pipeline (with the deploy job). Since the artifacts from the build jobs are created in the merge request pipeline, they aren't available to the branch pipeline, and so the deploy job fails.
It's difficult to say how to fix this without knowing what your intention is, but you need to run the build and deploy jobs in the same pipeline, so either have the deploy job run with
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
or have the build jobs run in a branch pipeline.
Additionally it's not clear to me how you're expecting the deploy job to always work, as if you only run the build jobs when there are changes sometimes not every build job will have an artifact and your deploy job will fail.
In your case, you can try to use needs keyword in order to force relationships between jobs: https://docs.gitlab.com/ee/ci/yaml/#needs
You also nee to add optional: true since you have rules conditions in your build jobs.
You can add it in your deploy:all job like this:
deploy:all:
needs:
- job: build:app1
optional: true
- job: build:app2
optional: true
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
Your deploy stage might need the condition for changes as well. Unless maybe you need the deploy:all stage to always run no matter if there is changes or not.
Also indicating that the deploy stage needs artifacts is always a good idea in your case because your deploy stage is using the artifacts anyways
needs:
- job: build:app1
artifacts: true
- job: build:app2
artifacts: true
Related
New to Gitlab CI/CD. My build job works wonderfully, but deploy job is never executed. I removed all rules and still it does not run.
Here is the contents of my .gitlab-ci.yml file:
build-job:
stage: build
script:
- echo "STAGE - BUILD"
- echo $CI_JOB_STAGE
- echo $CI_COMMIT_MESSAGE
- echo $CI_COMMIT_BRANCH
- echo $CI_ENVIRONMENT_NAME
- mkdir bin
- mkdir obj
- "dotnet build"
deploy-to-staging:
stage: deploy
script:
- echo "STAGE - DEPLOY (STAGING)"
Any idea why Gitlab would skip the deploy stage? Do I have to explicitly define my stages? I tried that, but it made no difference (These lines were at the bottom of the yml file for a while):
stages:
- build
- deploy
While it's not explicit in the stages documentation, you should generally set those at the top.
If you're getting a yaml invalid failure, then use the CI lint tool to double check your spacing and the like without having to run a pipeline.
Keep in mind that:
If a job fails, it doesn't start the next stage.
If you don't define stages, then it uses build, test, deploy.
Any job that doesn't have a stage defined is assumed to be test.
Any stage that isn't used should simply be hidden. However, the exact behaviour may depend on the version of GitLab you're on. (If memory serves correctly, this was changed, but I can't the merge request off hand.)
As you can see in the official documentation of Gitlab CI, by defining stages, the sequence and the order of execution of jobs will be specified.
So the following gitlab-ci.yml should works:
stages:
- build
- deploy
build-job:
stage: build
script:
- echo "STAGE - BUILD"
- echo $CI_JOB_STAGE
- echo $CI_COMMIT_MESSAGE
- echo $CI_COMMIT_BRANCH
- echo $CI_ENVIRONMENT_NAME
- mkdir bin
- mkdir obj
- "dotnet build"
deploy-to-staging:
stage: deploy
script:
- echo "STAGE - DEPLOY (STAGING)"
Screenshot:
I have like this gitlab ci cd configuration file:
image: docker:git
stages:
- develop
- production
default:
before_script:
- apk update && apk upgrade && apk add git curl
deploy:
stage: develop
script:
- echo "Hello World"
backup:
stage: develop
when:
- manual
- on_success
remove:
stage: develop
when:
- delayed
- on_success
start_in: 30 minutes
In my case job deploy runs automaticaly and job backup must runs manually only when successfully completed job deploy. But in my case this configuration doesn't works and I get error with message:
Found errors in your .gitlab-ci.yml:
jobs:backup when should be one of:
on_success
on_failure
always
manual
delayed
How I can use multiple when option arguments in my case?
Basically you can't because when does not expect an array. You can work around it though with needs. But this solution does only work if you run your jobs in different stages.
image: docker:git
stages:
- deploy
- backup
- remove
deploy:develop:
stage: deploy
script:
- exit 1
backup:develop:
stage: backup
script:
- echo "backup"
when: manual
needs: ["deploy:develop"]
remove:develop:
stage: remove
script:
- echo "remove"
when: delayed
needs: ["backup:develop"]
start_in: 30 minutes
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE
I have two simple stages. (build and test). And I want jobs in the pipeline to run sequentially.
Actually, I want when I run the test job, it doesn't run until the build job was passed completely.
My gitlab file:
stages:
- build
- test
build:
stage: build
script:
- mvn clean package
only:
- merge_requests
test:
stage: test
services:
script:
- mvn verify
- mvn jacoco:report
artifacts:
reports:
junit:
- access/target/surefire-reports/TEST-*.xml
paths:
- access/target/site/jacoco
expire_in: 1 week
only:
- merge_requests
Can I add
needs:
- build
in the test stage?
Based on the simplicity of your build file, i do not think that you actively need the needs. Based on the documentation, all stages are executed sequentially.
The pitfall you are in right now, is the only reference. The build stage will run for any branch, and for that ignore merge requests. if you add a only directive to your build job, you might get the result you are looking for like:
build:
stage: build
script:
- mvn clean package
only:
- merge_requests
- master # might be main, develop, etc. what ever your longliving branches are
This way it will not be triggered for each branch but only for merge requests and the long living branches. see the only documentation. Now the execution is not assigned to the branch but to the merge request, and you will have your expected outcome (at least what i assume)
I am having a problem with some Gitlab CI jobs where I specify a rule to run only if a file exists.
This is my .gitlab-ci.yml:
stages:
- build
- test
#Jobs
build:
stage: build
script:
- dotnet restore --no-cache --force
- dotnet build --configuration Release --no-restore
artifacts:
paths:
- test/
expire_in: 1 week
unit_tests:
stage: test
script: dotnet vstest test/*UnitTests/bin/Release/**/*UnitTests.dll --Blame
rules:
- exists:
- test/*UnitTests/bin/Release/**/*UnitTests.dll
integration_tests:
stage: test
script: dotnet vstest test/*IntegrationTests/bin/Release/**/*IntegrationTests.dll --Blame
rules:
- exists:
- test/*IntegrationTests/bin/Release/**/*IntegrationTests.dll
I want to run unit_tests only when there are *UnitTests.dll under the bin in test/ folder and integration_tests only when there are *IntegrationTests.dll under the bin in test/ folder also.
The problem is that both jobs are completely ignored. In other words, Gitlab seems to be evaluating the exists to false as if it was only evaluated at the beginning of the pipeline and not at the beginning of the job, but these paths exist because they're generated in the previous stage and the artifacts are automatically available.
If I remove the rules the unit_tests will run successfully but integration_tests will fail because at my specific project there are no integration tests.
I've tried replacing exists with changes, same problem.
How can I achieve this conditional job execution?
UPDATE 1: I have an ugly workaround but the question remains because the exists seems to be evaluated at the beginning of the pipeline and not at the beginning of the job and, therefore, anything regarding artifacts is ignored.
This trick works because I always assume that if there's a csproj there will be a dll later on as the result of the build stage.
stages:
- build
- test
build:
stage: build
script:
- dotnet restore --no-cache --force
- dotnet build --configuration Release --no-restore
artifacts:
paths:
- test/
expire_in: 1 week
unit_tests:
stage: test
script: dotnet vstest test/*UnitTests/bin/Release/**/*UnitTests.dll --Blame
rules:
- exists:
- test/*UnitTests/*UnitTests.csproj
integration_tests:
stage: test
script: dotnet vstest test/*IntegrationTests/bin/Release/**/*IntegrationTests.dll --Blame
rules:
- exists:
- test/*IntegrationTests/*IntegrationTests.csproj
At the time of writing, it appears that GitLab doesn't support the use of artifact files in rules. This issue confirms that it doesn't work.
My own workaround is to remove the conditional rule and instead write a wrapper script that first checks the presence of the file.