New to Gitlab CI/CD. My build job works wonderfully, but deploy job is never executed. I removed all rules and still it does not run.
Here is the contents of my .gitlab-ci.yml file:
build-job:
stage: build
script:
- echo "STAGE - BUILD"
- echo $CI_JOB_STAGE
- echo $CI_COMMIT_MESSAGE
- echo $CI_COMMIT_BRANCH
- echo $CI_ENVIRONMENT_NAME
- mkdir bin
- mkdir obj
- "dotnet build"
deploy-to-staging:
stage: deploy
script:
- echo "STAGE - DEPLOY (STAGING)"
Any idea why Gitlab would skip the deploy stage? Do I have to explicitly define my stages? I tried that, but it made no difference (These lines were at the bottom of the yml file for a while):
stages:
- build
- deploy
While it's not explicit in the stages documentation, you should generally set those at the top.
If you're getting a yaml invalid failure, then use the CI lint tool to double check your spacing and the like without having to run a pipeline.
Keep in mind that:
If a job fails, it doesn't start the next stage.
If you don't define stages, then it uses build, test, deploy.
Any job that doesn't have a stage defined is assumed to be test.
Any stage that isn't used should simply be hidden. However, the exact behaviour may depend on the version of GitLab you're on. (If memory serves correctly, this was changed, but I can't the merge request off hand.)
As you can see in the official documentation of Gitlab CI, by defining stages, the sequence and the order of execution of jobs will be specified.
So the following gitlab-ci.yml should works:
stages:
- build
- deploy
build-job:
stage: build
script:
- echo "STAGE - BUILD"
- echo $CI_JOB_STAGE
- echo $CI_COMMIT_MESSAGE
- echo $CI_COMMIT_BRANCH
- echo $CI_ENVIRONMENT_NAME
- mkdir bin
- mkdir obj
- "dotnet build"
deploy-to-staging:
stage: deploy
script:
- echo "STAGE - DEPLOY (STAGING)"
Screenshot:
Related
consider the following gitlab-ci.yaml, for a mono repo with multiple microfrontends
stages:
- build
- deploy
build:app1:
stage: build
script:
- sleep 30
- mkdir dist1
- touch dist1/output1.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app1/src/*
artifacts:
paths:
- dist1
build:app2:
stage: build
script:
- sleep 30
- mkdir dist2
- touch dist2/output2.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app2/src/*
artifacts:
paths:
- dist2
deploy:all:
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
when ran the order defined in stages is ignored and both the build and deploy jobs run simultaneously
causing a failure for the "deploy:all" job (since its still "building")
if i remove the condition for the changes the stage order is respected and build runs before deploy
how can i both only act on changes and enforce the defined build order?
in the real monorepo there are 10's of micro frontends not just 2 ...
When I run your gitlab_ci.yml in Gitlab CI, it does not run the build and deploy jobs simultaneously. It runs a merge request pipeline (with the build jobs) and a branch pipeline (with the deploy job). Since the artifacts from the build jobs are created in the merge request pipeline, they aren't available to the branch pipeline, and so the deploy job fails.
It's difficult to say how to fix this without knowing what your intention is, but you need to run the build and deploy jobs in the same pipeline, so either have the deploy job run with
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
or have the build jobs run in a branch pipeline.
Additionally it's not clear to me how you're expecting the deploy job to always work, as if you only run the build jobs when there are changes sometimes not every build job will have an artifact and your deploy job will fail.
In your case, you can try to use needs keyword in order to force relationships between jobs: https://docs.gitlab.com/ee/ci/yaml/#needs
You also nee to add optional: true since you have rules conditions in your build jobs.
You can add it in your deploy:all job like this:
deploy:all:
needs:
- job: build:app1
optional: true
- job: build:app2
optional: true
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
Your deploy stage might need the condition for changes as well. Unless maybe you need the deploy:all stage to always run no matter if there is changes or not.
Also indicating that the deploy stage needs artifacts is always a good idea in your case because your deploy stage is using the artifacts anyways
needs:
- job: build:app1
artifacts: true
- job: build:app2
artifacts: true
I got an error on GitLab Pipeline when I commit the next .gitlab-ci.yml for a repository.
Pipeline executed to build solution, deploy to Artifactory and trigger and API call
Deploy job have to be executed manually, and there are two different job options to execute.
stages:
- build
- deploy
- trigger
variables:
APP_PROJECT_ID: ${CUSTOMER_RELEASED}
build_job:
stage: build
tags:
- dotnet
script:
- echo "build"
only:
- tags
allow_failure: false
.deploy_job_base:
stage: deploy
needs: [build_job]
tags:
- dotnet
script:
- echo "deploy"
dependencies:
- build_job
only:
- tags
deploy_job_sport:
extends: .deploy_job_base
after_script:
- $APP_PROJECT_ID = "2096"
when: manual
allow_failure: false
deploy_job_others:
extends: .deploy_job_base
after_script:
- $APP_PROJECT_ID = "0"
when: manual
allow_failure: false
.trigger_base:
stage: trigger
script:
- echo "Customer Project ID '{$APP_PROJECT_ID}'"
- echo "Call API..."
trigger_sport:
extends: .trigger_base
needs: [deploy_job_sport]
trigger_others:
extends: .trigger_base
needs: [deploy_job_others]
Lint status is correct
but I get that error GitLab Pipeline when I commit changes:
Found errors in your .gitlab-ci.yml: 'trigger_sport' job needs
'deploy_job_sport' job but 'deploy_job_sport' is not in any previous
stage 'trigger_others' job needs 'deploy_job_others' job but
'deploy_job_others' is not in any previous stage
If I remove trigger_sport and trigger_others job and create only one job, it works fine but I don't know how I can target the two manual jobs (deploy_job_sport and deploy_job_others) to a single job.
Do you have any idea?
Thanks in advance.
I think this is related to the fact that you're using only: tags in your template for deploy jobs and the build job also is limited to run when commits contain a tag.
But the trigger template is missing this limitation which most likely causes this error when pushing a commit without a tag because the pipeline creation would add trigger_XY to the pipeline, which has dependencies to the previous deploy_XY jobs.
When updating your job template for trigger jobs to the following this error should be resolved:
.trigger_base:
stage: trigger
script:
- echo "Customer Project ID '{$APP_PROJECT_ID}'"
- echo "Call API..."
only:
- tags
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE
I currently have two jobs in my CI file which are nearly identical.
The first is for manually compiling a release build from any git branch.
deploy_internal:
stage: deploy
script: ....<deploy code>
when: manual
The second is to be used by the scheduler to release a daily build from develop branch.
scheduled_deploy_internal:
stage: deploy
script: ....<deploy code from deploy_internal copy/pasted>
only:
variables:
- $MY_DEPLOY_INTERNAL != null
This feels wrong to have all that deploy code repeated in two places. It gets worse. There are also deploy_external, deploy_release, and scheduled variants.
My question:
Is there a way that I can combine deploy_internal and scheduled_deploy_internal such that the manual/scheduled behaviour is retained (DRY basically)?
Alternatively: Is there is a better way that I should structure my jobs?
Edit:
Original title: Deploy job. Execute manually except when scheduled
You can use YAML anchors and aliases to reuse the script.
deploy_internal:
stage: deploy
script:
- &deployment_scripts |
echo "Deployment Started"
bash command 1
bash command 2
when: manual
scheduled_deploy_internal:
stage: deploy
script:
- *deployment_scripts
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Or you can use extends keyword.
.deployment_script:
script:
- echo "Deployment started"
- bash command 1
- bash command 2
deploy_internal:
extends: .deployment_script
stage: deploy
when: manual
scheduled_deploy_internal:
extends: .deployment_script
stage: deploy
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Use GitLab's default section containing a before_script:
default:
before_script:
- ....<deploy code>
job1:
stage: deploy
script: ....<code after than deploy>
job2:
stage: deploy
script: ....<code after than deploy>
Note: the default section fails to function as such if you try to execute a job locally with the gitlab-runner exec command - use YAML anchors instead.
I'm using Gitlab CI, and so have been working on a fairly complex .gitlab-ci.yml file. The file has an after_script section which runs when the main task is complete, or the main task has failed somehow. Problem: I need to do different cleanup based on whether the main task succeeded or failed, but I can't find any Gitlab CI variable that indicates the result of the main task.
How can I tell, inside the after_script section, whether the main task has succeeded or failed?
Since gitlab-runner 13.5, you can use the CI_JOB_STATUS variable.
test_job:
# ...
after_script:
- >
if [ $CI_JOB_STATUS == 'success' ]; then
echo 'This will only run on success'
else
echo 'This will only run when job failed or is cancelled'
fi
See GitLab's documentation on predefined_variables: https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
The accepted answer may apply to most situations, but it doesn't answer the original question and will only work if you only have one job per stage.
Note: There currently a feature request opened (issues/3116) to handle on_failure and on_success in after_script.
It could be possible to use variables to pass the job status to an after_script script, but this also has a feature request (issues/1926) opened to be able to share variables between before_script, script and after_script.
One workaround will be to write to a temporary file that will be accessed during the after_script block.
test_job:
stage: test
before_script:
- echo "FAIL" > .job_status
script:
- exit 1
- echo "SUCCESS" > .job_status
after_script:
- echo "$(cat .job_status)"
Instead of determining whether or not the task succeeded or failed in the after_script, I would suggest defining another stage, and using the when syntax, where you can use when: on_failure or when: on_success.
Example from the documentation:
stages:
- build
- cleanup_build
- test
- deploy
- cleanup
build_job:
stage: build
script:
- make build
cleanup_build_job:
stage: cleanup_build
script:
- cleanup build when failed
when: on_failure
test_job:
stage: test
script:
- make test
deploy_job:
stage: deploy
script:
- make deploy
when: manual
cleanup_job:
stage: cleanup
script:
- cleanup after jobs
when: always
Just another way to handle this if you want to setup if failure behavior.
scripts:
- ./script_that_fails.sh > /dev/null 2>&1 || FAILED=true
- if [ $FAILED ]
then ./do_something.sh
fi
Note: Other examples also worked for me, but I find this implementation more faster and suitable for me