cleanup job are defined as:
cleanup:
stage:e2e
needs:
- job: deploy
stage: e2e
script:
- make clean
resource_group: development
when: always
Main goal is to run cleanup job when deploy job successes or failed (whatever) why i use when:always and added needs:- job: deploy.
But problem is if any previous job failed it also trigger to run cleanup job, even if deploy job does not run.
I think you need to remove the when: always definition from you jobs config.
Instead you should adjust your needs to be optional like given below - that way pipeline will run
cleanup:
stage: e2e
needs:
- job: deploy
optional: true
script:
- make clean
resource_group: development
Another thing you could try would be to change your rules to say run this job on_success or on_failure
cleanup:
stage: e2e
needs:
- job: deploy
script:
- make clean
resource_group: development
rules:
# Dummy if rule to showcase multiple rules
- if: $CI_COMMIT_TAG
when: manual
- when: on_success
- when: on_failure
That second way the job would run if previous job succeeds and also if not, but would be limited to execution after deploy job is run.
Did not test this but I think this might work.
Related
Please in my gitlab-ci.yml, for instance, I want to have a cleanup that would help me each time a job is run successfully during Acceptance testing. All file generated in my server, I will like to purge and restart apache.
stages:
- build
- cleanup_build
- test
- deploy
- cleanup
build_job:
stage: build
script:
- make build
cleanup_build_job:
stage: cleanup_build
script:
- cleanup build when failed
when: on_failure
test_job:
stage: test
script:
- make test
deploy_job:
stage: deploy
script:
- make deploy
when: manual
cleanup_job:
stage: cleanup
script:
- cleanup after jobs
when: always
In global main.yml, I have something like to create a vhost for each test in my ansible. On job completion, I want to remove the vhost created during the UAT. It is long file and I can't be sharing all the configurations here. Ultimately, what I want to do is how to handle clause to know when a CI runner is done successfully then a call to the script to trigger removal of the vhost.
I setup jobs to run only when pushing/merging to branch "dev", but I also want it so I'm able to run them if I trigger that pipeline manually. Something like this:
test:
stage: test
<this step should be run always>
build:
stage: build
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
- if: <also run if the pipeline was run manually, but skip if it was triggered by something else>
This job is defined in a child "trigger" pipeline. This is how the parent looks like:
include:
- template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml'
stages:
- triggers
microservice_a:
stage: triggers
trigger:
include: microservice_a/.gitlab-ci.microservice_a.yml
strategy: depend
rules:
- changes:
- microservice_a/*
The effect I want to achieve is:
Run test in all cases
Run build in the child pipeline only when pushing/merging to "dev"
Also run the build job when the pipeline is run maually
Do not run the build job on any other cases (like a MR)
The rules examples showcase:
job:
script: echo "Hello, Rules!"
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
when: manual
allow_failure: true
- if: '$CI_PIPELINE_SOURCE == "schedule"'
The when:manual should be enough in your case: it does require that a job doesn’t run unless a user starts it.
Bonus question: This job is defined in a child "trigger" pipeline
Then it is related to gitlab-org/gitlab issue 201938, which is supposed to be fixed with GitLab 13.5 (Oct. 2020), but that only allow manual actions for parent-child pipeline (illustrated by this thread)
Double-check the environment variables as set in your child job
echo $CI_JOB_MANUAL
If true, that would indicate a job part of a manual triggered job.
While issue 22448 ("$CI_JOB_MANUAL should be set in all dependent jobs") points to this option not working, it includes a workaround.
I need a task that I can either manually execute or is run automatically on nightly schedule. I found this solution:
rules:
- changes:
- scheduled
when: always
- when: manual
The problem with the solution is that when a new pipeline is created and the task isn't run the pipeline is stuck in blocked state until I run it manually.
To avoid this I found a suggested workaround to add:
allow_failure: true
But this again brings a problem that if the task fails on nigtly run then the pipeline doesent fail and I don't get e-mail notifications.
Is there a way to solve this?
You can set allow_failure conditionally using rules:. So, instead of setting the allow_faulure: key on the job, set it in any rule that causes the job to be 'manual'.
rules:
- changes:
- scheduled
when: always
- when: manual
allow_failure: true
Also, based on your description it would probably be best to use a rule like so:
myjob:
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
when: always
allow_failure: false
- when: manual
allow_failure: true
Another alternative to prevent your pipeline being blocked by this job would be to have it run in the .post stage and use needs: [] to have it run immediately. That way, it'll never cause other jobs to wait on it.
myjob:
needs: []
stage: .post
# ...
There might be a better solution, but another workaround could be to have two separate jobs with the rules in and template out everything else. For example:
.build_template: &build_template
image: ubuntu:18.04
script:
- echo "hello world"
build_manual:
<<: *build_template
when: manual
except:
- schedules
build_nightly:
<<: *build_template
only:
- schedules
I can't find a satisfying solution for my case.
I want to start a job manually only when a certain previous job has failed. The job in question dose a validation. I want to make the next job manual so that the user acknowledges that something wasn't good and make him investigate the problem and continue only if he deems that the fail can be ignored.
stages:
- test
- validate
- build
lint:
stage: test
allow_failure: true
script:
- npm run lint
check:reducer:
stage: test
allow_failure: true
script:
- chmod +x ./check-reducers.py
- ./check-reducers.py $CI_PROJECT_ID $CI_COMMIT_BRANCH
except:
- master
- development
fail:pause:
stage: validate
allow_failure: true
script:
- echo The 'validate:reducer' job has failed
- echo Check the job and decide if this should continue
when: manual
needs: ["check:reducer"]
build:
stage: build
script:
- cp --recursive _meta/ $BUILD_PATH
- npm run build
artifacts:
name: "build"
expire_in: 1 week
paths:
- $BUILD_PATH
needs: ["fail:pause"]
I would like that if check:reducer fails, fail:pause to wait for the user input. If check:reducer exits with 0, fail:pause should start automatically or buildshould start.
Unfortunately, this isn't possible as the when keyword is evaluated at the very start of the pipeline (I.e., before any job execution has run), so you cannot set the when condition based on the previous job status.
This is possible if you use a generated gitlab-ci.yml as a child workflow.
stages:
- test
- approve
- deploy
generate-config:
stage: test
script:
- ./bin/run-tests.sh
- ./bin/generate-workflows.sh $?
artifacts:
paths:
- deploy-gitlab-ci.yml
trigger-workflows:
stage: deploy
trigger:
include:
- artifact: deploy-gitlab-ci.yml
job: generate-config
The generate-workflows.sh script writes out a deploy-gitlab-ci.yml that either has the approval job or not based on the return code of the run-test.sh passed as the first argument to the script.
You can make it easier on yourself using includes, where you either include the approve step or not in the generated deploy-gitlab-ci.yml file, and make the steps in the deploy optionally need the approal.
approve-gitlab-ci.yml
approve:
stage: approve
when: manual
script:
- echo "Approved!"
deploy-gitlab-ci.yml
deploy:
stage: deploy
needs:
- job: approve
- optional: true
Then the deploy-gitlab-ci.yml is simply an includes with the jobs to run:
includes:
- approve-gitlab-ci.yml
- deploy-gitlab-ci.yml
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE