We are using GitLab and GitLab CI which has a build, test and package step. We have written a script which checks if a user has done some house keeping for the feature they have added. The script returns a 1 or 0 depending on if the house keeping is done. At the moment the script goes in the build step of the GitLab CI, which means the build fails unless the house keeping is done. This is good, as the developer wont be able to merge until the house keeping is done. But, it would be better if the build and tests would continue to run, but the user was simply blocked from completing the merge request. E.g. by adding some automated "approve" step that is approved when the script passes.
Is there an obvious, easy or best way to do this?
Thanks
Set the job to use the builtin .post stage and use needs: []. This will cause the job to (1) run immediately with no delay and (2) will not stop your build/test or any other jobs from starting because it is in the last stage and (3) the failure of the job will still block the merge (provided you have the require pipelines to succeed setting enabled)
housekeeping:
stage: .post # always the last stage in any pipeline
needs: [] # start immediately with no dependencies
script: "..."
# ...
You could have this check as the very last step in the CI, this way everything else would go through and only this would fail.
Another way is marking all the other jobs as when: always instead of only running them when the previous jobs where successful.
Related
In GitLab CI, is it possible to keep the last pipeline status when no jobs are queued upon a push? I have a changes rule setup like this in my .gitlab-ci.yml:
changes:
- Assets/*
- Packages/*
- ProjectSettings/*
- .gitlab-ci.yml
which applies to all jobs in the pipeline (these are build jobs for Unity, though irrelevant). NOTE: I only wanted to run a build job if there are any actual files changes that would require a rebuild. changes to README.md and CONTRIBUTING.md are not changes that require a rebuild so this is why I have such a rule.
Problem is I require successful pipeline to merge branches and when I try to merge a branch that modified README.md there obviously is no pipeline.
Is there a way to just "reuse" the result of a previous pipeline or to have a "dummy" job that succeeds instantly upon any push, so as to be able to merge this branch without requiring an expensive rebuild of the whole project?
As you mentioned in your last paragraph, the only way to work around this would be to inject a dummy job that always succeed; something like echo "hello world" in the script.
However, depending on how long your tests run, your best bet may be to just have your tests run every time regardless of changes. Any sort of partial pipeline run using the changes keyword leaves you open to merging changes that break your pipeline. It essentially tightly couples your logic in your pipeline to the component structure of your code, which isn't always a good thing since one of the points of your pipeline is to catch those kinds of cross-component breakages.
I would like two jobs to run in parallel in Gitlab CI but I want to launch them manually.
Basically I have multiple jobs needed to deploy my code to staging (build the server, build the frontend). I want to deploy to staging by clicking ONE button (hence making the jobs manual) but I want both of them to run at the same time in parallel (faster). I don't want both jobs to be manual because it displays two buttons in the Gitlab UI and someone could deploy only one part.
I tried creating a third, empty job, that would be manual, and have the real deploy jobs run automatically when the empty job has completed. However, even an empty job with just an “echo” takes > 30 seconds and I feel this is dumb. Here is my current solution that does not convince me:
start-release:
stage: build-and-deploy-to-staging
rules:
- when: manual
script:
- echo "Starting staging deployment"
release-api-staging:
stage: build-and-deploy-to-staging
script:
- "ENV_NAME=staging STEP=release_api release/deploy.sh"
when: on_success
needs:
["start-release-staging"]
release-frontend-staging:
stage: build-and-deploy-to-staging
script:
- "ENV_NAME=staging STEP=release_frontend release/deploy.sh"
when: on_success
needs:
["start-release-staging"]
Do you have any idea how I could manage this?
Thanks a lot!
To follow up from the comments, if you were running your own Runners, you can check the pull-policy for the Runner. There are a few values that tell the Runner what to do when trying to find a Docker image for the job.
The first option is always, which means don't look for any local images, and pull from the available repositories every time the Runner processes a job.
The second is if-not-present, which instructs the Runner to use a local image if one is present, otherwise pull from available repositories.
The last is never, which means to never pull from available repositories and only ever look locally. In this case, if an image isn't available locally, any job using that image will fail.
Knowing this, changing the pull policy to if-not-present or even never if you can maintain the available images on the Runner host(s), can greatly speed up your pipelines.
Another factor that can speed up a job that doesn't require any specific software is to use the smallest image possible that still meets any requirements. For your example .yml file, if you were to use an image such as alpine, pulling the image would be much faster as it is considerably smaller than other images (the image's description claims it's <5mb).
As for the pipeline itself, if you define both jobs in the same stage, with the same dependencies/needs, and mark them both as manual, there will be a single button above the Stage label on the Pipeline view to run all manual jobs in that stage, at the same time (as long as there are enough runners).
job1:
stage: x
when: manual
script:
- ../do_something.sh
job2:
stage: x
when: manual
...
If all jobs in a stage are manual, you can start them all at once with the 'play' button for the Stage.
Otherwise, you could define it like you have in your question but the first job will need to be in a different stage. Per the docs, the needs keyword will only work if the "needed" jobs are in another stage.
stages:
- kickoff_build_and_deploy
- build_and_deploy
kickoff_job:
when: manual
image: alpine:latest
script:
- echo '' > /dev/null
build_deploy1:
needs: ['kickoff_job']
script:
- ./do_something.sh
build_deploy2:
needs: ['kickoff_job']
script:
- ./do_something.sh
In this example, the build_deploy* jobs will run as soon as kickoff_job finishes and runners are available.
I have this gitlab-ci job and I would like it to just ignore failures and keep going. Do you have a way of doing that? Note that allow_fail: true does not work because it will just ignore that the job have failed however I want that the job keep executing in spite of failing commands in the middle.
palms up, serious look: "We don't do that here"
The pipeline is supposed to work every time, and by design its commands cannot fail. You can however:
change the commands logic and avoid failure
split the commands in different jobs, using the on_failure parameter to manage workflow
force the commands to have a clean exit code (ie: using || true after the fallible command)
During debug I often use the third option after debug statement, or after commands that I'm not sure how will behave. The definitive version, however, is supposed to always work.
Is there a way fail my entire pipeline if the user triggered the pipeline and didn't set a variable called options with a value?
I've tried things like only and rules but they just skip the job instead of failing all jobs.
Yes, though the way you fail would be dependent on the system your runner's based on.
For example, in a Linux/bash based runner, all you need is exit 1 (as opposed to exit 0) to stop execution and fail the pipeline
I have an Azure Pipelines pipeline defined by a YAML file that compiles, runs some tests, and then publishes test results. It's clearly impossible to run tests without compiling, so the compilation task obviously has the continueOnError: false set. However, I would still like to publish the test results when the tests fail, so I set continueOnError to true under the testing task.
This seemed like it worked, until one of my tests failed. Then, instead of failing the build, Azure just reported a warning. How can I get it to still error the entire build but also execute the remaining tasks?
UPDATE:
Microsoft has added some new documentation since I posted this answer that explains it better than I have:
A given task or job can't unilaterally decide whether the job/stage continues. What it can do is offer a status of succeeded or failed, and downstream tasks/jobs each have a condition computation that lets them decide whether to run or not. The default condition which is effectively "run if we're in a successful state".
Continue on error alters this in a subtle way. It effectively "tricks" all downstream steps/jobs into treating any result as "success" for the purposes of making that decision. Or to put it another way, it says "don't consider the failure of this task when you're making a decision about the condition of the containing structure".
ORIGINAL ANSWER:
I think it's useful to clear up a misunderstanding about exactly what a couple of the YAML properties do (because this got me too). The question asked about a YAML pipeline which the existing answer didn't cover so I thought I'd give an example of that too.
continueOnError determines whether the current task continues if it encounters an error (with a warning), or if it fails straight away. Despite the name being potentially misleading, it does not (directly) determine whether execution continues to subsequent tasks or not.
condition decides whether a task runs or not. By default, if a previous task failed, then this one will not run. You can override this and have tasks run regardless of earlier failures.
Therefore, it is not necessary to use continueOnError if your tests fail, just in order for the Publish Test Results task to run, you can have it run anyway.
I don't know exactly how your pipeline is structured, but hopefully this demonstrates an example:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: 'echo "compiling....."; exit 0'
displayName: 'Compile'
- script: 'echo "testing....." ; exit 1' # exit 1" simulates task failure
displayName: 'Run Tests'
condition: succeeded()
- script: 'echo "publishing test results....."'
displayName: 'Publish Results'
condition: succeededOrFailed() # Run task even if previous ones fail
The result is that the overall pipeline is failed:
And we can see the test results are still published in the pipeline breakdown (despite a previous step failing, there is no need here for continueOnError):
You can play around with the exit code in the YAML example to determine which tasks fail. I found the documentation of exactly what can go in the condition field to be a bit poor, but specifically I'd refer you to Conditions, Expressions, and Expressions#Job status check functions.
You can set the Publish test results task's Control options as below.
Then you will get the result as this.
If your compilation task always run successful, you can use the default conditions. If not, I think you can use custom conditions. For example, add a task to create a variable after your comilation task, then use the variable's value as the publish test results conditions, like and(succeeded(), eq(variables['variableName'], 'variableValue'))
You can specify the conditions under which the task or job will run. More detailed information, you can refer to here.