of course I am aware, that single jobs run in sequence FIFO if these jobs are pointing to a single runner, so they have to wait.
That is fine.
For a test environment, where we can only deploy & test one version at a time, we need not only single jobs to be queryed, but whole pipelines.
To be clear: If we have Job_Deploy & Job_Test which are Part of a Pipeline Pipeline, we need to wait for the whole pipeline to be done.
By now we have the scenario, that the sequence can possible not be in order:
Job_Deploy1, Job_Deploy2, Job_Test2, Job_Test1 may be the case.
But we need it strict FIFO
Pipeline1(Job_Deploy1, Job_Test1), Pipeline(Job_Deploy2, Job_Test2)
How can we achieve this?
Why do we seem to be alone with this requirement? Do we have any wrong perceptions here? Is it best practice? If not: why?
Best regards
Try Edit an existing resource group with process_mode oldest_first
https://docs.gitlab.com/ee/api/resource_groups.html#edit-an-existing-resource-group
It seems like i misread the question and my initial answer is not applicable, i leave it at the end.
Right answer:
Use resource_group to create a resource group that ensures a job is mutually exclusive across different pipelines for the same project.
There are are so called resource_groups which you can use to force a certain order. and which ensures that jobs only run after each other even if they are in separate pipelines.
https://docs.gitlab.com/ee/ci/yaml/#resource_group
Wrong answer:
What you are looking for is the needs directive.
It offers you the possibility to express job dependencies within your ci file.
Something like
deploy1:
script: echo "deploy 1"
test1:
script: echo "test 1"
needs:
- deploy1
This means that test1 even if it is in the same stage, will not start as long as deploy1 has finished. Furthermore, you could also add needs: [] to deploy1 to start it immediately even if it is in a later stage. needs is powerful and allows you to work outside the stage boundaries.
Take a look at https://docs.gitlab.com/ee/ci/yaml/#needs
Related
We are using GitLab and GitLab CI which has a build, test and package step. We have written a script which checks if a user has done some house keeping for the feature they have added. The script returns a 1 or 0 depending on if the house keeping is done. At the moment the script goes in the build step of the GitLab CI, which means the build fails unless the house keeping is done. This is good, as the developer wont be able to merge until the house keeping is done. But, it would be better if the build and tests would continue to run, but the user was simply blocked from completing the merge request. E.g. by adding some automated "approve" step that is approved when the script passes.
Is there an obvious, easy or best way to do this?
Thanks
Set the job to use the builtin .post stage and use needs: []. This will cause the job to (1) run immediately with no delay and (2) will not stop your build/test or any other jobs from starting because it is in the last stage and (3) the failure of the job will still block the merge (provided you have the require pipelines to succeed setting enabled)
housekeeping:
stage: .post # always the last stage in any pipeline
needs: [] # start immediately with no dependencies
script: "..."
# ...
You could have this check as the very last step in the CI, this way everything else would go through and only this would fail.
Another way is marking all the other jobs as when: always instead of only running them when the previous jobs where successful.
In GitLab CI, is it possible to keep the last pipeline status when no jobs are queued upon a push? I have a changes rule setup like this in my .gitlab-ci.yml:
changes:
- Assets/*
- Packages/*
- ProjectSettings/*
- .gitlab-ci.yml
which applies to all jobs in the pipeline (these are build jobs for Unity, though irrelevant). NOTE: I only wanted to run a build job if there are any actual files changes that would require a rebuild. changes to README.md and CONTRIBUTING.md are not changes that require a rebuild so this is why I have such a rule.
Problem is I require successful pipeline to merge branches and when I try to merge a branch that modified README.md there obviously is no pipeline.
Is there a way to just "reuse" the result of a previous pipeline or to have a "dummy" job that succeeds instantly upon any push, so as to be able to merge this branch without requiring an expensive rebuild of the whole project?
As you mentioned in your last paragraph, the only way to work around this would be to inject a dummy job that always succeed; something like echo "hello world" in the script.
However, depending on how long your tests run, your best bet may be to just have your tests run every time regardless of changes. Any sort of partial pipeline run using the changes keyword leaves you open to merging changes that break your pipeline. It essentially tightly couples your logic in your pipeline to the component structure of your code, which isn't always a good thing since one of the points of your pipeline is to catch those kinds of cross-component breakages.
I have this gitlab-ci job and I would like it to just ignore failures and keep going. Do you have a way of doing that? Note that allow_fail: true does not work because it will just ignore that the job have failed however I want that the job keep executing in spite of failing commands in the middle.
palms up, serious look: "We don't do that here"
The pipeline is supposed to work every time, and by design its commands cannot fail. You can however:
change the commands logic and avoid failure
split the commands in different jobs, using the on_failure parameter to manage workflow
force the commands to have a clean exit code (ie: using || true after the fallible command)
During debug I often use the third option after debug statement, or after commands that I'm not sure how will behave. The definitive version, however, is supposed to always work.
Because "reasons", we know that when we use azureml-sdk's HyperDriveStep we expect a number of HyperDrive runs to fail -- normally around 20%. How can we handle this without failing the entire HyperDriveStep (and then all downstream steps)? Below is an example of the pipeline.
I thought there would be an HyperDriveRunConfig param to allow for this, but it doesn't seem to exist. Perhaps this is controlled on the Pipeline itself with the continue_on_step_failure param?
The workaround we're considering is to catch the failed run within our train.py script and manually log the primary_metric as zero.
thanks for your question.
I'm assuming that HyperDriveStep is one of the steps in your Pipeline and that you want the remaining Pipeline steps to continue, when HyperDriveStep fails, is that correct?
Enabling continue_on_step_failure, should allow the rest of the pipeline steps to continue, when any single steps fails.
Additionally, the HyperDrive run consists of multiple child runs, controlled by the HyperDriveConfig. If the first 3 child runs explored by HyperDrive fail (e.g. with user script errors), the system automatically cancels the entire HyperDrive run, in order to avoid further wasting resources.
Are you looking to continue other Pipeline steps when the HyperDriveStep fails? or are you looking to continue other child runs within the HyperDrive run, when the first 3 child runs fail?
Thanks!
I have an Azure Pipelines pipeline defined by a YAML file that compiles, runs some tests, and then publishes test results. It's clearly impossible to run tests without compiling, so the compilation task obviously has the continueOnError: false set. However, I would still like to publish the test results when the tests fail, so I set continueOnError to true under the testing task.
This seemed like it worked, until one of my tests failed. Then, instead of failing the build, Azure just reported a warning. How can I get it to still error the entire build but also execute the remaining tasks?
UPDATE:
Microsoft has added some new documentation since I posted this answer that explains it better than I have:
A given task or job can't unilaterally decide whether the job/stage continues. What it can do is offer a status of succeeded or failed, and downstream tasks/jobs each have a condition computation that lets them decide whether to run or not. The default condition which is effectively "run if we're in a successful state".
Continue on error alters this in a subtle way. It effectively "tricks" all downstream steps/jobs into treating any result as "success" for the purposes of making that decision. Or to put it another way, it says "don't consider the failure of this task when you're making a decision about the condition of the containing structure".
ORIGINAL ANSWER:
I think it's useful to clear up a misunderstanding about exactly what a couple of the YAML properties do (because this got me too). The question asked about a YAML pipeline which the existing answer didn't cover so I thought I'd give an example of that too.
continueOnError determines whether the current task continues if it encounters an error (with a warning), or if it fails straight away. Despite the name being potentially misleading, it does not (directly) determine whether execution continues to subsequent tasks or not.
condition decides whether a task runs or not. By default, if a previous task failed, then this one will not run. You can override this and have tasks run regardless of earlier failures.
Therefore, it is not necessary to use continueOnError if your tests fail, just in order for the Publish Test Results task to run, you can have it run anyway.
I don't know exactly how your pipeline is structured, but hopefully this demonstrates an example:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: 'echo "compiling....."; exit 0'
displayName: 'Compile'
- script: 'echo "testing....." ; exit 1' # exit 1" simulates task failure
displayName: 'Run Tests'
condition: succeeded()
- script: 'echo "publishing test results....."'
displayName: 'Publish Results'
condition: succeededOrFailed() # Run task even if previous ones fail
The result is that the overall pipeline is failed:
And we can see the test results are still published in the pipeline breakdown (despite a previous step failing, there is no need here for continueOnError):
You can play around with the exit code in the YAML example to determine which tasks fail. I found the documentation of exactly what can go in the condition field to be a bit poor, but specifically I'd refer you to Conditions, Expressions, and Expressions#Job status check functions.
You can set the Publish test results task's Control options as below.
Then you will get the result as this.
If your compilation task always run successful, you can use the default conditions. If not, I think you can use custom conditions. For example, add a task to create a variable after your comilation task, then use the variable's value as the publish test results conditions, like and(succeeded(), eq(variables['variableName'], 'variableValue'))
You can specify the conditions under which the task or job will run. More detailed information, you can refer to here.