I have a few independent scheduled CI jobs.
Check that the infrastructure matches Terraform code.
Check for npm vulnerabilities.
Check that external APIs pass tests.
These are not hermetic. They do not test the fitness of the code in isolation. They could succeed at commit 12345 and then fail at commit 12345 and then succeed again.
I run these daily.
Gitlab lacks the ability to have multiple pipeline types (unlike, say, Buildkite), so I use a variable to control which steps run.
However, I am left with the problem that these checks interfere with the main passed/failed status of commits.
For example, if the Terraform infra check fails, then it's broken and people are notified and whoever is the next to push is informed they fixed it.
These kinds of checks can't be that uncommon, right? How should thse be managed?
It sounds like you're looking for the allow_failure keyword.
https://docs.gitlab.com/ce/ci/yaml/#allow_failure
When allow_failure is set to true and the job fails, the job shows an orange warning in the UI. However, the logical flow of the pipeline considers the job a success/passed, and is not blocked.
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
Related
I was wondering and could not find a solution to our problem.
We have many CI pipelines running in scheduled times and we need to add a job in case of early job failure.
For example lets say that in the attached picture job "deploy-job1" failed (not like in the picture)
We want to have a "sleeping job" that will be activated and run only when a previous job did not succeed.
Gitlab pipeline
Are there any suggestions on a way to handle this kind of a task?
We have tried handle this within the scripts we are running but we want to have general "sleeping job" that will be similar to all stages
Something like this might help:
.sleeping_job:
needs: deploy-job1
when: on_failure
# do stuff
Is there a way of retriggering a successful child pipeline in gitlab ? I don't see any retry button in trigger job, which we otherwise see in normal pipline job. Is there a way/workaround to get an option to do so ?
I went through the gitlab doc, it only talks about retrying failed jobs in child pipeline.
That is currently (Q4 2022) not supported yet.
(And retry: is indeed only for failed job)
It is requested by issue 29456:
Ability to rerun a successful pipeline via "Retry" button
Not only failed pipelines sometimes need a rerun but also successful ones:
If your tests are unreliable and you are sceptical that the test success is repeatable
If your jobs depend on outside factors
If your job depends on some predefined CI variable which can change without a code change
So in general, a pipeline should show the retry button even in case of a success. Then, all jobs should be retried again
The currently suggested workaround of CI / CD -> Pipelines -> Run Pipeline does not always work, especially not for merge request pipelines.
In my case, I have all jobs defined as only: merge_requests and "Run Pipeline" responds with the error "No stages / jobs for this pipeline"
Basically I currently have 5 test licenses on a server. I have a pipeline that runs test scripts when I trigger it manually. It connects to the license server and acquires a floating license . This approach is good for now but soon I will be wanting to expand this so that when the application needs testing, I can run multiple pipelines in parallel to have multiple tests running. Here's the thing, I'm wanting to only sometimes run them in parallel based on what I need to test and I would want to manually trigger each pipeline. For example, one day I might only want to run the tests scripts on one job so this requires one pipeline running. Another day I might want to run 3 jobs at the same time or another day I may want to run 5 jobs throughout the day but may overlap with a running pipeline so it needs to run its own pipeline.
My question is, how do I go about setting this up in a gitlab yml file.
If anyone could also provide a basic example that would be helpful!
As a general rule, any job within the same stage is run in parallel. Within those jobs, you can define rules that specify when a given job runs. Note that needing to manually kick off jobs will cause your overall pipeline to be listed as "blocked" when the manual jobs are reached. Similarly, you'll need to set some jobs as allow_failure: true or they will block the next stage from executing. Example pipeline:
stages:
- build
- test
build_job:
image: alpine:latest
stage: build
script:
- echo "hello world"
test_job_1:
image: alpine:latest
stage: test
rules:
- when: manual
script:
- echo "test 1"
test_job_2:
image: alpine:latest
stage: test
rules:
- when: manual
script:
- echo "test 2"
When you run the above pipeline, you will have to manually click "play" on both jobs to start them.
I will note though, that this feels like an anti-pattern. It sounds like you want to conserve your 5 test licenses to make sure that you don't, for example, have 6 jobs running and have a job fail due to no free licenses. If so, this is one of the exact use-cases that the resource_group keyword is meant to address. Jobs added to the same resource group do not run concurrently (even if they are in different pipelines), so you could have a test_1 resource group, test_2 resource group, etc, and those jobs would always run in parallel automatically, but would never run more than one instance of that job at once even across different pipelines. This allows you ensure you only use 5 licenses, but your tests still run automatically and you don't need to manually trigger them, which also allows you to more easily define downstream jobs to run when your tests pass.
I use Gitlab CI to deploy my service. Currently I have a single deploy job that releases my changes to every server behind a load balancer.
I would like do a phased rollout where I deploy to one server in my load balancer, give it a few minutes to bake and set off any alarms if there is an issue, and then automatically continue deploying to the remaining servers. If any issue occurred before the delayed full automatic deploy happened I would manually cancel that job to prevent the bad change from going out more widely.
With this goal in mind I configured my pipeline with the following .gitlab-ci.yml:
stages:
- canary_deploy
- full_deploy
canary:
stage: canary_deploy
allow_failure: false
when: manual
script: make deploy-canary
full:
stage: full_deploy
when: delayed
start_in: 10 minutes
script: make deploy-full
This works relatively well but I ran into a problem when I tried to push a critical change out quickly. The canary deploy script was hanging and this prevented the second job from starting as it must wait for the first stage to complete. In this case I would have preferred to skip the canary entirely but because of the way the pipeline is configured it was not possible to manually invoke the full deploy.
Ideally I would like the full_deploy stage to run on the typical delay but allow me to forcefully start it if I didn't want to wait. I've reviewed the rules and needs and when configuration options hoping to find a way to achieve my goal but I haven't been able to find a working solution.
Some things I've tried, without luck:
I could create a duplicate full_deploy job which is manual and does not depend on the canary_deploy stage but it feels a bit hacky. And in reality my configuration is a bit more complex than what I've distilled here so there are actually several region-specific deploy jobs and I would prefer not to have to duplicate each of them.
I tried to use rules to consider the status of the prior stage and make the full_deploy manual unless the prior stage was successful. This isn't possible because rules are executed on pipeline creation and cannot dynamically adjust this property at runtime.
I changed the canary_deploy to allow failure, which effectively unblocked the second stage immediately. The problem here is that it caused the delay timer to start counting down immediately upon pipeline creation rather than waiting for the first stage to complete.
One thing you could do to make duplicating the full_deploy job feel a little bit less "hacky" is to define it once and then use extends two times:
stages:
- canary_deploy
- full_deploy
.full:
script: make deploy-full
canary:
stage: canary_deploy
allow_failure: false
when: manual
script: make deploy-canary
full_automatic:
extends: .full
stage: full_deploy
when: delayed
start_in: 10 minutes
full_manual:
stage: full_deploy
extends: .full
when: manual
needs: []
This way, you only need to define the scripts section once and both the full_manual and the full_automatic job use it. When running the pipeline, you can choose which job to run first (manual versus canary):
Screenshot of the GitLab UI for selecting which job to run
By specifying needs: [], you tell GitLab that the full_manual job does not depend on any other jobs and can be executed immediately without running jobs from canary_deploy before.
When executing full_manual, the canary job is not executed:
Overview of executed pipeline jobs
I have set up a PR Pipeline in Azure. As part of this pipeline I run a number of regression tests. These run against a regression test database - we have to clear out the database at the start of the tests so we are certain what data is in there and what should come out of it.
This is all working fine until the pipeline runs multiple times in parallel - then the regression database is being written to multiple times and the data returned from it is not what is expected.
How can I stop a pipeline running in parallel - I've tried Google but can't find exactly what I'm looking for.
If the pipeline is running, the the next build should wait (not for all pipelines - I want to set it on a single pipeline), is this possible?
Depending on your exact use case, you may be able to control this with the right trigger configuration.
In my case, I had a pipeline scheduled to kick off every time a Pull Request is merged to the main branch in Azure. The pipeline deployed the code to a server and kicked off a suite of tests. Sometimes, when two merges occurred just minutes apart, the builds would fail due to a shared resource that required synchronisation being used.
I fixed it by Batching CI Runs
I changed my basic config
trigger:
- main
to use the more verbose syntax allowing me to turn batching on
trigger:
batch: true
branches:
include:
- main
With this in place, a new build will only be triggered for main once the previous one has finished, no matter how many commits are added to the branch in the meantime.
That way, I avoid having too many builds being kicked off and I can still use multiple agents where needed.
One way to solve this is to model your test regression database as an "environment" in your pipeline, then use the "Exclusive Lock" check to prevent concurrent "deployment" to that "environment".
Unfortunately this approach comes with several disadvantages inherent to "environments" in YAML pipelines:
you must set up the check manually in the UI, it's not controlled in source code.
it will only prevent that particular deployment job from running concurrently, not an entire pipeline.
the fake "environment" you create will appear in alongside all other environments, cluttering the environment view if you happen to use environments for "real" deployments. This is made worse by this view being a big sack of all environments, there's no grouping or hierarchy.
Overall the initial YAML reimplementation of Azure Pipelines mostly ignored the concepts of releases, deployments, environments. A few piecemeal and low-effort aspects have subsequently been patched in, but without any real overarching design or apparent plan to get to parity with the old release pipelines.
You can use "Trigger Azure DevOps Pipeline" extension by Maik van der Gaag.
It needs to add to you DevOps and configure end of the main pipeline and point to your test pipeline.
Can find more details on Maik's blog.
According to your description, you could use your own self-host agent.
Simply deploy your own self-hosted agents.
Just need to make sure your self host agent environment is the same as your local development environment.
Under this situation, since your agent pool only have one available build agent. When multiple builds triggered, only one build will be running simultaneously. Others will stay in queue with a specific order for agents. Unless the prior build finished, it will not run with next build.
For other pipeline, just need to keep use the host agent pool.