At the moment, I have a couple of CI/CD jobs configured in gitlab-ci.yaml within a pipeline. For instance, I have the build and deploy configured within one branch. One of the jobs is creating a backup.
I would like to schedule only the backup job every night. However, in the schedule section of GitLab CI/CD, I can only select a branch or a tag. This will result in all the stages/jobs to be triggered, whereas I only want the backup to be triggered.
I've seen the option to configure rules and excepts to only trigger the jobs in a certain context. That, however, feels a bit like a patch solution. Is there another way to trigger only a single job in the Gitlab CI/CD scheduler?
I've seen the option to configure rules and excepts to only trigger the jobs in a certain context. That, however, feels a bit like a patch solution. Is there another way to trigger only a single job in the Gitlab CI/CD scheduler?
No, there's no other way than using rules: or only:/except: to control which jobs run when the pipeline is triggered by a schedule.
You can make this more manageable by splitting your pipeline configuration to multiple files then using the appropriate rules: on the include:
include:
- local: /do-backup.yml
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- local: /do-pipeline.yml
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule"'
That way you don't have to necessarily apply the same rules: to every single job and you can define additional rules: per job with less conflict.
Related
I have a repository that has multiple microservices in it. As part of that, each service has its own OpenAPI 3.0 spec.
There is only one .gitlab-ci.yaml file in the repository, but it is not clear in the existing documentation on how to have separate configurations for each service.
Has anyone done this before or have a suggestion for a work around?
The logic for running the Fuzz Testing is configured within the API-Fuzzing.gitlab-ci.yml template, and you can view the included logic within that template here: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml
Within that template, you can see that there is one job included: apifuzzer_fuzz. Each job can run one fuzz test, so if you want to run multiple fuzz tests for multiple services, you can simply create new jobs using the extends: apifuzzer_fuzz keyword in your .gitlab-ci.yml file. You can pass each job it's own variables to control what is being tested. If you want, you can turn off the existing job as well, to making naming easier to read. Example:
include:
template: API-Fuzzing.gitlab-ci.yml
# disable the existing job so that names make more sense
apifuzzer_fuzz:
rules:
- when: never
fuzz_microservice_one:
extends: apifuzzer_fuzz
variables:
FUZZAPI_TARGET_URL: https://my-microservice-1.com
FUZZAPI_OPENAPI: my-spec.json
fuzz_microservice_two:
extends: apifuzzer_fuzz
variables:
FUZZAPI_TARGET_URL: https://my-microservice-2.com
FUZZAPI_OPENAPI: my-spec-two.json
This can be repeated as many times as needed. Note that gitlab will combine all the security findings from the fuzz scanners into one report, which may make the results a bit interesting to read. GitLab sort of expects that if you have truly separate services to scan that you'll have them in different repositories, so any scans run of a given type are combined into the same report within an MR or on the security dashboard.
Basically I currently have 5 test licenses on a server. I have a pipeline that runs test scripts when I trigger it manually. It connects to the license server and acquires a floating license . This approach is good for now but soon I will be wanting to expand this so that when the application needs testing, I can run multiple pipelines in parallel to have multiple tests running. Here's the thing, I'm wanting to only sometimes run them in parallel based on what I need to test and I would want to manually trigger each pipeline. For example, one day I might only want to run the tests scripts on one job so this requires one pipeline running. Another day I might want to run 3 jobs at the same time or another day I may want to run 5 jobs throughout the day but may overlap with a running pipeline so it needs to run its own pipeline.
My question is, how do I go about setting this up in a gitlab yml file.
If anyone could also provide a basic example that would be helpful!
As a general rule, any job within the same stage is run in parallel. Within those jobs, you can define rules that specify when a given job runs. Note that needing to manually kick off jobs will cause your overall pipeline to be listed as "blocked" when the manual jobs are reached. Similarly, you'll need to set some jobs as allow_failure: true or they will block the next stage from executing. Example pipeline:
stages:
- build
- test
build_job:
image: alpine:latest
stage: build
script:
- echo "hello world"
test_job_1:
image: alpine:latest
stage: test
rules:
- when: manual
script:
- echo "test 1"
test_job_2:
image: alpine:latest
stage: test
rules:
- when: manual
script:
- echo "test 2"
When you run the above pipeline, you will have to manually click "play" on both jobs to start them.
I will note though, that this feels like an anti-pattern. It sounds like you want to conserve your 5 test licenses to make sure that you don't, for example, have 6 jobs running and have a job fail due to no free licenses. If so, this is one of the exact use-cases that the resource_group keyword is meant to address. Jobs added to the same resource group do not run concurrently (even if they are in different pipelines), so you could have a test_1 resource group, test_2 resource group, etc, and those jobs would always run in parallel automatically, but would never run more than one instance of that job at once even across different pipelines. This allows you ensure you only use 5 licenses, but your tests still run automatically and you don't need to manually trigger them, which also allows you to more easily define downstream jobs to run when your tests pass.
I have a few independent scheduled CI jobs.
Check that the infrastructure matches Terraform code.
Check for npm vulnerabilities.
Check that external APIs pass tests.
These are not hermetic. They do not test the fitness of the code in isolation. They could succeed at commit 12345 and then fail at commit 12345 and then succeed again.
I run these daily.
Gitlab lacks the ability to have multiple pipeline types (unlike, say, Buildkite), so I use a variable to control which steps run.
However, I am left with the problem that these checks interfere with the main passed/failed status of commits.
For example, if the Terraform infra check fails, then it's broken and people are notified and whoever is the next to push is informed they fixed it.
These kinds of checks can't be that uncommon, right? How should thse be managed?
It sounds like you're looking for the allow_failure keyword.
https://docs.gitlab.com/ce/ci/yaml/#allow_failure
When allow_failure is set to true and the job fails, the job shows an orange warning in the UI. However, the logical flow of the pipeline considers the job a success/passed, and is not blocked.
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
I use Gitlab CI to deploy my service. Currently I have a single deploy job that releases my changes to every server behind a load balancer.
I would like do a phased rollout where I deploy to one server in my load balancer, give it a few minutes to bake and set off any alarms if there is an issue, and then automatically continue deploying to the remaining servers. If any issue occurred before the delayed full automatic deploy happened I would manually cancel that job to prevent the bad change from going out more widely.
With this goal in mind I configured my pipeline with the following .gitlab-ci.yml:
stages:
- canary_deploy
- full_deploy
canary:
stage: canary_deploy
allow_failure: false
when: manual
script: make deploy-canary
full:
stage: full_deploy
when: delayed
start_in: 10 minutes
script: make deploy-full
This works relatively well but I ran into a problem when I tried to push a critical change out quickly. The canary deploy script was hanging and this prevented the second job from starting as it must wait for the first stage to complete. In this case I would have preferred to skip the canary entirely but because of the way the pipeline is configured it was not possible to manually invoke the full deploy.
Ideally I would like the full_deploy stage to run on the typical delay but allow me to forcefully start it if I didn't want to wait. I've reviewed the rules and needs and when configuration options hoping to find a way to achieve my goal but I haven't been able to find a working solution.
Some things I've tried, without luck:
I could create a duplicate full_deploy job which is manual and does not depend on the canary_deploy stage but it feels a bit hacky. And in reality my configuration is a bit more complex than what I've distilled here so there are actually several region-specific deploy jobs and I would prefer not to have to duplicate each of them.
I tried to use rules to consider the status of the prior stage and make the full_deploy manual unless the prior stage was successful. This isn't possible because rules are executed on pipeline creation and cannot dynamically adjust this property at runtime.
I changed the canary_deploy to allow failure, which effectively unblocked the second stage immediately. The problem here is that it caused the delay timer to start counting down immediately upon pipeline creation rather than waiting for the first stage to complete.
One thing you could do to make duplicating the full_deploy job feel a little bit less "hacky" is to define it once and then use extends two times:
stages:
- canary_deploy
- full_deploy
.full:
script: make deploy-full
canary:
stage: canary_deploy
allow_failure: false
when: manual
script: make deploy-canary
full_automatic:
extends: .full
stage: full_deploy
when: delayed
start_in: 10 minutes
full_manual:
stage: full_deploy
extends: .full
when: manual
needs: []
This way, you only need to define the scripts section once and both the full_manual and the full_automatic job use it. When running the pipeline, you can choose which job to run first (manual versus canary):
Screenshot of the GitLab UI for selecting which job to run
By specifying needs: [], you tell GitLab that the full_manual job does not depend on any other jobs and can be executed immediately without running jobs from canary_deploy before.
When executing full_manual, the canary job is not executed:
Overview of executed pipeline jobs
I have set up a PR Pipeline in Azure. As part of this pipeline I run a number of regression tests. These run against a regression test database - we have to clear out the database at the start of the tests so we are certain what data is in there and what should come out of it.
This is all working fine until the pipeline runs multiple times in parallel - then the regression database is being written to multiple times and the data returned from it is not what is expected.
How can I stop a pipeline running in parallel - I've tried Google but can't find exactly what I'm looking for.
If the pipeline is running, the the next build should wait (not for all pipelines - I want to set it on a single pipeline), is this possible?
Depending on your exact use case, you may be able to control this with the right trigger configuration.
In my case, I had a pipeline scheduled to kick off every time a Pull Request is merged to the main branch in Azure. The pipeline deployed the code to a server and kicked off a suite of tests. Sometimes, when two merges occurred just minutes apart, the builds would fail due to a shared resource that required synchronisation being used.
I fixed it by Batching CI Runs
I changed my basic config
trigger:
- main
to use the more verbose syntax allowing me to turn batching on
trigger:
batch: true
branches:
include:
- main
With this in place, a new build will only be triggered for main once the previous one has finished, no matter how many commits are added to the branch in the meantime.
That way, I avoid having too many builds being kicked off and I can still use multiple agents where needed.
One way to solve this is to model your test regression database as an "environment" in your pipeline, then use the "Exclusive Lock" check to prevent concurrent "deployment" to that "environment".
Unfortunately this approach comes with several disadvantages inherent to "environments" in YAML pipelines:
you must set up the check manually in the UI, it's not controlled in source code.
it will only prevent that particular deployment job from running concurrently, not an entire pipeline.
the fake "environment" you create will appear in alongside all other environments, cluttering the environment view if you happen to use environments for "real" deployments. This is made worse by this view being a big sack of all environments, there's no grouping or hierarchy.
Overall the initial YAML reimplementation of Azure Pipelines mostly ignored the concepts of releases, deployments, environments. A few piecemeal and low-effort aspects have subsequently been patched in, but without any real overarching design or apparent plan to get to parity with the old release pipelines.
You can use "Trigger Azure DevOps Pipeline" extension by Maik van der Gaag.
It needs to add to you DevOps and configure end of the main pipeline and point to your test pipeline.
Can find more details on Maik's blog.
According to your description, you could use your own self-host agent.
Simply deploy your own self-hosted agents.
Just need to make sure your self host agent environment is the same as your local development environment.
Under this situation, since your agent pool only have one available build agent. When multiple builds triggered, only one build will be running simultaneously. Others will stay in queue with a specific order for agents. Unless the prior build finished, it will not run with next build.
For other pipeline, just need to keep use the host agent pool.