I have a simple Azure Pipeline containing two stages:
Stage 1: Build
Stage 2: Run tests
Stage 1 builds the source code and pushes the binaries to Azure Artifacts. Stage 2 downloads the binaries of Stage 1 and runs multiple tests against them, using different jobs for different tests:
Job 1: Run tests for module A
Job 2: Run tests for module B
Job 3: Run tests for module C
These jobs are completely indepent of each other and run in parallel.
If all stages and jobs succeed, how can I manually trigger Job 3 in Stage 2 without triggering Stage 1 and Job 1 and 2 of Stage 2 again?
One ugly work around might be to use variables and somehow run a new pipeline, skip stage 1 if the variables are set and download the binaries from a previous pipeline. Are there better approaches?
Edit: There is a similar topic ongoing on ms dev community:
https://developercommunity.visualstudio.com/idea/697467/manually-triggered-stages-in-yaml-multi-stage-pipe.html?childToView=903723#comment-903723
You can do what you are after with multi-stage pipelines if you pull the individual Jobs that exist in your current Stage 2 up into their own independent Stages.
Once you do that, you can then manually trigger them independently for reruns.
Or if you had a need, you can also start a new run with only a subset of the stages.
If you define release pipeline in yaml , you can through adding condition: false to disable/skip a job in multiple-job pipeline.
- job: Foo
condition: false
For details , you can refer to this case.
Then you can choose which stages to run when running multi-stages pipeline ,through this, you can skip Stage1.
Related
Gitlab CI/CD
I want to run a job for the Merge Request that needs jar artefacts built during a job run in another pipelines (for commits).
I don't want to build them again because if takes too much time then the job will fail.
Can I do that?
I checked the
needs:
- project:
But that's only with Premium licences.
We are currently implementing Test Execution of Test Suites using Gitlab's CI.
One of the issue we encountered is that it keeps using a runner even though it is being used by another job.
Problem 1:
There are 3 available runners (different machines), config is already set to limit = 1 and request_concurrency = 1.
Pipeline with 3 jobs running in parallel will be triggered.
Result: The 2 jobs will use 1 runner.
Expectation: The 3 jobs will use the 3 runners (1 job per runner).
Problem 2:
Given the above conditions.
Pipeline with 3 jobs running in parallel will be triggered twice.
Result: almost all jobs are running, executing multiple jobs in a runner.
Expectation: First pipeline should be running (1 job per runner) and second pipeline should be pending.
Second pipeline's job/s will only be executed once done with first pipeline's job/s
You must set concurrent to 1 in the global settings.
Also double check your configuration file to ensure you have not registered multiple runners on the same machine.
I want to run jobs in the same stage sequentially instead of parallel in GitLab CI. Currently this is what I have:
I want unit-test to run before integration-test and not in parallel. I have looked into the docs and have encountered DAG but it needs the job to be in a prior stage and cannot be on the same stage. Is there a way to achieve this?
Yes its already described in the documentation for stages, jobs are started in parallel in one stage.
It says:
To make a job start earlier and ignore the stage order, use the needs keyword.
As you said, this is not possible in GitLab < 14.2 within a stage (needs):
needs: is similar to dependencies: in that it must use jobs from prior stages, meaning it's impossible to create circular dependencies. Depending on jobs in the current stage is not possible either, but support is planned.
As an alternative, you could define several stages and use the keyword needs between jobs in these stages.
Since GitLab 14.2 (issue) its possible.
I am reading about Concurrent pipelines in azure.
Concurrent pipelines
You can run concurrent pipelines (also called parallel jobs) in Azure
Pipelines. One parallel job in Azure Pipeline lets you run a single
build or release job at any given time. This rule is true whether you
run the job on Microsoft-hosted or self-hosted agents. Parallel jobs
are purchased at the organization level, and they are shared by all
projects in an organization.
My understanding is that - the azure build pipeline is organized into jobs (either agent/agentless jobs). Each job contains tasks. On auto/manual trigger the build pipeline runs and I thought that the number of pipelines that can run in parallel (assuming each pipeline has got only 1 job in them) depends on the availability of build agents (machines - either azure or hosted).
So what exactly is the concept of concurrent pipelines? What is the meaning of "One parallel job in Azure Pipeline lets you run a single build or release job at any given time."? In simple English, buying One parallel job should allow us to either a) run 2 build pipelines (assuming each pipeline contains only 1 job) or b) 1 pipeline with 2 jobs in parallel simultaneously. But this depends on availability of build agent as each pipeline (with 1 job) or 1 pipeline with 2 jobs will need 2 machines to run parallelly. Does it also mean that by default (free of charge) only one build pipeline can run at a time? There seems to be confusion between parallel job and parallel pipeline because one pipeline can have parallel job.
I need clarity on this topic with respect to pipeline/job/parallel pipeline/parallel job/count of build agents/count of parallel jobs.
I need clarity on this topic with respect to pipeline/job/parallel
pipeline/parallel job/count of build agents/count of parallel jobs.
Check Relationship between jobs and parallel jobs:
1.When you define a pipeline, you can define it as a collection of jobs. When a pipeline runs, you can run multiple jobs as part of that pipeline.
2.Each job consumes a parallel job that runs on an agent. When there aren't enough parallel jobs available for your organization, the jobs are queued up and run one after the other.
So if we have a pipeline with two jobs: When I queue the pipeline,these two jobs can't run at the same time if we only have one parallel job.
There're different count of parallel jobs available for microsoft-hosted and self-hosted agents, you can follow View available parallel jobs to check the parallel jobs you have.
And for count of build agents, there's no count limit for microsoft-hosted agents. If you're meaning self-hosted agents, you can own many agents in your agent pool.(The limit of count is something we won't meet in normal situation.) We can also install more than one agents in same local machine, see Can I install multiple self-hosted agents on the same machine?.
Hope all above helps :)
Well, the agents don't run pipelines.
They run jobs.
So if you are allowed to run "2 concurrent pipelines", it must mean 2 parallel jobs.
These jobs can be in a single pipeline if they are allowed to run in parallel (i.e. the other is not dependent on the first one).
Yes, on the free version it seems only one job can run in parallel.
I'm not sure when this was released, but there is a setting in the Pre-deployment conditions of an environment that fixed this for me. (Same place you'd find Triggers, Pre-depoyment approvals, Gates)
Pre-deployment conditions >> Deployment queue settings >> Number of parallel deployments = 1 Specific = Maximum number of parallel deployments = 1.
I have a Gitlab CI pipeline with multiple stages. By default, the same stage between two jobs will be run in parallel. One of the stages I have is an integration test stage that should only have one running instance at a time since it relies on specific resources being setup and torn down.
Is there a simple way to specify a certain stage should only have one concurrent instance?
Gitlab Enterprise Edition v10.0.3
That is possible however it's not straightforward. Define a job with a tag:
job:
tags:
- integration
script:
- run intergation tests
This will make sure the job runs only on a runner that has a specific tag. Now create a runner with the tag integration and concurrency set to 1.
The integration test will run one at a time because there's only one runner who can run them and there's only 1 slot on it.