How to delay a job in azure devops pipelines, I have multiple that will be running simultaneously, the problem is in the checkout phase I get the error saying files are used by another process.
I found "delayForMinutes" and running a powershell script but they only work for tasks not for jobs.
My goal is to have the checkout phase for the job to be delayed not the tasks in it.
You can do something like after the checkout job add a agentless Job with in that you can include a delay task. Then you can continue the other task in a separate agent job
Related
Gitlab CI/CD
I want to run a job for the Merge Request that needs jar artefacts built during a job run in another pipelines (for commits).
I don't want to build them again because if takes too much time then the job will fail.
Can I do that?
I checked the
needs:
- project:
But that's only with Premium licences.
In a pipeline I have 2 jobs , second job parses the logs of first job and for that I am using below API to get job id
https://source.golabs.io/api/v4/projects/<id>/jobs?scope[]=success
Now issue is what will happen if I will execute multiple parallel runs using that pipeline.how I can differentiate the job logs in respective pipelines.
I am reading about Concurrent pipelines in azure.
Concurrent pipelines
You can run concurrent pipelines (also called parallel jobs) in Azure
Pipelines. One parallel job in Azure Pipeline lets you run a single
build or release job at any given time. This rule is true whether you
run the job on Microsoft-hosted or self-hosted agents. Parallel jobs
are purchased at the organization level, and they are shared by all
projects in an organization.
My understanding is that - the azure build pipeline is organized into jobs (either agent/agentless jobs). Each job contains tasks. On auto/manual trigger the build pipeline runs and I thought that the number of pipelines that can run in parallel (assuming each pipeline has got only 1 job in them) depends on the availability of build agents (machines - either azure or hosted).
So what exactly is the concept of concurrent pipelines? What is the meaning of "One parallel job in Azure Pipeline lets you run a single build or release job at any given time."? In simple English, buying One parallel job should allow us to either a) run 2 build pipelines (assuming each pipeline contains only 1 job) or b) 1 pipeline with 2 jobs in parallel simultaneously. But this depends on availability of build agent as each pipeline (with 1 job) or 1 pipeline with 2 jobs will need 2 machines to run parallelly. Does it also mean that by default (free of charge) only one build pipeline can run at a time? There seems to be confusion between parallel job and parallel pipeline because one pipeline can have parallel job.
I need clarity on this topic with respect to pipeline/job/parallel pipeline/parallel job/count of build agents/count of parallel jobs.
I need clarity on this topic with respect to pipeline/job/parallel
pipeline/parallel job/count of build agents/count of parallel jobs.
Check Relationship between jobs and parallel jobs:
1.When you define a pipeline, you can define it as a collection of jobs. When a pipeline runs, you can run multiple jobs as part of that pipeline.
2.Each job consumes a parallel job that runs on an agent. When there aren't enough parallel jobs available for your organization, the jobs are queued up and run one after the other.
So if we have a pipeline with two jobs: When I queue the pipeline,these two jobs can't run at the same time if we only have one parallel job.
There're different count of parallel jobs available for microsoft-hosted and self-hosted agents, you can follow View available parallel jobs to check the parallel jobs you have.
And for count of build agents, there's no count limit for microsoft-hosted agents. If you're meaning self-hosted agents, you can own many agents in your agent pool.(The limit of count is something we won't meet in normal situation.) We can also install more than one agents in same local machine, see Can I install multiple self-hosted agents on the same machine?.
Hope all above helps :)
Well, the agents don't run pipelines.
They run jobs.
So if you are allowed to run "2 concurrent pipelines", it must mean 2 parallel jobs.
These jobs can be in a single pipeline if they are allowed to run in parallel (i.e. the other is not dependent on the first one).
Yes, on the free version it seems only one job can run in parallel.
I'm not sure when this was released, but there is a setting in the Pre-deployment conditions of an environment that fixed this for me. (Same place you'd find Triggers, Pre-depoyment approvals, Gates)
Pre-deployment conditions >> Deployment queue settings >> Number of parallel deployments = 1 Specific = Maximum number of parallel deployments = 1.
I want to automatically start a job on an Azure Batch AI cluster once a week. The jobs are all identical except for the starting time. I thought of writing a PowerShell Azure Function that does this, but Azure Functions v2 doesn't support PowerShell and I don't want to use v1 in case it will be phased out. I would prefer not to do this in C# or Java. How can I do this?
Currently, there's no option available to trigger a job on Azure Batch AI cluster. Maybe you want to run a shell script which in turn can create a regular schedule using system's task scheduler. Please see if this doc by Said Bleik helps:
https://github.com/saidbleik/batchai_mm_ad#scheduling-jobs
I assume this way you can add multiple schedules for the job!
Azure Batch portal has "Job schedules" tab. You can go there, add a Job, and set a schedule for the Job. You can specify the recurrence in the Schedule
Scheduled jobs
Job schedules enable you to create recurring jobs within the Batch service. A job schedule specifies when to run jobs and includes the specifications for the jobs to be run. You can specify the duration of the schedule--how long and when the schedule is in effect--and how frequently jobs are created during the scheduled period.
We have 4 deploy jobs in the same stage that can be run concurrently. From the Gitlab docs:
The ordering of elements in stages defines the ordering of jobs' execution:
Jobs of the same stage are run in parallel.
Jobs of the next stage are run after the jobs from the previous stage complete successfully.
What happens, however, is that only one of the jobs run at a time and the others stay on pending. Is there perhaps other things that I need to do in order to get it to execute in parallel. I'm using a runner with a shell executor hosted on an Ubuntu 16.04 instance.
Your runner should be configured to enable concurrent jobs( see https://docs.gitlab.com/runner/configuration/advanced-configuration.html)
concurrent = 4
or you may want to setup several runners.
I also ran into this problem. I needed to run several tasks at the same time. I used everything I could find (from needs to parallel). however, my tasks were still performed sequentially. every task I had was on standby. the solution turned out to be very simple. open file /etc/gitlab-runner/config.toml concurent for the required number of parallel tasks for you.