I use Azure DevOps to schedule jobs on Azure Batch AI. Launching of jobs works great, I have python code that does the same.
What I am trying to achieve is that all jobs in the Batch AI experiment should be terminated when the build is cancelled. Currently, cancelling the build doesn't affect the run status of the Batch AI jobs.
Hence, is there a sort of "OnCancel" event to hook on to in the build to run a command (which will be python code to terminate all jobs) ?
There is no need to look for an event, as a pipeline task can be configured to execute where the build was cancelled.
Note: this applies (as far as I am aware) to any task of the pipeline:
Specifically, the Run this task setting, under Control Options will let you dictate when and under what conditions a task will run.
In the example above, this task will execute even if previous tasks fail, and even if the build was canceled.
In your case, I would place this as the last task that will perform the cleanup that you want, regardless of the outcome of the build.
Related
I'm quite new to Azure Devops, so sorry if it's obvious questions.
I have a release pipeline with 3 stages like this:
First stage is run on Agent A on Machine A, Stages 2,3 run on Agent B on Machine B.
Once stage 1 in prev. pipeline is finished -> it will start stage 1 of next scheduled pipeline run.
Is there a way to prevent this? I would like to start next scheduled pipeline run only after all stages are finished in previous one.
You can add Dependencies: If you are using YAML:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/stages?view=azure-devops&tabs=yaml
IF you are using classic pipelines you can use "Pre deployment Condition"
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/define-multistage-release-process?view=azure-devops
Also you can use approval gate if you want manual intervention.
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/approvals/?view=azure-devops
According to Specify queuing policies, this is not currently possible except for using manual intervention.
YAML pipelines don't support queuing policies. Each run of a pipeline
is independent from and unaware of other runs. In other words, your
two successive commits may trigger two pipelines, and both of them
will execute the same sequence of stages without waiting for each
other. While we work to bring queuing policies to YAML pipelines, we
recommend that you use manual approvals in order to manually sequence
and control the order the execution if this is of importance.
You can use Post-Deployment approvals:
or Pre-Deployment approvals:
The deployment of the company product has several tasks to finish. For example,
task 1 to copy some build files to server A
task 2 to copy some build files to server B
task 1 or 2 could fail and we need to redeploy only the failed task because each task takes a long time to finish.
I can split the tasks into different stages but we have a long tasks list and if we include staging and production it will be difficult to manage.
so my question is
is there an easy way to redeploy partial tasks without editing and disabling the tasks in the stage?
or a better way to organize multiple stages into one group like 'Staging' or 'Production' so I can have a better visualization of the release stages
thanks
Update:
Thanks #jessehouwing
Found there is an option when I click redeploy. See screenshot below.
You can group each stage with one or more jobs. You can easily retry jobs without having to run the whole stage. You will get the overhead of each job fetching sources or downloading artifacts and to use the output of a previous job you need to publish the result. One advantage is that jobs can run in parallel, your overall duration may actually be shorter that way.
I have two pipelines (also called "build definitions") in azure pipelines, one is executing system tests and one is executing performance tests. Both are using the same test environment. I have to make sure that the performance pipeline is not triggered when the system test pipeline is running and vice versa.
What I've tried so far: I can access the Azure DevOps REST-API to check whether a build is running for a certain definition. So it would be possible for me to implement a job executing a script before the actual pipeline runs. The script then just checks for the build status of the other pipeline by checking the REST-API each second and times out after e.g. 1 hour.
However, this seems quite hacky to me. Is there a better way to block a build pipeline while another one is running?
If your project is private, the Microsoft-hosted CI/CD parallel job limit is one free parallel job that can run for up to 60 minutes each time, until you've used 1,800 minutes (30 hours) per month.
The self-hosted CI/CD parallel job limit is one self-hosted parallel job. Additionally, for each active Visual Studio Enterprise subscriber who is a member of your organization, you get one additional self-hosted parallel job.
And now, there isn't such setting to control different agent pool parallel job limit.But there is a similar problem on the community, and the answer has been marked. I recommend you can check if the answer is helpful for you. Here is the link.
I have an Azure Batch service set up with a job schedule that runs every minute.
The job manager task creates 3-10 tasks within the same job.
Sometimes, one of these tasks within the job may take extremely long to complete but usually are very fast.
In the event that one of the tasks takes long to apply, the next iteration of the job manager task does not begin in that case. It basically waits till all the tasks from the previous iteration have completed.
Is there a way to ensure that the job schedule keeps creating a version of the job every minute even if all the tasks from its previous iteration have not been completed?
I know one option is to make the job manager task create additional jobs instead of tasks. But preferably, I was hoping there is some configuration at the job schedule level that I can turn on that will allow the schedule to create tasks without the dependency of completion on the previous job.
This seems like more towards design question, AFAIK, No, the duplicate active job names should not be doable from az batch perspective. (I will get corrected if at all this is doable somehow)
Although in order to further think this you can read through various design recommendations via Azure batch technical overview page or posts like:
How to use Azure Batch in an event based design and terminate/cleanup finished jobs or
Add Tasks to a running Azure batch job and manually control termination
I think simplicity will be better like handling each iteration with unique job name or some thing of other sort but you will know your scenario better. Hope this helps.
Currently, a Job Schedule can have at most one active Job under it at any given time (link) so the behavour you're seeing is expected.
We don't have any simple feature you can just "turn on" to achieve concurrent jobs from a single job schedule - but I do have a suggestion:
Instead of using the JobSchedule to run all the processing directly, use it to create "worker" jobs that do the processing.
E.g.
At 10:03 am, your job schedule triggers to create job processing-20191031-1003.
At 10:04 am, your job schedule triggers to create job processing-20191031-1004.
At 10:05 am, your job schedule triggers to create job processing-20191031-1005.
and so on
Because the only thing your job schedule does is create another job, it will finish very quickly, ensuring the next job is created on time.
Since your existing jobs already create a variable number of tasks (you said 3-10 tasks, above), I'm hoping this won't be a very complex change for your code.
Note that you will need to ensure your concurrent worker jobs don't step on each others toes by trying to do the same work multiple times.
We have an Azure-batch job that uses some quite large files which we are uploading to Azure Blob storage asynchronously so that we don't have to wait for all files to upload before starting our batch job made up of a collection of Tasks that will process each file and generate output. All good so far - this is working fine.
I'd like to be able to create an Azure Task and Add it to an existing, running Azure Job increasing the length of the Task list but I cant find how to do this. It seems that Azure expects you to define ALL jobs for a Task before the Job starts and then it runs until all tasks are complete and terminates the job (which makes sense in some scenarios - but not mine).
I would like to suppress this Job completion behavior and be able to queue up additional Azure Tasks for the same job. I could then monitor the Azure Job status (via the Tasks) and determine myself if the Job is complete.
Our issue is that uploads of multi-MB files takes time and we want Task processing to start as soon as the first file is available. If we have to wait until all files are available, then our processing start is delayed which is not what we need.
We 'could'create a job per task and manage it in our application but that is a little 'messy' and I would like to use the encapsulating Azure Job entity and supporting functionality if I possibly can.
Has anyone done this and can offer some guidance? Many thanks?
You can add new tasks to an existing Azure Batch job in the active state. There is no running state for an Azure Batch job. You can find a list of Azure Batch job states here.
Azure Batch Jobs, by default, do not automatically complete by terminating upon all tasks completing. You can view this related question regarding this subject.