I have a pipelines in Azure DataFactory which is scheduled to run hourly.
Since every schedule task will have start time and end time (e.g. 1am - 2am) to copy files within this interval. I would like to know if old task overrun like finishing at 2:15am, what will be behaviour of next task?
(a) running task with start time and end time 2am-4am
(b) running task with start time and end time 3am-4am
My aim is to make sure no missing copying files.
I have tested this in my ADF.
Conclusion:
The previous pipeline's status won't affect the next task start time. So in your case, if you the previous pipeline started at 1am and finished at 2:15am, your next task will still start at 2am.
My test:
I create a Schedule trigger which runs every 3 min. My pipeline runs about 6 min.
Monitor pipeline runs and trigger runs:
My first task ends at 3/4/21, 3:32:41 PM, and the next task starts at 3/4/21, 3:30:00 PM. So if old task overrun, it won't affect the next task start time.
Related
Lets assume a scenario where pipeline A runs every day and pipeline B runs once in every month and it is dependent on pipeline A (pipeline B should trigger after successful completion of pipeline A).
Using scheduled trigger, we cannot have hard dependencies between 2 pipelines, where as with tumbling window, we cannot exactly specify the day which the pipeline B should run(it has only two options, minutes and hours where as scheduled trigger has months and weeks also)
Both the triggers has its disadvantages with respect to this scenario.
What could be the best possible solution for this scenario?
You can Run Pipeline A everyday, and have an IF check that checks if its a specific date today, then run Pipeline B if TRUE and nothing if FALSE.
For the settings of If Condition, you can use this as variable, if you want to run it every 1st of every month:
#Contains('01',Substring(formatDateTime(utcnow()),8,2))
With an Azure Data Factory "Tumbling Window" trigger, is it possible to limit the hours of each day that it triggers during (adding a window you might say)?
For example I have a Tumbling Window trigger that runs a pipeline every 15 minutes. This is currently running 24/7 but I'd like it to only run during business hours (0700-1900) to reduce costs.
Edit:
I played around with this, and found another option which isn't ideal from a monitoring perspective, but it appears to work:
Create a new pipeline with a single "If Condition" step with a dynamic Expression like this:
#and(greater(int(formatDateTime(utcnow(),'HH')),6),less(int(formatDateTime(utcnow(),'HH')),20))
In the true case activity, add an Execute Pipeline step executing your original pipeline (with "Wait on completion" ticked)
In the false case activity, add a wait step which sleeps for X minutes
The longer you sleep for, the longer you can possibly encroach on your window, so adjust that to match.
I need to give it a couple of days before I check the billing on the portal to see if it has reduced costs. At the moment I'm assuming a job which just sleeps for 15 minutes won't incur the costs that one running and processing data would.
there is no easy way but you can create two deployment pipelines for the same job in Azure devops and as soon as your winodw 0700 to 1900 expires you replace that job with a dummy job using azure dev ops pipeline.
I have scheduled the K8s cron to run every 30 mins.
If the current job is still running and the next cron schedule has reached it shouldn't create a new job but rather wait for the next schedule.
And repeat the same process if the previous job is still in Running state.
set the following property to Forbid in CronJob yaml
.spec.concurrencyPolicy
https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#concurrency-policy
spec.concurrencyPolicy: Forbid will hold off starting a second job if there is still an old one running. However that job will be queued to start immediately after the old job finishes.
To skip running a new job entirely and instead wait until the next scheduled time, set .spec.startingDeadlineSeconds to be smaller than the cronjob interval (but larger than the max expected startup time of the job).
If you're running a job every 30 minutes and know the job will never take more than one minute to start, set .spec.startingDeadlineSeconds: 60
I am dealing with a workflow where I need to start three processes. I have the first process which is to be scheduled at the beginning of every hour and the rest two at 45th minute of every hour and the 52nd minute of every hour.
But Instead of making the client schedule two different jobs on their server what I would rather want is to have just one job configured to run in the beginning of every hour which does a bunch of stuff and then starts these cron jobs at their respective times. i.e. 45th minute and 52nd minute of the hour.
Is there any way to do this.
I don't have any experience with shell scripting and always schedule cron jobs manually on cron-tab.
Thanks!
I have a cron job which runs every minute. Sometimes, if the cron is running more than a minute then another cron job is instantiated to do the same task. Hence duplicate cron jobs are created which is NOT I want. I want to make a conditional check that if a cron for a specific task is running, wait till the cron job completes or skip creating new cron job till the existing cron completes.
Create a text file somewhere which will store a value. (for example 0 or 1) When the task execute, change the value to 1. In the cron job, add a check that if the value in the file is 1 then don't execute the job. When your task is complete, remember to switch the value back to the default (for example 0).
You can even create a file when the task starts, and delete the file when task end, and only execute the cron job if file doesn't exist.
You can even put the check in the task itself instead of cluttering your cron table