I have scheduled the K8s cron to run every 30 mins.
If the current job is still running and the next cron schedule has reached it shouldn't create a new job but rather wait for the next schedule.
And repeat the same process if the previous job is still in Running state.
set the following property to Forbid in CronJob yaml
.spec.concurrencyPolicy
https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#concurrency-policy
spec.concurrencyPolicy: Forbid will hold off starting a second job if there is still an old one running. However that job will be queued to start immediately after the old job finishes.
To skip running a new job entirely and instead wait until the next scheduled time, set .spec.startingDeadlineSeconds to be smaller than the cronjob interval (but larger than the max expected startup time of the job).
If you're running a job every 30 minutes and know the job will never take more than one minute to start, set .spec.startingDeadlineSeconds: 60
Related
I have an Airflow job that I wish to run every 130 minutes. I have set the cron schedule like this: "*/130 * * * *".
This schedule functions normally, until the clock hits midnight. Each day at midnight, if a job is currently underway, Airflow will terminate the job and start a new job. I do NOT want this behavior. Thanks in advance for your advice!
I have a pipelines in Azure DataFactory which is scheduled to run hourly.
Since every schedule task will have start time and end time (e.g. 1am - 2am) to copy files within this interval. I would like to know if old task overrun like finishing at 2:15am, what will be behaviour of next task?
(a) running task with start time and end time 2am-4am
(b) running task with start time and end time 3am-4am
My aim is to make sure no missing copying files.
I have tested this in my ADF.
Conclusion:
The previous pipeline's status won't affect the next task start time. So in your case, if you the previous pipeline started at 1am and finished at 2:15am, your next task will still start at 2am.
My test:
I create a Schedule trigger which runs every 3 min. My pipeline runs about 6 min.
Monitor pipeline runs and trigger runs:
My first task ends at 3/4/21, 3:32:41 PM, and the next task starts at 3/4/21, 3:30:00 PM. So if old task overrun, it won't affect the next task start time.
I am using NodeJS,MongoDB and node-cron npm module to schedule jobs. For 10K of jobs it is taking less time and less memory. But when i am scheduling 100k jobs it is taking more than 10 minutes to schedule jobs and taking nearly 1.5GB of RAM and some times out of memory. Is there any best way achieve this like using activemq or rabbitmq?
One strategy is that you only schedule the next job to run. When it runs, you query the database and find the next job and schedule it.
If you add a new job, you check if it wants to run sooner than the now current next job and, if so, you schedule it and deschedule the previous next job (it will get rescheduled later after this new job runs).
If you remove a job, you check if it is the current next job. If it is, you deschedule it and find the next job in the database and schedule it.
If your database is configured for efficiently querying by job run time, this can be very efficient, uses hardly any memory and scales to an infinitely large number of jobs.
I am dealing with a workflow where I need to start three processes. I have the first process which is to be scheduled at the beginning of every hour and the rest two at 45th minute of every hour and the 52nd minute of every hour.
But Instead of making the client schedule two different jobs on their server what I would rather want is to have just one job configured to run in the beginning of every hour which does a bunch of stuff and then starts these cron jobs at their respective times. i.e. 45th minute and 52nd minute of the hour.
Is there any way to do this.
I don't have any experience with shell scripting and always schedule cron jobs manually on cron-tab.
Thanks!
I have a cron job which runs every minute. Sometimes, if the cron is running more than a minute then another cron job is instantiated to do the same task. Hence duplicate cron jobs are created which is NOT I want. I want to make a conditional check that if a cron for a specific task is running, wait till the cron job completes or skip creating new cron job till the existing cron completes.
Create a text file somewhere which will store a value. (for example 0 or 1) When the task execute, change the value to 1. In the cron job, add a check that if the value in the file is 1 then don't execute the job. When your task is complete, remember to switch the value back to the default (for example 0).
You can even create a file when the task starts, and delete the file when task end, and only execute the cron job if file doesn't exist.
You can even put the check in the task itself instead of cluttering your cron table