I am using Django-Q to schedule a periodic simple task that has to be repeated every < 1 minute.
Croniter, used under the hood to parse cron expressions for the scheduler, specifies that cron "seconds" support is available:
https://pypi.org/project/croniter/#about-second-repeats
So I created a cron-type schedule that looks like this:
Schedule.objects.update_or_create(name='mondrian_scheduler', defaults= {'func':'mondrianapi.tasks.run_scheduler', 'schedule_type':Schedule.CRON,
'cron': '* * * * * */20'} )
Django-q correctly parses and schedules the job, but the real frequency doesn't seem to go below 30 seconds (31, actually), whatever the 6th argument says:
2021-05-12 10:17:08.528307+00:00---run_bot ID 1
2021-05-12 10:17:39.166822+00:00---run_bot ID 1
2021-05-12 10:18:09.899772+00:00---run_bot ID 1
2021-05-12 10:18:40.648140+00:00---run_bot ID 1
2021-05-12 10:19:11.176563+00:00---run_bot ID 1
2021-05-12 10:19:41.857376+00:00---run_bot ID 1
The guard (or sentinel) process is responsible for querying for any scheduled tasks which are due, and it only does this twice per minute:
Scheduler
Twice a minute the scheduler checks for any scheduled tasks that should be starting.
Creates a task from the schedule
Subtracts 1 from django_q.Schedule.repeats
Sets the next run time if there are repeats left or if it has a negative value.
https://django-q.readthedocs.io/en/latest/architecture.html?highlight=scheduler#scheduler
The guard process is also responsible for checking that all of the other processes are still running, so it is not exactly thirty seconds.
Unfortunately the scheduler interval is not configurable. If you're comfortable modifying django_q, the relevant code is in django_q/cluster.py, in Sentinel.guard().
Related
Every hour, new data will be loaded in a table. As soon as that is ready I want to call a stored procedure.
Is it possible with Snowflake Tasks and Streams to schedule that starting every hour on the hour, check every minute if the table has new data and if it does, call a stored procedure and stop checking every minute?
At first I thougt about having 1 task with an hourly schedule. The next task would be scheduled every minute and have the hourly task as its predecessor. Unfortunately it is not possible for a task to have both a predecessor and a schedule. What would be a way to create this kind of scheduling?
For example
CREATE OR REPLACE TASK tsk_X_X_1
WAREHOUSE = X
SCHEDULE = 'USING CRON 0 1 * * * UTC'
AS INSERT INTO XX.Test SELECT 'hourly', sysdate()
;
CREATE OR REPLACE TASK tsk_X_X_2
WAREHOUSE = X
SCHEDULE = '5 minutes'
AFTER TASK tsk_X_X_1
As INSERT INTO XX.Testing SELECT '5min', sysdate()
;
Error: Task TSK_X_X_2 cannot have both a schedule and a predecessor.
I'm having an unexpected problem with Slurm when it comes to jobs that are requeued due to preemption.
Job ID 5 is queued with priority 1 and PreemptMode=requeue
Job ID 66 is queued with priority 1 and PreemptMode=requeue
Job ID 777 is queued with priority 3
Job ID 777 is cancelled within a short period of time from being queued. Job ID 66 starts.
What appears to be happening is that when Job ID 5 is requeued it's given a BeginTime roughly two minutes from current time. If Job ID 7777 is cancelled after that BeginTime, Job ID 66 will start because Job ID 5 is queued for a later time.
How can I set up Slurm to get around this problem? I want it to respect queue order in situations like this and always start Job ID 5 before Job ID 66.
I want to create a cron expression which will run the scheduler every 2.5 min of every hour. e.g. 2.5min, 5.0min, 7.5min, 10.0min etc. of every hour.
I am using Spring to create the scheduler. I tried various combination but nothing worked. One of them is as below but it is not working.
#Scheduled(cron = "*/30 */2 * * * *")
Thanks in advance.
That should works for you
0 0/5 0 ? * * *
30 2/5 0 ? * * *
At second :00, every 5 minutes starting at minute :00, at 00am, of every day
At second :30, every 5 minutes starting at minute :02, at 00am, of every day
You are right in this case you need to schedule your task twice using expression like on example.
There is a danger of becoming fixated on the 30 seconds. My problem was that I needed to check 18000 records for updates every month ~ 1 record every 2.5 minutes. I spent too much time trying techniques to run a job at exactly 02:32:30 before I realised that accuracy was not important.
In my situation, I realised I could execute every 2 minutes, updating my full database every 25 days instead of every 31 days.
Alternatively, I could have had 2 cron jobs running every 5 minutes. First, a 2-minute gap, followed by a 3-minute gap.
02:30 02:32 02:35 02:37 02:40 02:42 02:45 02:47
My point is that when the cron job is live, it runs unseen. Obviously, everyone has their own specific problem, but before introducing complexity, consider if it is necessary. As long as the job executes, does it really matter the exact time it ran?
If I wish to run my function app at 11.15 PM to next day 1.15 AM(That mean MONDAY 11.15 PM start and TUESDAY 1.15 AM will end) and it will trigger every minute during this time.
Can anyone help me to write this CRON expression?
The timer triggers are designed for a single repeating interval. The only way to do this completely within a Function is to run the trigger once per minute, then abort if the current time isn't in the desired target time period.
Alternately, put your logic into an HTTP trigger configured to act as a webhook, then use an Azure scheduler to configure start and stop times and intervals.
You won't be able to use the scheduler free plan since it can only run once per hour, but the standard plan can run once per minute. Scheduler pricing here.
You'll have to do this in three lines I think,
15-59 23 * * *
* 0 * * *
0-15 1 * * *
This will run it from 23:15-23:59 then 00:00-00:59 then 1:00-1:15
In AWS glue service there is an option to trigger job by custom CRON expression. Before i used this (0/2 * * ? *) cron expression to trigger job for every 2 hours.
Now I need to change the cron expression to trigger every 90 minutes, i.e for every 1 and a half hour. I tried with many cron expressions but that did not triggered for every 90 minutes. Even if i give for 90 minutes, it trigged for every 1 hour.
Can anyone help me out by providing the correct cron expression to trigger job for every 90 minutes ?
You can use the following pattern which was based on Bill Weiss' answer on Server Fault. It was modified to comply with the unique syntax AWS uses (reference here):
0 0-21/3 * * ? *
30 1-22/3 * * ? *
You'll have to define two separate Glue Triggers to accomplish this, each with the same job settings.
If curious, the syntax reads:
Run every 0th minute for every third hour for 0-21 hours
Run every 30th minute for every third hour for 1-22 hours