Rewrite of the question:
Using airflow, I would like to schedule a process to run every two hours from 2 till 10 am and a single time at 22:30. The schedule_interval parameter accepts a cron-expression, but it is not possible to define a single cron-expression to achieve the above scheduling. Currently, I did:
dag = DAG(process_name, schedule_interval='30 2,4,6,8,10,12,14,16,18,20,22,23 * * *', default_args=default_args)
But this will execute the process every 30 minutes past the hour, and this every 2 hours from 2 till 23.
Is there a way I can combine two cron-schedules in Airflow?
0 2-10/2 * * *
30 22 * * *
Original question:
I have 2,4,6,10,12,14,16,18,20,22 00 * *
I need to have 23, 30 in my schedule, but I don't want 2-22 to be run at the 30 min interval.
So, I realized, it is not possible!
You can't use two cron expressions for the same DAG (Might change in the future if PR: Add support for multiple cron expressions in schedule_interval is accepted)
Starting Airflow >=2.2.0:
It is possible to get custom time based triggering using custom Timetable by customizing the DAG scheduling to match what you expect.
To do so you need to define the scheduling logic by implementing next_dagrun_info and infer_manual_data_interval functions - Airflow will leverage this logic to schedule your DAG.
You can view an example can be found here.
Related
Like i have tried multiple combinations in cron guru but havent found any possible solution.
I have these two expressions to combine
30 10 1 * *
10 12 2-31 * *
You cannot implement this in one cron expression, you have two options to do that:
easy one: create two copy of the dag, each one with its own cron expression
hard one: create a Timetable and register it in an Airflow plugin, then you can use pass it to your dag as timetable argument instead of schedule_interval.
In this timetable, you can implement two methods:
next_dagrun_info: The scheduler uses this to learn the timetable’s regular schedule
infer_manual_data_interval: When a DAG run is manually triggered (from the web UI, for example), the scheduler uses this method to learn about how to reverse-infer the out-of-schedule run’s data interval (not mandatory).
I can't figure out how to create a job that ends at a specific hour and minute
If you break your cronjob into two, it would look like:
* 8-19 * * * command
0-30/1 19 * * * command
first line runs every minute from 8-19, and second line every minute from 19-19:30.
Cron triggers are not quite suitable for these types of schedules. If you do not insist on using a Cron trigger, I recommend that you check the Daily Time Interval trigger that is designed for use-cases such as yours. I am attaching a screenshot of a Daily Time Interval Trigger configuration for your use-case.
In AWS glue service there is an option to trigger job by custom CRON expression. Before i used this (0/2 * * ? *) cron expression to trigger job for every 2 hours.
Now I need to change the cron expression to trigger every 90 minutes, i.e for every 1 and a half hour. I tried with many cron expressions but that did not triggered for every 90 minutes. Even if i give for 90 minutes, it trigged for every 1 hour.
Can anyone help me out by providing the correct cron expression to trigger job for every 90 minutes ?
You can use the following pattern which was based on Bill Weiss' answer on Server Fault. It was modified to comply with the unique syntax AWS uses (reference here):
0 0-21/3 * * ? *
30 1-22/3 * * ? *
You'll have to define two separate Glue Triggers to accomplish this, each with the same job settings.
If curious, the syntax reads:
Run every 0th minute for every third hour for 0-21 hours
Run every 30th minute for every third hour for 1-22 hours
I wonder if it is possible to write an cron expression with several conditions:
Job should be run with given interval in minutes. For example with interval 42 minutes the fire times would be 10:00, 10:42, 11:24, 12:06 and etc.
If the current minute does not end with 0 (e.g. 10:28,10:29), then cron first fire time should be 10:30. So it means that first fire time should have "round" minutes.
I hope that you understand these conditions. Is it possible to describe them with quartz cron?
You can use job trigger like described below in Quartz.net 3.0:
var jobTrigger = TriggerBuilder.Create()
.StartNow()
.WithSimpleSchedule(s => s
.WithIntervalInMinutes(42)
.RepeatForever())
.Build();
And you can restart app at first round time, so it will fire first time at the same time only.
I usually use http://www.cronmaker.com/ to generate my cron expressions. And if you try the every 42 mins option you'll get the following expression: " 0 0/42 * 1/1 * ? *". As for the "round" minutes thing, you can try this when building your trigger:
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity(JobTrigger, JobGroup)
.WithCronSchedule(CroneExpression)
.StartAt(new DateTimeOffset(DateTime.Now,
TimeSpan.FromMinutes(DateTime.Now.Minute % 10)))
.Build();
It is not possible, see for explanation and similar issue: Quartz.net - Repeat on day n, of every m months?
it is also not possible by Cron expressions. To do this, you would need to apply some complex logic, use some operator that is not present in evaluators. Why do you need this? Would you like to combine those 2 requirements and create single complex pattern?
I want to schedule a job to run every 5 mins during morning (4-8)AM and evening (7-11)PM and rest of the day every 30 minutes, how can i write a cron expression for it?
I can do this by using two cron expressions for scheduling, but can i do the task using one expression?
No, I believe you'll need 2 separate CRON expressions.
The timings (or intensity) are different, so you need different expressions.
So something like the following is needed:
*/5 4-7,19-22 * * * your/command
*/30 0-3,8-18,23 * * * your/command