I installed node-cron library and configured 2 crons in the same file to run at different intervals. The first cron runs every 45 seconds and the second cron runs every 60 seconds, it works perfectly. The problem happens when the 60-second cron is run, automatically the 45-second cron is also run again (notice that the difference is 15 seconds). Why does this happen?
const cron = require('node-cron');
cron.schedule('*/45 * * * * *', async () => {
console.log('running 45 seconds')
})
cron.schedule('*/60 * * * * *', async () => {
console.log('running 60 seconds')
})
Cron will run this at "every 45th second, of every minute...".
Starts at 0 seconds, then at 45 seconds.
Starts at 0 seconds, then at 45 seconds.
This will be noticeable whenever the */x interval isn't divisible by the parent time unit (seconds in a minute, minutes in an hour etc).
For example, */17 * * * * *
2021-08-17T23:07:00.230Z running */17 seconds
2021-08-17T23:07:17.273Z running */17 seconds
2021-08-17T23:07:34.302Z running */17 seconds
2021-08-17T23:07:51.344Z running */17 seconds
2021-08-17T23:08:00.374Z running */17 seconds
2021-08-17T23:08:17.424Z running */17 seconds
2021-08-17T23:08:34.467Z running */17 seconds
2021-08-17T23:08:51.494Z running */17 seconds
2021-08-17T23:09:00.525Z running */17 seconds
2021-08-17T23:09:17.564Z running */17 seconds
To run something on a 45 second timer via cron, you probably need a task every 15 seconds then some logic to check if the previous run was >= 45 seconds ago.
Related
Problem: We're having difficulties having a DAG fire off given a defined interval. It's also preventing manual DAG executions as well. We've added catchup=False as well to the DAG definition.
Context: We're planning to have a DAG execute on a 4HR interval from M-F. We've defined this behavior using the following CRON expression:
0 0 0/4 ? * MON,TUE,WED,THU,FRI *
We're unsure at this time whether the interval has been defined properly or if there are extra parameters we're ignoring.
Any help would be appreciated.
I think you are looking for is 0 0/4 * * 1-5 which will run on 4th hour from 0 through 23 on every day-of-week from Monday through Friday.
Your DAG object can be:
from airflow import DAG
with DAG(
dag_id="my_dag",
start_date=datetime(2022, 2, 22),
catchup=False,
schedule_interval="0 0/4 * * 1-5",
) as dag:
# add your operators here
I'm running Apache Airflow 1.10.0 and I want to take advantage of the new timezone aware Dag feature. I must admit that the Airflow scheduler is a bit confusing and I'm not quite sure how to accomplish what I'm trying to do. I am trying to define a Dag that will run at 5 past midnight (Eastern time) every day.
So far I've tried defining the Dag with a timezone aware start_date using Pendulum. My schedule interval is timedelta(days=1). For some reason this has resulted in runs at seemingly odd times 12:00, etc.
My current Dag definition:
...
dag_tz = pendulum.timezone('US/Eastern')
default_args = {
'owner': 'airflow',
'email': '<email_address>',
'email_on_failure': True,
'email_on_retry': True,
'retries': 3,
'depends_on_past': False,
'retry_delay': timedelta(minutes=5),
'provide_context': True,
'start_date': datetime(2019, 5, 1, tzinfo=dag_tz)
}
dag = DAG('my_dag_id', default_args=default_args,
catchup=False, schedule_interval=timedelta(days=1))
...
What I'd like is for the Dag to run at the same time each day. I've seen that I can use a cron expression for schedule_interval but that's confusing as well because I'm not sure if I need to include my UTC offset in the cron expression or if the fact that the Dag is timzeone aware will take care of this.
For example, should my schedule_interval be 05 04 * * * or 05 00 * * * or something else entirely?
After some experimentation I have concluded that in order to get the dag to run at 5 past midnight every day I need to use a schedule interval of 05 00 * * * along with the timezone aware start date.
You can also write it without 0-prefix. Like 5 0 * * *
I am trying to have an airflow script to be scheduled to run every Tuesday at 9:10 AM UTC. Given below is how I have defined it.
dag = DAG(
dag_id=DAG_NAME,
default_args=args,
schedule_interval="10 9 * * 2",
catchup=False
I however find that when the time comes, the script does not get triggered automatically. However if I do not have the value defined in the day column (last column), the scheduler works fine. Any idea where I am going wrong.
Thanks
Update:
args = {
'owner': 'admin',
'start_date': airflow.utils.dates.days_ago(9)
}
dag = DAG(
dag_id=DAG_NAME,
default_args=args,
schedule_interval = "10 9 * * 2",
catchup = False
)
This one stumps people more than anything else in Airflow, but as commenter and Airflow documentation state,
The scheduler runs your job one schedule_interval AFTER the start date, at the END of the period.
In this case you can either bump back your DAG start_date one schedule_interval or wait for the next schedule_interval to complete.
I need to schedule a job from Mon-Thur at 7pm and on Friday i need it to be scheduled at 11pm. I am using Airflow and need the cron tab notation like
0 19 * * Mon-Thu
Any suggestion are welcome.
Thank you
Regards,
CJ
You can create your dag as:
dag = DAG("Your_dag", default_args=default_args, schedule_interval="0 19 * * 1-4")
You could do something like this:
schedules = {
'M-Th': '0 19 * * 1-4',
'F': '0 23 * * 5',
}
for name, schedule in schedules.items():
globals()[name] = DAG('<base_dag_name.' + name, default_args=default_args, schedule_interval=schedule)
This will create two DAGs from a single file. DAGs need to be in the global scope to be recognized by Airflow.
I need to run a cron job in linux at 20 minutes interval everyday. The most important thing is, it must be on specifically 10th, 30th and 50th minutes.
I think I need to run 3 cron jobs as :
10 * * * * /path_to_script
30 * * * * /path_to_script
50 * * * * /path_to_script
Is it possible to meet this requirement using a single cron job ?
10,30,50 * * * * /path_to_script
or
10/20 * * * * /path_to_script
aggregate all scripts in one with sleep(1200) time separator
#script
#!/bin/bash
./wayto/script1;
sleep(1200);
./wayto/script2;
sleep(1200)
./wayto/script3;
and make one job in cron via crontab -e
10 * * * * /bin/bash /way/to/script