Bull MQ repeatable job not triggering - node.js

This question is in continuation with this thread Repeatable jobs not getting triggered at given cron timing in Bull
I am also facing the same problem. How should I specify the timezone? I tried to specify as
repeat: { cron: '* 7 14 * * *', tz: 'Europe/Berlin'}
Meaning trigger the job at 14:07 German time zone. Though the job is listed in the queue, but the job is not triggered.
I also tried repeat:
{
cron: '* 50 15 * * *',
offset: datetime.getTimezoneOffset(),
tz: 'Europe/Berlin'
}

I finally figured out the solution.
One thing to note is that I had not initialized a Queuescheduler instance. Ofcourse timezone also plays a crucial role. But without a Queuescheduler instance (which has the same name as the Queue), the jobs doesnt get added into the queue. The Queuescheduler instance acts as a book keeper. Also take care about one more important parameter "limit". If you dont set the limit to 1, then the job which is scheduled at a particular time will get triggered unlimited number of times.
For example: To run a job at german time 22:30 every day the configuration would look like:
repeat: {
cron: '* 30 22 * * *',
offset: datetime.getTimezoneOffset(),
tz: 'Europe/Berlin',
limit: 1
}
Reference: https://docs.bullmq.io/guide/queuescheduler In this above link, the documentation clearly mentions that the queuescheduler instance does the book keeping of the jobs.
In this link - https://docs.bullmq.io/guide/jobs/repeatable, the documentation specifically warns us to ensure that we instantiate a Queuescheduler instance.

You need to manage repeatable queues with the help of QueueSchedular. QueueSchedular takes the queue name as first parameter and connection as second. The code will be as following:
const queueSchedular = new QueueSchedular(yourQueue.name, { connection });

Related

Combine two cron-scheduling intervals in a single DAG

Rewrite of the question:
Using airflow, I would like to schedule a process to run every two hours from 2 till 10 am and a single time at 22:30. The schedule_interval parameter accepts a cron-expression, but it is not possible to define a single cron-expression to achieve the above scheduling. Currently, I did:
dag = DAG(process_name, schedule_interval='30 2,4,6,8,10,12,14,16,18,20,22,23 * * *', default_args=default_args)
But this will execute the process every 30 minutes past the hour, and this every 2 hours from 2 till 23.
Is there a way I can combine two cron-schedules in Airflow?
0 2-10/2 * * *
30 22 * * *
Original question:
I have 2,4,6,10,12,14,16,18,20,22 00 * *
I need to have 23, 30 in my schedule, but I don't want 2-22 to be run at the 30 min interval.
So, I realized, it is not possible!
You can't use two cron expressions for the same DAG (Might change in the future if PR: Add support for multiple cron expressions in schedule_interval is accepted)
Starting Airflow >=2.2.0:
It is possible to get custom time based triggering using custom Timetable by customizing the DAG scheduling to match what you expect.
To do so you need to define the scheduling logic by implementing next_dagrun_info and infer_manual_data_interval functions - Airflow will leverage this logic to schedule your DAG.
You can view an example can be found here.

Cron Schedule for a job that runs every minute from 8 am to 7:30 pm

I can't figure out how to create a job that ends at a specific hour and minute
If you break your cronjob into two, it would look like:
* 8-19 * * * command
0-30/1 19 * * * command
first line runs every minute from 8-19, and second line every minute from 19-19:30.
Cron triggers are not quite suitable for these types of schedules. If you do not insist on using a Cron trigger, I recommend that you check the Daily Time Interval trigger that is designed for use-cases such as yours. I am attaching a screenshot of a Daily Time Interval Trigger configuration for your use-case.

Quzrtz scheduler with multiple cron schedules in a single trigger

I am trying to create a windows service that executes twice a day.
and I was successfully able to create it using two triggers added to a single job.
var job = JobBuilder.Create<Job>().StoreDurably().WithIdentity("Report_Name", "Report_Group").Build();
scheduler.AddJob(job, true);
var trigger_1 = TriggerBuilder.Create()
.WithIdentity("Report_Name_1", "Report_Group_A")
.StartNow()
.WithCronSchedule(string.Format("0 {0} {1} ? * *", Utility.Schedule_StartTime_1.Minute, Utility.Schedule_StartTime_1.Hour)) //0 Min hour
.ForJob(job)
.Build();
var trigger_2 = TriggerBuilder.Create()
.WithIdentity("Report_Name_2", "Report_Group_B")
.StartNow()
.WithCronSchedule(string.Format("0 {0} {1} ? * *", Utility.Schedule_StartTime_2.Minute, Utility.Schedule_StartTime_2.Hour)) //0 Min hour
.ForJob(job)
.Build();
scheduler.ScheduleJob(trigger_1);
scheduler.ScheduleJob(trigger_2);
scheduler.Start();
Can I use a single trigger to add multiple cron schedules
No, the trigger can have only one schedule.
One of the main reason why this is done is to prevent a situation when it is not clear for scheduler how to resolve competition between
conditions.
Imagine you have a job with 2 intersected schedules: let's say you want to run the job every 15 mins and every hour, and it takes up to 10 mins to execute it. In this case, you would need to specify how you want to handle scenarios, when
a job is executing, but scheduler fires new execution.
a job should be fired by both schedules
To allow handling such cases, the trigger has attributes like Priority and Misfire Instructions.

Scheduling for Spark jobs on Bluemix

I'm trying to run my Spark application on Bluemix by schedule. For now I'm using scheduling of spark-submit.sh script locally on my machine. But I'd like to use Bluemix for this purpose. Is there any way to set scheduling directly inside Bluemix infrastructure for running Spark notebooks or Spark applications?
The Bluemix OpenWhisk offering provides an easy way to schedule actions run on a periodic schedule similar to cron jobs.
Overview of OpenWhisk-based solution
OpenWhisk provides a programming model based actions, triggers, and rules. For this use case, you would
Create an action that kicks off your spark job.
Use the /whisk.system/alarms package to arrange for triggers to arrive periodically according to your schedule.
Create a rule that declares that your action should fire whenever a trigger event occurs.
Your action can be coded in javascript if it's easy to kick off your job from a javascript function. If not, and you'd like your action to be implemented by a shell script, you can use whisk docker actions to manage your shell script as an action.
Using the whisk.system/alarms package to generate events on a schedule.
This page in the whisk docs includes a detailed description of how to accomplish this. Briefly:
The /whisk.system/alarms/alarm feed configures the Alarm service to fire a trigger event at a specified frequency. The parameters are as follows:
cron: A string, based on the Unix crontab syntax, that indicates when to fire the trigger in Coordinated Universal Time (UTC). The string is a sequence of six fields separated by spaces: X X X X X X. For more details on using cron syntax, see: https://github.com/ncb000gt/node-cron. Here are some examples of the frequency indicated by the string:
* * * * * *: every second.
0 * * * * *: top of every minute.
* 0 * * * *: top of every hour.
0 0 9 8 * *: at 9:00:00AM (UTC) on the eighth day of every month
trigger_payload: The value of this parameter becomes the content of the trigger every time the trigger is fired.
maxTriggers: Stop firing triggers when this limit is reached. Defaults to 1000.
Here is an example of creating a trigger that will be fired once every 20 seconds with name and place values in the trigger event.
$ wsk trigger create periodic --feed /whisk.system/alarms/alarm --param cron '*/20 * * * * *' --param trigger_payload '{"name":"Odin","place":"Asgard"}'
Each generated event will include as parameters the properties specified in the trigger_payload value. In this case, each trigger event will have parameters name=Odin and place=Asgard.

node js scheduler it definite time with agenda

I am trying to create scheduler with a help of agenda module. I want to do something simple: to run some job every 30 min. For this one I can use agenda.every('30 minutes', 'my task') function. But how can I tell to agenda to do it exactly in 00 or in 30 minutes of every hour? Since I use for deployment heroku that reboots dynos from time to time I can't be sure that heroku won't move my scheduler from 30 minutes to 45 for example.
You can use cron package from npm also - https://www.npmjs.com/package/cron.
It supports cron syntax. As a result you can write schedules like 0,30 * * * * *.

Resources