I was recently looking at npm bull package to use for delayed jobs. I can see that for delayed jobs it is using redis sorted set and mapping the value of the job to a hashset in redis. But I fail to understand how does the process receives the delayed job on time without using any iterable function like a cron or a settimeout which might be used to constantly looks for the jobs, and pops them by the score? can any one help me point to the code which is responsible for getting the delayed jobs?
Related
i implement cron job A(EVERY_DAY_AT_8AM) on the batch server
when cron job A executed it queues a job to bull queue
and the other server consumes the job from bull queue
on batch server other cronjob which is executed more frequently than cron job A exists
The problem is although cron job A executed, job doesn't be queued to bull queue
In shorts, the job which has to be consumed had gone somewhere.
I can't even guess why this situation happen..
any one have ideas why does this issue happen??
I have hazelcast cluster and I'm currently using in my project TaskScheduler from Spring to execute cron tasks.
I would like to use hazelcast IScheduledExecutorService to schedule a cron task on all members, and to schedule a multiple cron tasks which will be distributed equally across cluster, but I can't find the appropriate method.
I can't find method in IScheduledExecutorService that have cron trigger in parameter, only timeUnit. Do you know any solutions to this?
IScheduledExecutorService is not a full cron scheduler, for it only allows the scheduling of a single future execution and/or a fixed rate execution but not periodic executions at fixed times, dates or intervals. So if you are happy to do something like:
executorService.schedule(new SomeTask(), 10, TimeUnit.SECONDS)
where SomeTask will run after 10 seconds, then you can choose any of the following APIs to submit your task:
scheduleOnMember: On a specific cluster member.
scheduleOnKeyOwner: On the partition owning that key.
scheduleOnAllMembers: On all cluster members.
scheduleOnAllMembers: On all given members.
Check reference manual for details: https://docs.hazelcast.org/docs/4.0.1/manual/html-single/index.html#scheduled-executor-service
I want to run a python cron-type scheduler and schedule a job for one time execution at a particular date and time. Once the job is done, the job has to disappear, while the scheduler still continues to run. Also looking for examples of using scheduler and storing jobs in mongodb or redis.
You can use celery for this.Celery helps you run tasks in background and schedule cron jobs as well.
These docs state that you can
schedule tasks to execute at a specific time
other useful resources.
Sorry but can't find he configuration point a need. I schedule spark application, sometimes they may not succeed after 1 hour, in this case I want to automatically kill this task (because I am sure it will never succeed, and another scheduling may start).
I found a timeout configuration, but as I understand it, this is used to delay the start of a workflow.
So is there a kind of living' timeout ?
Oozie cannot kill a workflow that it triggered. However you can ensure that a single workflow is running at same time by setting Concurrency = 1 in the Coordinator.
Also you can have a second Oozie workflow monitoring the status of the Spark job.
Anyawy, you should investigate the root cause of Spark job not successful or being blocked.
I use PBS job arrays to submit a number of jobs. Sometimes a small number of jobs get screwed up and not been ran successfully. Is there a way to automatically detect the failed jobs and restart them?
pbs_server supports automatic_requeue_exit_code:
an exit code, defined by the admin, that tells pbs_server to requeue the job instead of considering it as completed. This allows the user to add some additional checks that the job can run meaningfully, and if not, then the job script exits with the specified code to be requeued.
There is also a provision for requeuing jobs in the case where the prologue fails (see the prologue/epilogue script documentation).
There are probably more sophisticated ways of doing this, but they would fall outside the realm of built-in Torque options.