Using Agenda, is it possible to set jobs to retry after several times after failure?
If it's a repeating job, you can change the nextRunAt value when it fails and it will run again:
agenda.on("fail", async (error, job) => {
const nextRunAt = new Date();
job.attrs.nextRunAt = nextRunAt;
await job.save();
});
if it's a scheduled job (meaning it doesn't repeat, the above code won't work. You have to create a new job.
agenda.on("fail", async (error, job) => {
if(job.attrs.repeatInterval) {
// create a new job and pass it job.attrs.data
}
});
Retry is not directly supported as an option, but it has a simple workaround by setting the failed jobs nextRunAt to a future time.
Refer this article :
https://crossbrowsertesting.com/blog/product-update/task-scheduling-with-agenda/
Related
I am trying to create a Schedule service using nestJs' Cron decorator
I have a Cron-decorated method below:
#Cron(CronExpression.EVERY_5_SECONDS)
triggerDataloaderCron() {
this.logger.debug('called every 10 seconds');
return this.healthService.getPCFHealth();
}
And this cron job calls a method in another service, which is shown below
getHealth() {
//code to form up an endpoint, saved as the variable fullUrl
//Does not come into this block
return this.httpService.get(fullUrl).pipe(
map((axiosResponse: AxiosResponse) => {
return axiosResponse.data;
}),
catchError((err) => {
console.log("in error", err);
throw new CustomException(ExceptionConstants.EurekaConnectionException);
})
);
}
When the cron job runs, i am able to enter the getHealth() method, but the this.httpService etc... block of code does not run.
Any suggestions on how this can be achieved? Or if I am going about this the wrong way?
Thanks!
The getHealth method returns an Observable. An observable does not execute unless there is at least one subscriber.
In your cron method, add a subscription as follows:
this.healthService.getPCFHealth()
.subscribe(response => console.log(response))
Since the cron method executes at regular intervals, I don't think you need to return a value from it.
I am using node bull for scheduling. Now for my jobs i pass a cron time to it (which is a specific time and a date). Now since i have provided a strict cron then it must run for only one time or even if it run second time then it must be removed from the queue (i have provided removeOnComplete: true).
But still it doesn't get removed and it runs again and my queue process this job again. So i want to stop this.
My add job function: (data is my job data, timings is my cron timing)
public async addJob(data: any, timings: any, jobId: any) {
console.log('adding job');
console.log(timings);
const job = await this.videoQueue.add(
{data: data},
{
repeat: {
cron: timings,
},
jobId: jobId,
removeOnComplete: true,
},
);
return job;
}
My process function:
this.videoQueue.process(async (job: any) => {
// processing function for my job.
});
I am new node and node bull. Maybe i am making a mistake but i am not able to debug it.
Thanks.
I can suggest you use node-schedule package to schedule with cron time. It's very easy to use actually
const schedule = require('node-schedule');
const job = schedule.scheduleJob('* * * * *', function(){
console.log('Job runned.');
job.cancel();
});
var que = async.queue(async function({id}) {
const res = await fetchData(id);
return res;
}, 2);
que.push({id}, function (err, res) {
console.log('handle res');
});
Here concurrency is set as 2. so after first two tasks are done and if I push a new task in the queue. nothing happens. No err, even the task in also not pushed.
If I update concurrency to 5, it will perform the first 5 tasks and then it fails for the 6th task.
Do we have to clear the queue after each task is done?
I am using it in Reactjs project.
NOTE: I am able to make this work correctly on my local, but when I create the react build and deploy it, it does not work there.
I started using AWS Lambda to perform a very simple task which is executing an SQL query to retrieve records from an RDS postgres database and create SQS message base on the result.
Because Amazon is only providing aws-sdk module (using node 4.3 engine) by default and we need to execute this SQL query, we have to create a custom deployment package which includes pg-promise. Here is the code I'm using:
console.info('Loading the modules...');
var aws = require('aws-sdk');
var sqs = new aws.SQS();
var config = {
db: {
username: '[DB_USERNAME]',
password: '[DB_PASSWORD]',
host: '[DB_HOST]',
port: '[DB_PORT]',
database: '[DB_NAME]'
}
};
var pgp = require('pg-promise')({});
var cn = `postgres://${config.db.username}:${config.db.password}#${config.db.host}:${config.db.port}/${config.db.database}`;
if (!db) {
console.info('Connecting to the database...');
var db = pgp(cn);
} else {
console.info('Re-use database connection...');
}
console.log('loading the lambda function...');
exports.handler = function(event, context, callback) {
var now = new Date();
console.log('Current time: ' + now.toISOString());
// Select auction that need to updated
var query = [
'SELECT *',
'FROM "users"',
'WHERE "users"."registrationDate"<=${now}',
'AND "users"."status"=1',
].join(' ');
console.info('Executing SQL query: ' + query);
db.many(query, { status: 2, now: now.toISOString() }).then(function(data) {
var ids = [];
data.forEach(function(auction) {
ids.push(auction.id);
});
if (ids.length == 0) {
callback(null, 'No user to update');
} else {
var sqsMessage = {
MessageBody: JSON.stringify({ action: 'USERS_UPDATE', data: ids}), /* required */
QueueUrl: '[SQS_USER_QUEUE]', /* required */
};
console.log('Sending SQS Message...', sqsMessage);
sqs.sendMessage(sqsMessage, function(err, sqsResponse) {
console.info('SQS message sent!');
if (err) {
callback(err);
} else {
callback(null, ids.length + ' users were affected. SQS Message created:' + sqsResponse.MessageId);
}
});
}
}).catch(function(error) {
callback(error);
});
};
When testing my lambda function, if you look at the WatchLogs, the function itself took around 500ms to run but it says that it actually took 30502.48 ms (cf. screenshots).
So I'm guessing it's taking 30 seconds to unzip my 318KB package and start executing it? That for me is just a joke or am I missing something? I tried to upload the zip and also upload my package to S3 to check if it was faster but I still have the same latency.
I noticed that the Python version can natively perform SQL request without any custom packaging...
All our applications are written in node so I don't really want to move away from it, however I have a hard time to understand why Amazon is not providing basic npm modules for database interactions.
Any comments or help are welcome. At this point I'm not sure Lambda would be benefic for us if it takes 30 seconds to run a script that is triggered every minute...
Anyone facing the same problem?
UPDATE: This is how you need to close the connection as soon as you don't need it anymore (thanks again to Vitaly for his help):
exports.handler = function(event, context, callback) {
[...]
db.many(query, { status: 2, now: now.toISOString() }).then(function(data) {
pgp.end(); // <-- This is important to close the connection directly after the request
[...]
The execution time should be measured based on the length of operations being executed, as opposed to how long it takes for the application to exit.
There are many libraries out there that make use of a connection pool in one form or another. Those typically terminate after a configurable period of inactivity.
In case of pg-promise, which in turn uses node-postgres, such period of inactivity is determined by parameter poolIdleTimeout, which defaults to 30 seconds. With pg-promise you can access it via pgp.pg.defaults.poolIdleTimeout.
If you want your process to exit after the last query has been executed, you need to shut down the connection pool, by calling pgp.end(). See chapter Library de-initialization for details.
It is also shown in most of the code examples, as those need to exit right after finishing.
I want to get the list of all the jobs that are scheduled by node-schedule
var scheduleIds = schedule.scheduleJob(scheduleId,rule, function() {
console.log("HELLO")
})
here scheduleId is a unique key and there are multiple jobs running with different names. How can I find a job from its name and cancel it?
var schedule = require('node-schedule');
var scheduleIds = schedule.scheduleJob(scheduleId,rule, function() {
console.log("HELLO")
})
The schedule object that is created contains all the names of the jobs that are scheduled.