I am using kue for my job queue, and I'd like to know without using the GUI how many jobs are still left, how many have failed, etc. How can I retrieve this kind of information?
For example, after a few minutes of starting the processing of the job queue, I'd like to o update the status of all jobs that failed so far to 'inactive', in order to restart them.
The only related question I could find on stackoverflow was this, however, it deals with one job at a time, after it fires a certain event as it is being processed. My concern is different, as I am interested in retrieving all jobs in the database with a certain status.
The answer to this question mentions the function .complete of the kue library, which retrieves all the completed jobs in the database. Are there similar functions for other possible job statuses?
I found a solution by browsing the kue source code. The following code achieves what I need:
var redis = require ('redis'),
kue = require ('kue'),
redisClient = redis.createClient(6379, "127.0.0.1");
kue.redis.createClient = function () {
return redisClient;
};
kue.app.listen(3000);
kue.Job.rangeByType ('job', 'failed', 0, 10, 'asc', function (err, selectedJobs) {
selectedJobs.forEach(function (job) {
job.state('inactive').save();
});
});
For reference, here is the relevant kue source code:
/queue/job.js:123:
/**
* Get jobs of `type` and `state`, with the range `from`..`to`
* and invoke callback `fn(err, ids)`.
*
* #param {String} type
* #param {String} state
* #param {Number} from
* #param {Number} to
* #param {String} order
* #param {Function} fn
* #api public
*/
exports.rangeByType = function(type, state, from, to, order, fn){
redis.client().zrange('q:jobs:' + type + ':' + state, from, to, get(fn, order));
};
Kue source code indicating that:
type is the job type
from, to is the job ranges by index (for example, you can specify load jobs from index 0 to 10, 11 jobs in total.)
order specifies the order of fetched jobs. Default is asc. You can also sort it by desc
The following works, uses the pre-existing queue object and hence, no double Redis connection issue as mentioned by Japrescott in the comments of the accepted answer.
queue.cardByType("notifications", "complete", function( err, count ) {
console.log(count);
});
Feel free to replace with a valid state, the following is a list of valid states.
inactive
complete
active
failed
delayed
Related
I am writing a node application that uses node-cron to schedule certain function to run at specific times. Currently the code for this is embedded in the main application. I want to move the configuration of scheduled tasks out to a config file, so want need to store the function names in a variable to achieve this, as the schedule tasks call out a number of functions in different modules etc.
For information the syntax to schedule a cron task is this:
cron.schedule('* * * * *', () => {functiontocall()});
The code block below shows how I am currently storing the tasks in an object and trying to schedule them with node-cron.
mycronobj = [
{scheduletime : "* * * * *", schedulefunction : 'testfunction1'},
{scheduletime : "*/5 * * * *", schedulefunction : 'testfunction2'},
{scheduletime : "*/10 * * * *", schedulefunction : 'testfunction3'},
]
for (item in mycronobj) {
cron.schedule(mycronobj[item].scheduletime, () => {mycronobj[item].schedulefunction()});
}
However this doesn't work, the scheduled functions don't run. I have tried storing the functions names as a string (as shown) or direct as
{scheduletime : "* * * * *", schedulefunction : testfunction1()}
When trying to add the scheduled function I have tried this with the following syntaxes:
mycronobj[item].schedulefunction()
mycronobj[item]schedulefunction
mycronobj[item].schedulefunction
None of which have worked for me. I have tried looking for an answer to this and I tried using eval(), but this also didn't work correctly, the outcome was that a task with first schedule ("* * * * *") was scheduled with the last function 'testfunction3' was applied, also I dont really want to use eval as I have read its not great practice and can be avoided if you know what your doing (clearly I don't).
Other options I have come across is to use Window which doesn't exist in Node or This, which I cannot get to work either.
Thanks in advance.
The reason your code isn't working is that mycronobj[item]schedulefunction is a string so is not invokable. You need to turn this string into a reference to the function with the same name.
The simplest solution to this is using eval, e.g:
for (item of mycronobj) {
cron.schedule(item.scheduletime, () =>
eval(`${item.schedulefunction}()`);
}
However this is a really bad idea, and allows for arbitrary code execution which is generally considered a very bad thing (tm).
Node gives you a slightly safer alternative to this using the vm module, which can create a limited sandbox in which your code can execute:
const vm = require("vm");
let context = vm.createContext({ foo });
for (item of mycronobj) {
cron.schedule(item.scheduletime, () =>
vm.runInContext(`${item.schedulefunction}()`, context);
}
This is only marginally better though since it can still allow arbitrary code execution, so by far the safest option is just to explicitly list exactly which functions are allowed to be called:
const commands = {
testfunction1: () => console.log("I am 1"),
testfunction2: () => console.log("I am 2"),
testfunction3 // Reference existing function
}
function testfunction3() { console.log("I am 3"); }
for (item of mycronobj) {
let func = commands[item.schedulefunction];
if (!func) { throw new Error(`Unknown command: "${item.schedulefunction}"`); }
cron.schedule(item.scheduletime, func);
}
This also has the benefit of determining during setup whether functions are valid rather than only when the cron job runs.
I want to have a cron job/scheduler that will run every 30 minutes after an onCreate event occurs in Firestore. The cron job should trigger a cloud function that picks the documents created in the last 30 minutes-validates them against a json schema-and saves them in another collection.How do I achieve this,programmatically writing such a scheduler?
What would also be fail-safe mechanism and some sort of queuing/tracking the documents created before the cron job runs to push them to another collection.
Building a queue with Firestore is simple and fits perfectly for your use-case. The idea is to write tasks to a queue collection with a due date that will then be processed when being due.
Here's an example.
Whenever your initial onCreate event for your collection occurs, write a document with the following data to a tasks collection:
duedate: new Date() + 30 minutes
type: 'yourjob'
status: 'scheduled'
data: '...' // <-- put whatever data here you need to know when processing the task
Have a worker pick up available work regularly - e.g. every minute depending on your needs
// Define what happens on what task type
const workers: Workers = {
yourjob: (data) => db.collection('xyz').add({ foo: data }),
}
// The following needs to be scheduled
export const checkQueue = functions.https.onRequest(async (req, res) => {
// Consistent timestamp
const now = admin.firestore.Timestamp.now();
// Check which tasks are due
const query = db.collection('tasks').where('duedate', '<=', new Date()).where('status', '==', 'scheduled');
const tasks = await query.get();
// Process tasks and mark it in queue as done
tasks.forEach(snapshot => {
const { type, data } = snapshot.data();
console.info('Executing job for task ' + JSON.stringify(type) + ' with data ' + JSON.stringify(data));
const job = workers[type](data)
// Update task doc with status or error
.then(() => snapshot.ref.update({ status: 'complete' }))
.catch((err) => {
console.error('Error when executing worker', err);
return snapshot.ref.update({ status: 'error' });
});
jobs.push(job);
});
return Promise.all(jobs).then(() => {
res.send('ok');
return true;
}).catch((onError) => {
console.error('Error', onError);
});
});
You have different options to trigger the checking of the queue if there is a task that is due:
Using a http callable function as in the example above. This requires you to perform a http call to this function regularly so it executes and checks if there is a task to be done. Depending on your needs you could do it from an own server or use a service like cron-job.org to perform the calls. Note that the HTTP callable function will be available publicly and potentially, others could also call it. However, if you make your check code idempotent, it shouldn't be an issue.
Use the Firebase "internal" cron option that uses Cloud Scheduler internally. Using that you can directly trigger the queue checking:
export scheduledFunctionCrontab =
functions.pubsub.schedule('* * * * *').onRun((context) => {
console.log('This will be run every minute!');
// Include code from checkQueue here from above
});
Using such a queue also makes your system more robust - if something goes wrong in between, you will not loose tasks that would somehow only exist in memory but as long as they are not marked as processed, a fixed worker will pick them up and reprocess them. This of course depends on your implementation.
You can trigger a cloud function on the Firestore Create event which will schedule the Cloud Task after 30 minutes. This will have queuing and retrying mechanism.
An easy way is that you could add a created field with a timestamp, and then have a scheduled function run at a predefined period (say, once a minute) and execute certain code for all records where created >= NOW - 31 mins AND created <= NOW - 30 mins (pseudocode). If your time precision requirements are not extremely high, that should work for most cases.
If this doesn't suit your needs, you can add a Cloud Task (Google Cloud product). The details are specified in this good article.
i want to send a notification for my users and I need check users states everyday and each user that get this situation send him/her notif.
for this purpose i use agenda.js library.
when I run this code i want to create a job Once and after that repeat call this job. so for this i write this code:
const job = await agenda.create(SCHEDULE_MISS_YOU_NOTIF, data)
job.repeatEvery('40 16 * * *', { timezone: 'Asia/Tehran' })
await job.save()
but my problem is every time i run this code, immediately i get a notif and after that i get notif on time that scheduled. how can i remove first one notif and why this notif occure?
job.repeatEvery('40 16 * * *', {
skipImmediate: true,
timezone: 'Asia/Tehran'
});
I've written a simple service using redis to store data in memory or fetch from disc and then store in memory. I'm trying to now control for rare cases where fetching to redis is slow. I've seen one example (https://gist.github.com/stockholmux/3a4b2d1480f27df8be67#file-timelimitedredis-js) which appears to solve this problem but I've had trouble implementing.
The linked implementation is:
/**
* returns a function that acts like the Redis command indicated by cmd except that it will time out after a given number of milliseconds
*
* #param {string} cmd The redis commmand to execute ('get','hset','sort', etc.)
* #param {integer} timeLimit The number of milliseconds to wait until returning an error to the callback.
*
*/
function timeLimited(cmd, timeLimit) {
return function() {
var
argsAsArr = Array.prototype.slice.call(arguments),
cb = argsAsArr.pop(),
timeoutHandler;
timeoutHandler = setTimeout(function(){
cb(new Error('Redis timed out'));
cb = function() {};
}, timeLimit);
argsAsArr.push(function(err, values){
clearTimeout(timeoutHandler);
cb(err,values);
});
client[cmd].apply(client,argsAsArr);
};
}
however I don't understand how to implement this because client is never defined and the the redis key/value are never passed in. Could someone explain a little about how one could go about implementing this example? I've been searching for more information or a working example but not had any luck so far. Thank you.
This isn't very clearly written but when you call it with cmd (eg. SET, HSET, etc) and time limit it returns a function. You call this returned function with the values. I don't know where client comes from, I guess you need to have it in scope. This isn't very good code, I would suggest posting what you've written and asking how to achieve what you want with that.
I have node application in which I want to run tasks on daily basis. So I want to use node-cron to schedule tasks but not like this:
var cron = require('node-cron');
cron.schedule('* * * * *', function(){
console.log('running a task every minute');
});
I need some generic solution so that at one place I have just empty body for cron job that takes different functions from a different location. That means in total we have two files, one which has just cron job function handler and other have a list of cron jobs. From where we want to run these jobs we just need require and some basic calling.
Can anyone suggest me this solution?
I got my solution. Using node-cron-job module I have achieved what I wanted. Let me explain in detail:-
First i created new file which contains jobs, lets name it jobs.js :-
// jobs.js
exports.first_job = {
on:"* * * * * " //runs every minute
},
job: function () {
console.log("first_job");
},
spawn: true
}
exports.second_job = {
on: "*/2 * * * * *", //runs every 2 second
job: function () {
console.log("second_job");
},
spawn: false // If false, the job will not run in a separate process.
}
Here, we define two jobs name first_job and second_job. We can define as many required.
Finally, we need to call these jobs from any location by these simple steps:-
// main.js
var cronjob = require('node-cron-job');
cronjob.setJobsPath(__dirname + '/jobs.js'); // Absolute path to the jobs module.
cronjob.startJob('first_job');
cronjob.startJob('second_job');
We can call all jobs in a single call like this:-
cronjob.startAllJobs();