I am planning for a nodeJS Application where the server has to handle time-based jobs. For example:
Job 1: Every 2 hours
Job 2: Every 5 minutes
Job 3: Every 10 days
...
These jobs are saved in a database and there will be a lot of jobs - it could be a millions of these.
So now is my question which is the right way in nodeJS to handle this ? There are a lot cron packages for cron like processes but I thought its maybe "to much" for this.
I was thinking about .... Timeouts! Right - timeouts.
This is the schema I was thinking of:
var job = getJob();
var executeInMs = getMSUntilExecute(job);
setTimeout(function(){handle();}, executeInMs);
With this schema it really could be that there are a million timeouts running at the same time in my application.
The question now is: Is this a Server-Killer or is this the right way to handle this ?
Related
I have a website that is live. I have a cron job that executes every 24 hours. the cron job fetches and analyzes the data from a database table.
The problem is that the website gets very slow during the time when cron job is running. And gets back to normal after that. It gives me error Too many connections during this time.
I set the maximum allowed connections to 500 in mysql. The number of active connections that I checked in mysql were less than limit during that time.
I am unable to find any relevant help or even a clue to think in a particular direction.
Update:
I noticed one thing. the number of mysql connection continuously increases in this time. Although still less than the maximum limit.
nice command can change priority of a process. You want to lower the priority of the background process so it will try not to execute be executing while the website is being busy. E.g.
0 3 * * * nice -n 20 myjob arg arg
to execute myjob arg arg with lowered priority every day at 3am.
EDIT: Although, if the job is spending most of its time in database queries, this will not affect it much. MySQL has LOW_PRIORITY flag for INSERT and UPDATE statements that will do kind of the same thing for those queries.
This question already has answers here:
Schedule query in BigQuery
(3 answers)
Closed 4 years ago.
What I want to do is run a daily query and insert that daily data into another table.
I am a beginner to big query and the jobs part confuses me a bit since it cannot be done through the GUI.
I read here that it should be done programmatically.
https://cloud.google.com/bigquery/docs/jobs-overview
The problem here is that this all looks like one time jobs which cannot be automatically rescheduled. Is there any way to do this in nodejs or do I need to setup a chronjob or something to use for scheduling?
Does anybody have a good example or some pointers to get me started correctly.
Thanks
You can schedule a job in node with something like this:
node-cron
For this the node server will have to be running all time so that it can execute the job at the required time.
Alternatively you can create a cron job that calls a script which would perform the job.
My platform is nodejs and started using kue as the queue engine. It works perfectly so far and I'm just curious the max job id it allows, as I have billions of jobs per day to start with.
it use Number for id so i think it's Number.MAX_SAFE_INTEGER = 9007199254740991 or maybe Number.MAX_VALUE
I currently do service using beanstalkd and node.js.
I would like when jobs fail, retry n time before give up the job.
If the job succede i want do it the same job 10 time.
So, what is the best practice, stock in mongo db with the jobId the error and success count, or delete and put a new job with a an error and success count in the body.
I dont know if i'm clear? so tell me , thanks a lot
There is a stats-job <id>\r\n that should also be available via the API library that returns, among other things, how many times the specific job has been reserved, released, buried, and so on.
This allows for a number of retries of failed jobs by checking previous reservation/releases.
To run the same job multiple times, I would personally create either one additional job, with a success count that would then be incremented (into another new job) - or, all nine new jobs, with optional delays before they start.
You have a couple of ways to do this:
you can release the job, and obtain from stats the number of reserves
you can put a new job with a retry count, and keep track of history in the data payload
You should do the later, and you don't need MongoDB as a second dependency.
I am recently implementing a system that automatically replies to tweets that contain arbitrary hashtags. This system consists of a process that periodically crawls Twitter and a process that periodically replies to these tweets. Following the tradition of my company, these periodical jobs are implemented with working tables on RDMS that have a status column which would have values like "waiting", "processing" or "succeed". To ensure redundancy, I make multiple same processes running by leveraging low level locks.
My question is, I'm implementing periodically jobs with working tables in RDMS, how these jobs are implemented in generally.
There's a node package cron which allows you to execute code at some specified interval, just like crontab. Here's a link to the package: https://www.npmjs.org/package/cron
For example:
var cronJob = require("cron").CronJob;
// Run this cron job every Sunday (0) at 7:00:00 AM
new cronJob("00 00 7 * * 0", function() {
// insert code to run here...
}, null, true);
You might be able to use that module to run some job periodically, which crawls twitter or replies to tweets.