I use setInterval and later node-schedule to create a scheduled call every hour or so, but it looks like the call typically gets canceled after a few days.
const { scheduleJob } = require('node-schedule');
for (let i = 0; i < 24; ++i)
scheduleJob(`0 ${i} * * *`, tensorFlowAPIs);
This is the node-schedule version.
setInterval(tensorFlowAPIs, 60 * 60 * 1000);
And this is the setInterval version.
tensorFlowAPIs is essentially just an async request.
Does Node.js cancel setInterval automatically?
No. setInterval() does not automatically get cancelled after some long duration of time.
If your program is stopped and then restarted, you would have to manually restart the setInterval() to keep it going.
Often issues with long running programs are due to a resource leak of some kind (file handles, memory leak or some other system resource) that causes the process to not be able to do its job after a long while. Obviously the best way to avoid this is to have a perfect program that never leaks any important resource, but one way that programmers handle slow, hard-to-detect leaks over time (particularly if they may be in library code that the developer isn't familiar with or occur only in certain hard to simulate error conditions) is to restart their process every so often (once a day, perhaps). That clears out the process of any leak objects and starts afresh.
Related
If I understand correctly, setInterval(() => console.log('hello world'), 1000) will place the function to some queue of tasks to run. But if there are other tasks in-front of it, it won't run exactly at 1000 millisecond or every time.
In a single complex program, is it possible to also make calls to some function every n millisecond exactly in real world time with node.js ?
If I understand correctly, setInterval(() => console.log('hello world'), 1000) will place the function to some queue of tasks to run. But if there are other tasks in-front of it, it won't run exactly at 1000 millisecond or every time.
That is correct. It won't run exactly at the desired time if node.js happens to be busy doing something else when the timer is ready to run. node.js will wait until it finishes it's other task before running the timer callback. You can think of node.js as if it has a one-track mind (can only do one thing at a time) and timers don't ever interrupt existing tasks that are running.
In a single complex program, is it possible to also make calls to some function every n millisecond exactly in real world time with node.js ?
No, it is not possible to do that in node.js. node.js runs your Javascript as single-threaded, it's event driven and not-preemptive. All of these mean that you cannot rely on code running at a precise real-world time.
What happens under the covers in node.js is that you set a timer for a specific time in the future. That timer goes is registered with the node.js event loop so that each time it gets through the event loop, it will check if there are any pending timers. But, it only gets through the event loop when other code that was running before the timer was ready to fire finishes running. Here's the sequence of events:
Run some code
Set timer for some time in the future (say time X)
Run some more code
Nothing to do for awhile
Run some more code (while this code is running, time X passes - the time for your timer to run)
Previous block of code finishes running and control returns back to the node.js event loop at time X + n (some time after the timer X was supposed to fire).
Event loop checks to see if there are any pending timers. It finds a timer and calls its callback at time X + n.
So, the only way that your timer gets called at approximately time X is if node.js has nothing else to do at exactly time X. If your program is ever doing anything else, you can't guarantee that your program will be free at exactly time X to run the timer exactly when you want it to run. node.js is NOT a real-time system in any way. single-threaded and non-pre-emptive mean that a timer may have to wait for node.js to finish some other things before it gets to run and thus there is no guarantee that the timer will run exactly on time. Instead, it will run as not before time X when the interpreter is next free to return back to the event loop (done running whatever else might have been running at the time). This could be close to time X or it could be a significant time after time X.
If you really need something to run precisely at a specific time, then you likely need a pre-emptive system (not node.js) that is much more real-time than node.js is.
You could create a "work-around" in node.js by firing up another node.js process (you could use the child_process module) and start a program in that other process that has nothing else to do except serve your timer and execute the code associated with that timer. Then, at least you timer won't be pre-empted by some other Javascript task that might be running and will get to run pretty close to the desired time. Keep in mind that even this work-around still isn't a true real-time system, but might serve some purposes.
Otherwise, you probably want to write this in a more real-time system language that has pre-emptive timers (probably even with thread priorities).
But if there are other tasks in-front of it, it won't run exactly at 1000 millisecond or every time.
Your question is actually operating system specific, assuming the computer is running some (usual) operating system (like Windows, Android, Linux, MacOSX, etc...). I recommend reading Operating Systems: Three Easy Pieces to learn more.
In practice, your computer has many other processes managed by its operating system. Some of them might be running. Your computer might be in a situation where it is loaded enough by other processes to the point of not being able to run your tasks or threads exactly every second. Read about thrashing.
You might want to use some genuine real-time operating system. But then, node.js probably won't run on it.
How to call a function every n milliseconds in “real world” time exactly?
You cannot do that reliably. Because your node.js process (it is actually single threaded, at the system threads level, see pthreads(7) and jfriend00's answer) might not get enough resources from your OS (so if other processes are loading your computer too much, node.js would be starved and won't be able to progress like you want; be also aware of possible priority inversions).
On Linux, see also shed(7), chrt(1), renice(1)
I suggest to make a cron which will run at every n seconds. If your program is complex and it may take more time then you can go with async.
npm install cron
var CronJob = require('cron').CronJob;
new CronJob('* * * * * *', function() {
console.log('You will see this message every second');
callYourFunc();
}, null, true, 'America/Los_Angeles');
For more read this link
Perhaps you could spawn a worker thread and block it while it’s waiting to do the work, in the way suggested by CertainPerformance in the comments. It may not be the most elegant way to do it but at least you can put the blocking logic aside so that it doesn’t affect the rest of the application.
Check out the example in the docs if you’re unfamiliar with the cluster module: https://nodejs.org/docs/latest-v10.x/api/cluster.html
I'm currently developing a gambling website built on top of Ethereum blockchain. Since recording all the bets made by a gambler is very complex (because they can make a bet without even visiting the website, by interacting directly with the blockchain) I came to conclusion that I need a function on my server that will run every 0.5 - 1 minute and download all the new bets that came up from the blockchain and shadow them in my database (yes I need to have them in my database as well).
I am not experienced too much with all this backend stuff, I've read somewhere that I could use setInterval(30 seconds) function on the server and run it on the server start. But is this a real option? Do people even do things like this? Won't an infinite function running every 30 second just clog up the whole server?
I've done similar a number of times with no issue. Events will just be queued and run as appropriate. However, one thing to be wary of:
const timeout = 1000;
setInterval(() => {
// some process that takes longer than timeout
}, timeout);
When you write something like this, setInterval will run again after the timeout period even if the first operation hasn't completed yet. THIS can easily lead to you blocking your thread. Instead, I prefer to use setTimeout to achieve this:
const timeout = 1000;
const withTimeout = () => {
// some process that takes longer than timeout
setTimeout(withTimeout, 1000);
}
This way, your second invocation of withTimeout is only queued up AFTER the first execution is run. With this mechanism, you don't get your operations strictly every timeout period, but rather timeout AFTER the last one.
Solved : It's a node bug. Happens after ~25 days (2^32 milliseconds), See answer for details.
Is there a maximum number of iterations of this cycle?
function do_every_x_seconds() {
//Do some basic things to get x
setTimeout( do_every_x_seconds, 1000 * x );
};
As I understand, this is considered a "best practice" way of getting things to run periodically, so I very much doubt it.
I'm running an express server on Ubuntu with a number of timeout loops .
One loop that runs every second and basically prints the timestamp.
One that calls an external http request every 5 seconds
and one that runs every X 30 to 300 seconds.
It all seemes to work well enough. However after 25 days without any usage, and several million iterations later, the node instance is still up, but all three of the setTimout loops have stopped. No error messages are reported at all.
Even stranger is that the Express server is still up, and I can load http sites which prints to the same console as where the periodic timestamp was being printed.
I'm not sure if its related, but I also run nodejs with the --expose-gc flag and perform periodic garbage collection and to monitor that memory is in acceptable ranges.
It is a development server, so I have left the instance up in case there is some advice on what I can do to look further into the issue.
Could it be that somehow the event-loop dropped all it's timers?
I have a similar problem with setInterval().
I think it may be caused by the following bug in Node.js, which seem to have been fixed recently: setInterval callback function unexpected halt #22149
Update: it seems the fix has been released in Node.js 10.9.0.
I think the problem is that you are relying on setTimeout to be active over days. setTimeout is great for periodic running of functions, but I don't think you should trust it over extended time periods. Consider this question: can setInterval drift over time? and one of its linked issues: setInterval interval includes duration of callback #7346.
If you need to have things happen intermittently at particular times, a better way to attack this would be to schedule cron tasks that perform the tasks instead. They are more resilient and failures are recorded at a system level in the journal rather than from within the node process.
A good related answer/question is Node.js setTimeout for 24 hours - any caveats? which mentions using the npm package cron to do task scheduling.
This is my objective:
5 minutes after the last document update, I want to execute a one-time task
So this is what the flow of actions will look like:
User updates document - timer is started and counts down from 5 minutes
If user updates the same document again and the previous timer (identified by the document._id) is running still, reset that timer back to 5 minutes and repeat countdown
When timer has elapsed - the one time task is executed
I have not done something like this before and I am at a loss at where to begin.
I can hook into document changes easily using methods available in Mongoose (i.e. on save, do func to setup or reset timer)
However, I cannot figure out the way to:
create a timer that waits 5 minutes - then executes a task
making the timer accessible so that I can reset the timer
making the timer accessible so I can add extra variables which will be used in the funtion when timer has elapsed
I've investigated Cron jobs but they seem to tasks that schedule at the same time everyday.
I need a timer that delays a task, but also the ability to reset that timer, and add extra data to the timer.
Any hints appreciated.
This is the solution I managed to complete up with.
First of all, it's worth noting that my original assumptions are correct:
Cron jobs are great for repetitive, tasks that are scheduled at the same time everyday. However, for tasks that are created on the fly, and have a countdown element, then cron jobs isn't the right tool.
However, enter node-schedule (https://www.npmjs.com/package/node-schedule)
Node Schedule is a flexible cron-like and not-cron-like job scheduler for Node.js. It allows you to schedule jobs (arbitrary functions) for execution at specific dates, with optional recurrence rules. It only uses a single timer at any given time (rather than reevaluating upcoming jobs every second/minute).
The use of timers is really what these, on-the-fly tasks need.
So my solution looks like this:
var schedule = require('node-schedule'),
email_scheduler = {};
email_scheduler["H1e5nE2GW"] = schedule.scheduleJob(Date.now() + 10000, function(){
console.log('its been a total of 10 seconds since initalization / reset.');
});
var second_attempt = schedule.scheduleJob(Date.now() + 5000, function(){
console.log("5 seconds elapsed from start, time to reset original scheduled job");
email_scheduler["H1e5nE2GW"].cancel();
email_scheduler["H1e5nE2GW"] = schedule.scheduleJob(Date.now() + 10000, function(){
console.log('its been a total of 10 since last one seconds.');
});
schedule.scheduleJob(Date.now() + 5000, function(){
console.log("it's been another 5 seconds since reset");
});
});
My thinking (though not yet tested) is that I can create a singleton-like instance for the email_scheduler object by creating a node module. Like such:
// node_modules/email-scheduler/index.js
module.exports = {};
This way, I can access the scheduleJobs and reset the timers in every file of the node application.
You can use events for that. call a event with some parameter and before your task add sleep and exec your task. But add sleep or settimeout is not a good idea in nodejs.
I had the same puzzle but I ended settling on the javascript native functions.
function intervalFunc() {
//inside this function you can request to get the date and time then decide what to do and how to do it
autogens.serverAutoGenerate();//my function to autogenerate bills
}
setInterval(intervalFunc, 1000);//repeat task after one seconds
Adding the code at the index file or entry file of your node js application will make the function repeatedly execute. You can opt to change the frequency depending on your needs. If you wanted to run the cronjob three times a day, get the number of seconds in 24 hours then divide by 3. If you wanted to run the job on a specific hour, by using the new Date(), you can check to determine if it is the right hour to execute your event or not.
I would like to add a one hour timeout to this process so that it will not stay forever in the case of a stream deadlock. The only problem is if I say setTimeout, the process has no opportunity to end ahead of schedule.
Is there a way to put in a forced exit timeout after about an hour, without keeping the process running? Or am I stuck between using process.exit and doing without this timeout?
I don't know when unref was added to Node but this is now one possible solution. Reusing Matt's code:
var timeoutId = setTimeout(callback, 3600000);
timeoutId.unref(); // Now, Node won't wait for this timeout to complete if it needs to exit earlier.
The doc says:
In the case of setTimeout when you unref you create a separate timer that will wakeup the event loop, creating too many of these may adversely effect event loop performance -- use wisely.
Don't go hog wild with it.
If you save the value returned by setTimeout you can always cancel it before it fires with clearTimeout, like this:
var timeoutId = setTimeout(callback, 3600000); // 1 hour
// later, before an hour has passed
clearTimeout(timeoutId);
Possible Solution using a new feature that was implemented as process.watchers(), but I don't know whether it was included in a released version yet. The pull request is still open as of this post.
But generally speaking, you would write a custom setTimeout function that add all timeouts to a counter, and right before the timeouts to the callback, it would remove from the counter.
Then create an ongoing interval, and the interval would check and notice when all the walkers are just timeouts set by your custom function.
When it notices this, it would (clear all of its those timeouts, causing an) exit.
This of course is not great, because
The concept is not proven. (I do not know what kind of data will be returned by process.watchers(), or if it even works yet)
The interval is a polling method. (not nice if you want immediate return at the same time as low CPU usage)