I'm currently developing a gambling website built on top of Ethereum blockchain. Since recording all the bets made by a gambler is very complex (because they can make a bet without even visiting the website, by interacting directly with the blockchain) I came to conclusion that I need a function on my server that will run every 0.5 - 1 minute and download all the new bets that came up from the blockchain and shadow them in my database (yes I need to have them in my database as well).
I am not experienced too much with all this backend stuff, I've read somewhere that I could use setInterval(30 seconds) function on the server and run it on the server start. But is this a real option? Do people even do things like this? Won't an infinite function running every 30 second just clog up the whole server?
I've done similar a number of times with no issue. Events will just be queued and run as appropriate. However, one thing to be wary of:
const timeout = 1000;
setInterval(() => {
// some process that takes longer than timeout
}, timeout);
When you write something like this, setInterval will run again after the timeout period even if the first operation hasn't completed yet. THIS can easily lead to you blocking your thread. Instead, I prefer to use setTimeout to achieve this:
const timeout = 1000;
const withTimeout = () => {
// some process that takes longer than timeout
setTimeout(withTimeout, 1000);
}
This way, your second invocation of withTimeout is only queued up AFTER the first execution is run. With this mechanism, you don't get your operations strictly every timeout period, but rather timeout AFTER the last one.
Related
I use setInterval and later node-schedule to create a scheduled call every hour or so, but it looks like the call typically gets canceled after a few days.
const { scheduleJob } = require('node-schedule');
for (let i = 0; i < 24; ++i)
scheduleJob(`0 ${i} * * *`, tensorFlowAPIs);
This is the node-schedule version.
setInterval(tensorFlowAPIs, 60 * 60 * 1000);
And this is the setInterval version.
tensorFlowAPIs is essentially just an async request.
Does Node.js cancel setInterval automatically?
No. setInterval() does not automatically get cancelled after some long duration of time.
If your program is stopped and then restarted, you would have to manually restart the setInterval() to keep it going.
Often issues with long running programs are due to a resource leak of some kind (file handles, memory leak or some other system resource) that causes the process to not be able to do its job after a long while. Obviously the best way to avoid this is to have a perfect program that never leaks any important resource, but one way that programmers handle slow, hard-to-detect leaks over time (particularly if they may be in library code that the developer isn't familiar with or occur only in certain hard to simulate error conditions) is to restart their process every so often (once a day, perhaps). That clears out the process of any leak objects and starts afresh.
If I understand correctly, setInterval(() => console.log('hello world'), 1000) will place the function to some queue of tasks to run. But if there are other tasks in-front of it, it won't run exactly at 1000 millisecond or every time.
In a single complex program, is it possible to also make calls to some function every n millisecond exactly in real world time with node.js ?
If I understand correctly, setInterval(() => console.log('hello world'), 1000) will place the function to some queue of tasks to run. But if there are other tasks in-front of it, it won't run exactly at 1000 millisecond or every time.
That is correct. It won't run exactly at the desired time if node.js happens to be busy doing something else when the timer is ready to run. node.js will wait until it finishes it's other task before running the timer callback. You can think of node.js as if it has a one-track mind (can only do one thing at a time) and timers don't ever interrupt existing tasks that are running.
In a single complex program, is it possible to also make calls to some function every n millisecond exactly in real world time with node.js ?
No, it is not possible to do that in node.js. node.js runs your Javascript as single-threaded, it's event driven and not-preemptive. All of these mean that you cannot rely on code running at a precise real-world time.
What happens under the covers in node.js is that you set a timer for a specific time in the future. That timer goes is registered with the node.js event loop so that each time it gets through the event loop, it will check if there are any pending timers. But, it only gets through the event loop when other code that was running before the timer was ready to fire finishes running. Here's the sequence of events:
Run some code
Set timer for some time in the future (say time X)
Run some more code
Nothing to do for awhile
Run some more code (while this code is running, time X passes - the time for your timer to run)
Previous block of code finishes running and control returns back to the node.js event loop at time X + n (some time after the timer X was supposed to fire).
Event loop checks to see if there are any pending timers. It finds a timer and calls its callback at time X + n.
So, the only way that your timer gets called at approximately time X is if node.js has nothing else to do at exactly time X. If your program is ever doing anything else, you can't guarantee that your program will be free at exactly time X to run the timer exactly when you want it to run. node.js is NOT a real-time system in any way. single-threaded and non-pre-emptive mean that a timer may have to wait for node.js to finish some other things before it gets to run and thus there is no guarantee that the timer will run exactly on time. Instead, it will run as not before time X when the interpreter is next free to return back to the event loop (done running whatever else might have been running at the time). This could be close to time X or it could be a significant time after time X.
If you really need something to run precisely at a specific time, then you likely need a pre-emptive system (not node.js) that is much more real-time than node.js is.
You could create a "work-around" in node.js by firing up another node.js process (you could use the child_process module) and start a program in that other process that has nothing else to do except serve your timer and execute the code associated with that timer. Then, at least you timer won't be pre-empted by some other Javascript task that might be running and will get to run pretty close to the desired time. Keep in mind that even this work-around still isn't a true real-time system, but might serve some purposes.
Otherwise, you probably want to write this in a more real-time system language that has pre-emptive timers (probably even with thread priorities).
But if there are other tasks in-front of it, it won't run exactly at 1000 millisecond or every time.
Your question is actually operating system specific, assuming the computer is running some (usual) operating system (like Windows, Android, Linux, MacOSX, etc...). I recommend reading Operating Systems: Three Easy Pieces to learn more.
In practice, your computer has many other processes managed by its operating system. Some of them might be running. Your computer might be in a situation where it is loaded enough by other processes to the point of not being able to run your tasks or threads exactly every second. Read about thrashing.
You might want to use some genuine real-time operating system. But then, node.js probably won't run on it.
How to call a function every n milliseconds in “real world” time exactly?
You cannot do that reliably. Because your node.js process (it is actually single threaded, at the system threads level, see pthreads(7) and jfriend00's answer) might not get enough resources from your OS (so if other processes are loading your computer too much, node.js would be starved and won't be able to progress like you want; be also aware of possible priority inversions).
On Linux, see also shed(7), chrt(1), renice(1)
I suggest to make a cron which will run at every n seconds. If your program is complex and it may take more time then you can go with async.
npm install cron
var CronJob = require('cron').CronJob;
new CronJob('* * * * * *', function() {
console.log('You will see this message every second');
callYourFunc();
}, null, true, 'America/Los_Angeles');
For more read this link
Perhaps you could spawn a worker thread and block it while it’s waiting to do the work, in the way suggested by CertainPerformance in the comments. It may not be the most elegant way to do it but at least you can put the blocking logic aside so that it doesn’t affect the rest of the application.
Check out the example in the docs if you’re unfamiliar with the cluster module: https://nodejs.org/docs/latest-v10.x/api/cluster.html
I created a discord bot using Node.js and the discord.js-commando framework.
One of the features is to create writing sprints, which essentially is a timer, so you could say: I want to write for 20 minutes, starting in 5 minutes. The bot would then wait 5 minutes and start the sprint, then after 20 minutes it notifies the users doing it that it's ended and waits for wordcounts to come in, then posts the results.
This was working fine when the bot was only on one server, but it's been added to several more recently (78 according to the !stats command, though I don't know how many are actively using it), and since then, it's been very erratic.
Sometimes the sprint never starts, sometimes it never ends, sometimes it ends and then after you post your wordcounts, it never posts the final results.
This is my first ever dabbling with Node.js, so I don't know if I'm doing something wrong. I am doing all of the timers with the setTimeout function.
Here is the command file: link to GitHub
As an example, this is the timeout that is set after a user submits their wordcount, if everyone has now submitted their wordcount, so we can display the results:
msg.say('The word counts are in. Results coming up shortly...');
this.finished = 1;
// Clear original timeout
this.clear();
// Set new one
this.messageTimeout = setTimeout(function() {
obj.finish(msg);
}, 10000);
Where clear is:
clear() {
clearTimeout(this.messageTimeout);
}
Is there something inherently wrong with doing it this way? I know very little about Node.js... Should I perhaps look at doing a cron every minute instead to process sprints? Or could this be a server issue? I am running it on a free EC2 AWS server, but the reports all look okay, no resources are being used at abnormally high levels.
Thanks.
Solved : It's a node bug. Happens after ~25 days (2^32 milliseconds), See answer for details.
Is there a maximum number of iterations of this cycle?
function do_every_x_seconds() {
//Do some basic things to get x
setTimeout( do_every_x_seconds, 1000 * x );
};
As I understand, this is considered a "best practice" way of getting things to run periodically, so I very much doubt it.
I'm running an express server on Ubuntu with a number of timeout loops .
One loop that runs every second and basically prints the timestamp.
One that calls an external http request every 5 seconds
and one that runs every X 30 to 300 seconds.
It all seemes to work well enough. However after 25 days without any usage, and several million iterations later, the node instance is still up, but all three of the setTimout loops have stopped. No error messages are reported at all.
Even stranger is that the Express server is still up, and I can load http sites which prints to the same console as where the periodic timestamp was being printed.
I'm not sure if its related, but I also run nodejs with the --expose-gc flag and perform periodic garbage collection and to monitor that memory is in acceptable ranges.
It is a development server, so I have left the instance up in case there is some advice on what I can do to look further into the issue.
Could it be that somehow the event-loop dropped all it's timers?
I have a similar problem with setInterval().
I think it may be caused by the following bug in Node.js, which seem to have been fixed recently: setInterval callback function unexpected halt #22149
Update: it seems the fix has been released in Node.js 10.9.0.
I think the problem is that you are relying on setTimeout to be active over days. setTimeout is great for periodic running of functions, but I don't think you should trust it over extended time periods. Consider this question: can setInterval drift over time? and one of its linked issues: setInterval interval includes duration of callback #7346.
If you need to have things happen intermittently at particular times, a better way to attack this would be to schedule cron tasks that perform the tasks instead. They are more resilient and failures are recorded at a system level in the journal rather than from within the node process.
A good related answer/question is Node.js setTimeout for 24 hours - any caveats? which mentions using the npm package cron to do task scheduling.
I would like to add a one hour timeout to this process so that it will not stay forever in the case of a stream deadlock. The only problem is if I say setTimeout, the process has no opportunity to end ahead of schedule.
Is there a way to put in a forced exit timeout after about an hour, without keeping the process running? Or am I stuck between using process.exit and doing without this timeout?
I don't know when unref was added to Node but this is now one possible solution. Reusing Matt's code:
var timeoutId = setTimeout(callback, 3600000);
timeoutId.unref(); // Now, Node won't wait for this timeout to complete if it needs to exit earlier.
The doc says:
In the case of setTimeout when you unref you create a separate timer that will wakeup the event loop, creating too many of these may adversely effect event loop performance -- use wisely.
Don't go hog wild with it.
If you save the value returned by setTimeout you can always cancel it before it fires with clearTimeout, like this:
var timeoutId = setTimeout(callback, 3600000); // 1 hour
// later, before an hour has passed
clearTimeout(timeoutId);
Possible Solution using a new feature that was implemented as process.watchers(), but I don't know whether it was included in a released version yet. The pull request is still open as of this post.
But generally speaking, you would write a custom setTimeout function that add all timeouts to a counter, and right before the timeouts to the callback, it would remove from the counter.
Then create an ongoing interval, and the interval would check and notice when all the walkers are just timeouts set by your custom function.
When it notices this, it would (clear all of its those timeouts, causing an) exit.
This of course is not great, because
The concept is not proven. (I do not know what kind of data will be returned by process.watchers(), or if it even works yet)
The interval is a polling method. (not nice if you want immediate return at the same time as low CPU usage)