Recently I came across code that makes me wonder about how Node.js engine works. Specifically in regards to event loop, for loop, and asynchronous processing. For example:
const x = 100000000; // or some stupendously large number
for (let i = 0; i < x; i++) {
asyncCall(callback);
}
const callback = () => {
console.log("done one");
}
Let's say asyncCall takes anywhere from 1ms to 100ms. Is there a possibility that console.log("done one") will be called before the for loop finishes?
To try to find my own answer, I've read this article: https://blog.sessionstack.com/how-javascript-works-event-loop-and-the-rise-of-async-programming-5-ways-to-better-coding-with-2f077c4438b5. But I'm not sure if there is a case where the call stack will be empty in the middle of the for loop so that the event loop puts the callback in between asyncCall calls?
Related
I have a simple scenario of:
process.on("SIGINT", s => {
console.log('trapped sigint:', s);
signals.INT = true;
});
let i = 0;
while(true){
if(signals.INT){
console.log('got the SIGINT'); break;
}
console.log('process.pid:',process.pid, 'next val:', i++);
}
console.log('exited loop, done');
process.exit(0);
the problem is that once we are in the synchronous loop, we can't seem to handle any i/o events, even signals. Is there any way around this?
The only solution I can think of, while maintaining a synchronous loop, is to actually go into the guts of node and look for i/o events ourselves from the synchronous loop, but I severely doubt that this is exposed to users of the runtime.
Obviously, one way around this is to create an async loop, like so:
process.on("SIGINT", s => {
console.log('trapped sigint:', s);
signals.INT = true;
});
let i = 0;
(async () => {
while(true){
if(signals.INT){
console.log('got the SIGINT'); break;
}
await console.log('process.pid:',process.pid, 'next val:', i++);
}
})()
console.log('exited loop, done');
process.exit(0);
that makes the loop a lot slower but allows us to capture i/o events. Is there any middle ground? I would love to be able to keep synchronous loop but still listen for i/o events somehow. I thought signals might be able to solve it, but I understand why even signals are not an exception in Node.js / JavaScript.
Why don t you create an async loop externaly of your main loop that set a value to a variable that is global to both loop.
So when the loop run for one more loop it set the variable to the correct value?
I am new to node.js and little bit confused on understanding the event-loop. As far as i know from https://github.com/nodejs/node/blob/master/doc/topics/event-loop-timers-and-nexttick.md, the event-loop phases only process setTimeout, setInterval, setImmediate, process.nextTick, promises and some I/O callbacks.
My question is, if i have following code:
for (var i = 0; i < 100000000; i++)
;
in which phase the above code will get executed ?
Regular JavaScript code, like the for loop in your example, is executed before the queues are cleared. The first thing node will do is run your code, and will only call callbacks, timeout results, I/O results, and so on after your code finishes.
As an example, you could try this code:
fs.open('filename', 'r', () => {
console.log('File opened.');
});
for (var i = 0; i < 100000000; i++);
console.log('Loop complete.');
No matter how big or small your loop variable, 'Loop complete' will always appear before 'File opened'. This is because with only one thread, node can't run the callback you've supplied to the fs.open function until the loop code has finished.
Remember that there isn't a "main" thread that node keeps going back to. Most long-running node programs will run through the code in main.js pretty quickly, and subsequent code is all going to come from callbacks. The purpose of the initial execution is to define how and when those callbacks happen.
In the node event loop doc (https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick), the following code is given as an example:
const fs = require('fs');
function someAsyncOperation(callback) {
// Assume this takes 95ms to complete
fs.readFile('/path/to/file', callback);
}
const timeoutScheduled = Date.now();
setTimeout(() => {
const delay = Date.now() - timeoutScheduled;
console.log(`${delay}ms have passed since I was scheduled`);
}, 100);
// do someAsyncOperation which takes 95 ms to complete
someAsyncOperation(() => {
const startCallback = Date.now();
// 10ms loop
while (Date.now() - startCallback < 10) {
// do nothing
}
});
The loop keeps scanning according to phases and after fs.readFile() finishes, the poll queue is is empty, so its callback will be added and immediately executed. The callback holds a blocking 10ms loop before the timer is executed. That is why the delay will display:
105ms have passed since I was scheduled instead of the 100ms you might expect.
Most of your code will live in callbacks so will be executed in the poll phase. If not, like in your example, it will be executed before entering any phases as it will block the event loop.
The caveat are callbacks scheduled by setImmediate that will enter the check phase before resuming the poll phase in the next loop.
I am experimenting with the event loop. First I begin with this straightforward code to read and print the contents of a file:
var fs = require('fs');
var PATH = "./.gitignore";
fs.readFile(PATH,"utf-8",function(err,text){
console.log("----read: "+text);
});
Then I place it into an infinite loop. In this case, the readFile function is never executed. If I am not mistaken it's because Node's single thread is busy iterating without letting I/O calls be executed.
while(true){
var fs = require('fs');
var PATH = "./.gitignore";
fs.readFile(PATH,"utf-8",function(err,text){
console.log("----read: "+text);
});
}
So, I would like to do something so that I/O calls are assigned process time intertwined with the loop. I tried with process.nextTick() but it doesn't work:
while(true){
process.nextTick(function(){
fs.readFile(PATH,"utf-8",function(err,text){
console.log("----read: "+text)
});
});
}
Why isn't it working and how could I make it?
Because your while loop is still running. It's just infinitely adding things to do in the next tick. If you let it go, your node process will crash as it runs out of memory.
When you work with async code, your normal loops and control structures tend to trip you up. The reason is that they execute synchronously in one step of the event loop. Until something happens that yields control to the event loop again, nothing 'nextTick' will happen.
Think of it like this, You are in Pass B of the event loop when your code runs. When you call
process.nextTick(function foo() { do.stuff(); })'
you are adding the foo to the list of 'things to do before you start pass C of the event loop.' Every time you call nextTick, you add one more thing to the list, but none of them will run until the synchronous code is done.
What you need to do instead is create 'do the next thing' links in your callbacks. Think linked-lists.
// var files = your list of files;
function do_read(count) {
var next = count+1;
fs.readFile(files[count], "utf-8", function(err,text) {
console.log("----read: " + text);
if (next < files.length) {
// this doesn't run until the previous readFile completes.
process.nextTick(function() { do_read(next) });
}
});
}
// kick off the first one:
do_read(files[0], 0);
(obviously this is a contrived example, but you get the idea)
This causes each 'next file' to be added to the 'nextTick' to-do queue only after the previous one has been fully processed.
TL;DR: Most of the time, you don't want to start it doing the next thing until the previous thing is completed
Hope that helps!
I am trying to run a function in a async way, but I always recive the result in the order that i call them instead of receive the quickest function.
This is an example. The expected result is to print first 100 even it is called last.
function f1(callback){
i = 0;
for (i;i< 100;i++){
}
if (i == 100){
return callback(i);
}
}
function f2(callback){
i = 0;
for (i;i< 99999999999;i++){
}
if (i == 99999999999){
return callback(i);
}
}
f2(function(i){
console.log(i);
})
f1(function(i){
console.log(i);
});
In node (and in javascript in general), functions are not asynchronous by default.
In your code above, at no point are you yielding control to the javascript's event loop. As you call each function, it runs through its for loop until finished and then calls back which is why you are seeing the output the way you are.
The asynchronous nature of node kicks in when you start doing IO or purposely yield control to the VM and give it a chance to process other events off the event loop.
To see the code above run in an asynchronous fashion for the purposes of your test:
function f1(callback){
setTimeout(callback('f1'), 100);
}
function f2(callback){
setTimeout(callback('f2'), 500);
}
f2(function(i){
console.log(i);
})
f1(function(i){
console.log(i);
});
The setTimeout call basically just says: call back to the provided function as close to n milliseconds as you can (but not before). Basically acting as a sleep statement here and yielding control back to the currently executing code.
I have a massive for loop and I want to allow I/O to continue while I'm processing. Maybe every 10,000 or so iterations. Any way for me to allow for additional I/O this way?
A massive for loop is just you blocking the entire server.
You have two options, either put the for loop in a new thread, or make it asynchronous.
var data = [];
var next = function(i) {
// do thing with data [i];
process.nextTick(next.bind(this, i + 1));
};
process.nextTick(next.bind(this, 0));
I don't recommend the latter. Your just implementing naive time splicing which the OS level process scheduler can do better then you.
var exec = require("child_process").exec
var s = exec("node " + filename, function (err, stdout, stderr) {
stdout.on("data", function() {
// handle data
});
});
Alternatively use something like hook.io to manage processes for you.
Actually you probably want to aggressively redesign your codebase if you have a blocking for loop.
Maybe something like this to break your loop into chunks...
Instead of:
for (var i=0; i<len; i++) {
doSomething(i);
}
Something like:
var i = 0, limit;
while (i < len) {
limit = (i+10000);
if (limit > len)
limit = len;
process.nextTick(function(){
for (; i<limit; i++) {
doSomething(i);
}
});
}
}
The nextTick() call gives a chance for other events to get in there, but it still does most looping synchronously which (I'm guessing) will be a lot faster than creating a new event for every iteration. And obviously, you can experiment with the number (10,000) till you get the results you want.
In theory, you could also use setTimeout() rather than nextTick(), in case it turns out that giving other processes a somewhat bigger "time-slice" helps. That gives you one more variable (the timeout milliseconds) that you can use for tuning.