I'm computing some heavy recursive function (let's say Fibonacci for the following code)
I have two versions :
- first one, vanilla recursive (uncommenting "if(false)" )
- the second one, using the process.nextTick every 5 calls in order to let other code run (uncommenting "if(n%5==0)" )
now running first one gets me a "RangeError: Maximum call stack size exceeded"
while running the second one provides me with a nice "196418"
here's the code. Could you tell me what's going on, because I can't believe that process.nextTick is cleaning the stack.
'use strict';
function fibo_cb( n, cb ){ fibo_rec( n, (res) => cb( res ) ); }
function fibo_rec( n, ret ){
if( n<2 ){
ret( 1 );
} else {
let rr = (res_1) => { return (res_2) => ret(res_1+res_2) };
let r = (res_1) => fibo_rec( n-2, rr(res_1) );
//if(false){ //no tick
if(n%5==0){ //do tick, once every 5
process.nextTick( () => fibo_rec( n-1, r ) );
} else {
fibo_rec( n-1, r );
}
}
}
fibo_cb( 26, console.log );
The JavaScript code is executed by putting all the code in Call Stack by creating scopes at function level. Since JavaScript is single threaded(apart from Asynchronous tasks), starting from Global level each scope is put into the call stack one by one until it reaches all the functions. Once the function gets completed it is popped out of the Call Stack.
In case of Recursive Functions, for each recursive call a function is pushed to Call Stack. So when there are large number of recursive calls the Call Stack gets full since a maximum limit is fixed for Call Stack.
Any Asynchronous Task scheduled using process.nextTick waits till call stack is empty and takes each callback from the Event Queue and pushes it to Call Stack. This happens repeatedly till all the recursive callbacks are finished.
Since Call Stack is emptied each time the is no "RangeError: Maximum call stack size exceeded" observed.
Related
I am new to node.js and very inexpert when it comes to async processing.
I have a list of functions in an array, which I need to call asynchronously when a message arrives. I planned to use process.nextTick to schedule the calls. This is my (simplified) code:
// These are example message handlers
// All have to be called on every message
function fn1( msg ) { console.log( "fn1: " + msg ) };
function fn2( msg ) { console.log( "fn2: " + msg ) };
function fn3( msg ) { console.log( "fn3: " + msg ) };
// the functions to be called are saved in a list
var fns = [ fn1, fn2, fn3 ];
// function to schedule async callbacks to every
// function in the list.
function multicall( msg ){
// schedule the call back to each function
for( var ix = 0; ix < fns.length; ix++ ){
var fn = fns[ix];
process.nextTick( function() {
// call the nth function with the message
fn( msg );
}
);
}
}
// now make the call to schedule callbacks
console.log( "start" );
multicall( "information" );
console.log( "end" );
The calls are queued as expected, but the problem is that when the calls are made, they are all to the last function, instead of one call to each function. Presumably this is because the call to fn( msg ) actually references the value of the variable fn at the time of the call, not at the time that the function is was scheduled. The calls are therefore all made to the last function in the list:
S:\test>node asyncTest
start
end
fn3: information
fn3: information
fn3: information
I have tried various approaches but can't seem to get round this problem of the difference between the function referenced when the call is scheduled, and the function referenced when the call is executed.
EDIT - - - -
Although I accept that #jishi's answer below is better, I inadvertently found another answer to this problem, which is to use let fn = fns[ix]; instead of var fn = fns[ix];. When using let, the for loop creates a completely new variable on each iteration, so the value set in one cycle is not overwritten by the value set in the next cycle. I hadn't previously understood that the scope of a let variable is limited to each specific execution of the code block, not just by the static lexical structure of the code.
That's because the fn variable will be in scope for all of the scheduled calls, and you actually change fn.
You should just send in the reference of the function to the nextTick (and maybe rather use setImmediate()) and bind the variable that you want the function to be executed with, something like this:
process.nextTick(fn.bind(null, msg));
This will schedule the function currently assigned to fn, with null as this reference, and the current value of msg as first argument to the function.
With setImmediate, I think you can do this:
setImmediate(fn, msg)
Because first argument is function, and the following arguments are arguments for the function that will be used when invoking (similar to bind())
In my example, I actually schedule the function as a reference, whereas in your example, you are scheduling an anonymous function that lives in the same scope as multicall(), meaning that any changes in the multicall scope, will affect execution of your anonymous function.
Before bind() existed, you had to sort of declare a function, send in the variables you wanted to be set when execution the function, and then return a new function from that. Very cumbersome, and I argue that should be avoided since the bind() approaches makes it in a much neater way.
Recently I came across code that makes me wonder about how Node.js engine works. Specifically in regards to event loop, for loop, and asynchronous processing. For example:
const x = 100000000; // or some stupendously large number
for (let i = 0; i < x; i++) {
asyncCall(callback);
}
const callback = () => {
console.log("done one");
}
Let's say asyncCall takes anywhere from 1ms to 100ms. Is there a possibility that console.log("done one") will be called before the for loop finishes?
To try to find my own answer, I've read this article: https://blog.sessionstack.com/how-javascript-works-event-loop-and-the-rise-of-async-programming-5-ways-to-better-coding-with-2f077c4438b5. But I'm not sure if there is a case where the call stack will be empty in the middle of the for loop so that the event loop puts the callback in between asyncCall calls?
Do I need to call start function from process.nextTick when start function is already inside process.nextTick? I want to make both the accept and start function asynchronous? What should be the desired way to achieve this?
function accept() { // ACCEPT FUNCTION
process.nextTick(() => {
start(); // START CALL
});
}
function start() { // START FUNCTION
process.nextTick(() => {
startPolling();
});
}
No. If the startPolling() function is blocking, then putting it in nextTick() callback won't help. It will block the entire thread on the next tick of the event loop - but it will block it nonetheless.
Example:
Without nextTick:
function x() {
fs.readFileSync('file');
}
x(); // <-- it blocks here
console.log('...');
With nextTick:
function x() {
pricess.nextTick(() => fs.readFileSync('file'));
}
x();
console.log('...');
// <-- it blocks here
// (end of file)
It doesn't actually block on the last line of the file but it is more or less what would happen - your code will keep going and as soon as there is no more code to run and the next tick of the event loop can be started, your process will block on the function and nothing will be able to happen before the fs.readFileSync() returns.
process.nextTick() is never a solution for a blocking code. It can be helpful for non-blocking code in some situations where you want to keep the execution order consistent for example, but the solution to blocking code is not using nextTick but rather avoiding the blocking functions (like the "Sync" functions from fs) altogether.
In fact, the only place where you should ever use a blocking functions (like readFileSync() or require()) is on the first tick of the event loop - and using nextTick() guarantees that you're not running them on the first tick.
I add this as a question answer wiki since I ran into this a couple of times.
This time it was for a line reader and processor that could be fed with a large array or line by line from a stream.
Since getLine would call lineHandler and lineHandler would call getLine there would be a problem with the stack getting too big when processing hundreds of thousands of lines.
To solve this I use setTimeout to wipe out the call stack instead of directly calling the functions. Here is some sample code:
function startPromise(){
return new Promise(function(resolve,reject){
resolve();
})
}
var lineCount=0;
var totalLines=2000000;
var totalCount=0;
var linesProcessed=0;
function processFile(){console.log('we are done');}
function lineHandler(){
if(linesProcessed%100===0){
console.log('processed:',linesProcessed);
}
lineCount++;
linesProcessed++;
setTimeout(getLine,0);
return;
}
function getLine(){
startPromise()
.then(function(){
if(lineCount>=totalLines){
setTimeout(processFile,0);
return;
}
setTimeout(lineHandler,0);
})
.then(null,function(error){//failed to process this line
console.log('fail:',error);
})
}
getLine();
Because of the 2 timeouts in every line this code runs very slow (100 lines in a second). This is way too slow because I'm hoping to process 10 million in an Hour.
I replaced all setTimeout calls with speedup and the speedup function is like the following:
var speedup = (function() {
var stack=0;
return function(fn,t){
stack++;
//if call stack gets over 5000 then do a setTimeout
// to reset the stack
if(stack%5000===0){
setTimeout(fn,t);
stack=0;
}else{
fn();
}
};
}())
[update]
With native Promise there is no need to use setTimeout to control the execution flow and keep the stack short. Some libraries may not break out or use setTimeout (=slow down) but tried the code without setTimeout and did not run out of stack space even with 50000 recursions.
I am trying to run a function in a async way, but I always recive the result in the order that i call them instead of receive the quickest function.
This is an example. The expected result is to print first 100 even it is called last.
function f1(callback){
i = 0;
for (i;i< 100;i++){
}
if (i == 100){
return callback(i);
}
}
function f2(callback){
i = 0;
for (i;i< 99999999999;i++){
}
if (i == 99999999999){
return callback(i);
}
}
f2(function(i){
console.log(i);
})
f1(function(i){
console.log(i);
});
In node (and in javascript in general), functions are not asynchronous by default.
In your code above, at no point are you yielding control to the javascript's event loop. As you call each function, it runs through its for loop until finished and then calls back which is why you are seeing the output the way you are.
The asynchronous nature of node kicks in when you start doing IO or purposely yield control to the VM and give it a chance to process other events off the event loop.
To see the code above run in an asynchronous fashion for the purposes of your test:
function f1(callback){
setTimeout(callback('f1'), 100);
}
function f2(callback){
setTimeout(callback('f2'), 500);
}
f2(function(i){
console.log(i);
})
f1(function(i){
console.log(i);
});
The setTimeout call basically just says: call back to the provided function as close to n milliseconds as you can (but not before). Basically acting as a sleep statement here and yielding control back to the currently executing code.