Confusion related to node.js code - node.js

I found this code where it says that I can run some db queries asynchronously
var queries = [];
for (var i=0;i <1; i++) {
queries.push((function(j){
return function(callback) {
collection.find(
{value:"1"},
function(err_positive, result_positive) {
result_positive.count(function(err, count){
console.log("Total matches: " + count);
positives[j] = count;
callback();
});
}
);
}
})(i));
}
async.parallel(queries, function(){
// do the work with the results
}
I didn't get the part what is callback function how is it defined ? second in the queries.push, it is passing the function(j) what is j in this and what is this (i) for
queries.push((function(j){})(i));
I am totally confused how this code is working?

The loop is preparing an array of nearly-identical functions as tasks for async.parallel().
After the loop, given it only iterates once currently, queries would be similar to:
var queries = [
function (callback) {
collection.find(
// etc.
);
}
];
And, for each additional iteration, a new function (callback) { ... } would be added.
what is callback function how is it defined ?
callback is just a named argument for each of the functions. Its value will be defined by async.parallel() as another function which, when called, allows async to know when each of the tasks has completed.
second in the queries.push, it is passing the function(j) what is j in this and what is this (i) for
queries.push((function(j){})(i));
The (function(j){})(i) is an Immediately-Invoked Function Expression (IIFE) with j as a named argument, it's called immediately with i a passed argument, and returns a new function (callback) {} to be pushed onto queries:
queries.push(
(function (j) {
return function (callback) {
// etc.
};
})(i)
);
The purpose of the IIFE is to create a closure -- a lexical scope with local variables that is "stuck" to any functions also defined within the scope. This allows each function (callback) {} to have its own j with a single value of i (since i will continue to change value as the for loop continues).

Callback is one of the coolest features. Callback is just another normal function. You can actually pass a function itself as a parameter in another function.
Say function foo() does something. You may want to execute something else right after foo() is done executing. So, in order to achieve this, you define a function bar() and pass this function as a parameter to function foo()!
function foo(callback){
// do something
callback();
}
function bar(){
// do something else
}
foo(bar());
//here we can see that a function itself is being passed as a param to function foo.
For more understanding here's the right link.
And
queries.push((function(j){})(i));
in javascript, this is another way of calling a function.
function(){
// do something.
}();
You don't actually need to define a function name and it can be called directly without a name. You can even pass params to it.
function(a){
alert(a)'
}(10);

Related

Best Way to Chain Callbacks?

I have a question about the best way to chain callbacks together while passing data between them. I have an example below which works, but has the flaw that the functions have to be aware of the chain and know their position / whether to pass on a call back.
function first(data, cb1, cb2, cb3){
console.log("first()", data);
cb1(data,cb2, cb3);
}
function second(data, cb1, cb2) {
console.log("second()",data);
cb1(data, cb2);
}
function third(data, cb) {
console.log("third()",data);
cb(data);
}
function last(data) {
console.log("last() ", data);
}
first("THEDATA", second, third, last); // This work OK
first("THEDATA2", second, last); // This works OK
first("THEDATA3", second); // This doesn't work as second tries to pass on a callback that doesn't exit.
I can think of a number of ways around this, but they all involve in making the called functions aware of what is going on. But my problem is that the real functions I’m dealing with already exist, and I don’t want to modify them if I can avoid it. So I wondered if I was missing a trick in terms of how these could be called?
Thanks for the answers, and I agree that promises are probably the most appropriate solution, as they meet my requirements and provide a number of other advantages.
However I have also figured out how to do what I want without involving any extra modules.
To recap the specific ask was to:
chain together a number of functions via callbacks (this is so the first function can use a non-blocking I/O call the other functions are dependent upon),
while passing arguments (data) between them, and
without the existing functions having to be modified to be aware of their position in the callback chain.
The 'trick' I was missing was to introduce some extra anonymous callback functions to act as links between the existing functions.
// The series of functions now follow the same pattern, which is how they were before
// p1 data object and p2 is a callback, except for last().
function first(data, cb){
console.log("first()", data);
cb(data);
}
function second(data, cb) {
console.log("second()",data);
cb(data);
}
function third(data, cb) {
console.log("third()",data);
cb(data);
}
function last(data) {
console.log("last() ", data);
}
// And the named functions can be called pretty much in any order without
// the called functions knowing or caring.
// first() + last()
first("THEDATA", function (data) { // The anonymous function is called-back, receives the data,
last(data); // and calls the next function.
});
// first() + second() + last()
first("THEDATA2", function (data) {
second(data, function (data){
last(data);
}); // end second();
}); // end first();
// first() + third()! + second()! + last()
first("THEDATA3", function (data) {
third(data, function (data){
second(data, function (data){
last(data);
}); // end third();
}); // end second();
}); // end first();
Promises are the way to go as they provide following benefits :
Sequential callback chaining
Parallel callback chaining (kind of)
Exception handling
Easily passing around the objects
In general, a Promise performs some operation and then changes its own state to either rejected - that it failed, or resolved that its completed and we have the result.
Now, each promise has method then(function(result), function(error)). This then() method is executed when the Promise is either resolved or rejected. If resolved, the first argument of the then() is executed, with the result and if Promise gets rejected 2nd argument is executed.
A typical callback chaining looks like :
example 1 :
someMethodThatReturnsPromise()
.then(function (result) {
// executed after the someMethodThatReturnsPromise() resolves
return somePromise;
}).then(function (result) {
// executed after somePromise resolved
}).then(null, function (e) {
// executed if any of the promise is rejected in the above chain
});
How do you create a Promise at the first place ? You do it like this :
new Promise (function (resolve, reject) {
// do some operation and when it completes
resolve(result /*result you want to pass to "then" method*/)
// if something wrong happens call "reject(error /*error object*/)"
});
For more details : head onto here
Best practice would be to check whether the callback you are calling is provided in the args.
function first(data, cb1, cb2, cb3){
console.log("first()", data); // return something
if(cb1){
cb1(data,cb2, cb3);
}
}
function second(data, cb1, cb2) {
console.log("second()",data); // return something
if(cb1){
cb1(data, cb2);
}
}
function third(data, cb) {
console.log("third()",data); // return something
if(cb){
cb(data);
}
}
function last(data) {
console.log("last() ", data); // return something
}
first("THEDATA", second, third, last); // This work OK
first("THEDATA2", second, last); // This works OK
first("THEDATA3", second);
This will work fine.
Alternatively there are many more options like Promises and Async library e.tc
Maybe you want to try it this way, yes right using "Promise" has more features, but if you want something simpler, we can make it like this, yes this is without error checking, but of course we can use try / catch
function Bersambung(cb){
this.mainfun = cb
this.Lanjut = function(cb){
var thisss = this;
if(cb){
return new Bersambung(function(lanjut){
thisss.mainfun(function(){
cb(lanjut);
})
})
} else {
this.mainfun(function(){});
}
}
}
//You can use it like this :
var data1 = "", data2 = "" , data3 = ""
new Bersambung(function(next){
console.log("sebelum async")
setTimeout(function(){
data1 = "MENYIMPAN DATA : save data"
next();
},1000);
}).Lanjut(function(next){
if(data1 == ""){
return; // break the chain
}
console.log("sebelum async 2")
setTimeout(function(){
console.log("after async")
console.log(data1);
next();
},1000)
}).Lanjut();
Okay this might not be ideal but it works.
function foo(cb = [], args = {}) // input a chain of callback functions and argumets
{
args.sometext += "f";
console.log(args.sometext)
if (cb.length == 0)
return; //stop if no more callback functions in chain
else
cb[0](cb.slice(1), args); //run next callback and remove from chain
}
function boo(cb = [], args = {})
{
newArgs = {}; // if you want sperate arguments
newArgs.sometext = args.sometext.substring(1);
newArgs.sometext += "b";
console.log(newArgs.sometext)
if (cb.length == 0)
return;
else
cb[0](cb.slice(1), newArgs);
}
//execute a chain of callback functions
foo ([foo, foo, boo, boo], {sometext:""})

Avoid callback multi-invocation when forEach is used

I have a function that processes an array of data (first parameter) and, once the procesing is finished, it invokes only one time a callback function (second parameter). I'm using forEach to process data item by item, consisting the processing of each item in some checkings and storing the param in database. The function storeInDB() does the storing work and uses a callback (second parameter) when the item has been stored.
A first approach to the code is the following:
function doWork(data, callback) {
data.forEach(function (item) {
// Do some check on item
...
storeInDB(item, function(err) {
// check error etc.
...
callback();
});
});
}
However, it's wrong, as the the callback function will be invoked several times (as many as element in the data array).
I'd like to know how to refactor my code in order to achieve the desired behaviour, i.e. only one invocation to callback once the storing work is finished. I guess that async could help in this task, but I haven't find the right pattern yet to combine async + forEach.
Any help is appreciated!
You can use a library such as async to do this, although I would recommend using promises if possible. For your immediate problem you can use a counter to determine how many storage calls have completed and call the callback when the total number are completed.
let counter = 0;
data.forEach(function (item) {
// Do some check on item
...
storeInDB(item, function(err) {
// check error etc.
counter++
if (counter == data.length) {
callback();
}
});
});
you can also utilize the three parameters passed to the function to execute on each array method
function doWork(data, callback) {
data.forEach(function (value,idx,arr) {
// Do some check on item
...
storeInDB(arr[idx], function(err) {
// check error etc.
...
if ( (idx + 1) === arr.length ) {
callback();
}
});
});
}
If storeInDB function returns a promise, you can push all async functions to an array and use Promise.all. After all tasks run successfully, It will invokes callback function.
Hope this helps you.
function doWork(data, callback) {
let arr = [];
data.map(function(itm) {
// Do some check on item
...
arr.push(storeInDB(item));
});
Promise.all(arr)
.then(function(res) {
callback();
});
}

For loop value skipped in loop

I have this for loop in my node.js application:
for (var runNumber=1; runNumber<=numberOfRuns; runNumber++) {
runNumberQueue.place(function(){
versionRunningService.runMultiVersionJobs(
queue, jobId, sVersion, jobType, runNumber, currentConfig, allJobsConfig, appDir, function(){
runNumberQueue.next();
currentIterationCount++;
if (currentIterationCount === numberOfIterations) {
callback();
}
});
});
}
Is there somethingwrong with the structure of it? It seems to increment runNumber then execute the runNumberQueue.place function with runNumber 1, skip execution of runNumberQueue.place for runNumber 2 and execute again correctly for runNumber 3.
I wondered if it could be some asynch issue either but think that runNumberQueue.place would execute with runNumber 2 at some stage?
The way JavaScript works inside loops is that the variable defined inside the loop score (e.g., runNumber here) are bound to the same scope. So when you run an anonymous function inside the loop, it uses those variables bound to the loop scope, and not the function scope.
This is a common situation in JS loops. To avoid this, you can either define a secondary variable to capture the value of loop index variable or define an external function instead of the anonymous function.
A secondary variable:
for (var runNumber=1; runNumber<=numberOfRuns; runNumber++) {
var runNumberCopy = runNumber;
runNumberQueue.place(function(){
versionRunningService.runMultiVersionJobs(
queue, jobId, sVersion, jobType, runNumberCopy, currentConfig, allJobsConfig, appDir, function(){
runNumberQueue.next();
currentIterationCount++;
if (currentIterationCount === numberOfIterations) {
callback();
}
});
});
}
An external function:
for (var runNumber=1; runNumber<=numberOfRuns; runNumber++) {
placeRunNumberQueue(runNumber);
}
// this function can be defined elsewhere and passed all the required arguments
function placeRunNumberQueue(runNumber) {
runNumberQueue.place(function(){
versionRunningService.runMultiVersionJobs(
queue, jobId, sVersion, jobType, runNumber, currentConfig, allJobsConfig, appDir, function(){
runNumberQueue.next();
currentIterationCount++;
if (currentIterationCount === numberOfIterations) {
callback();
}
});
});
}
There is another post going over this issue in more details.

Asynchronous control flow - returning true after a for loop is completely done

I have the following method in Node.js:
var foo = function(items){
for (var i=0; i<items.length; i++){
var item = items[i];
bar(item);
}
return true;
}
I want foo to return true after all of the bar's have finished updating. How do I do that?
EDIT: I am looking for a way to do this without any external libraries if possible.
The definition of the for loop being "completely done" depends on whether or not bar is synchronous or asynchronous. If bar is synchronous--perhaps it is doing some long-running computation--then your code already returns when the loop is "completely done".
But based on the part of your question that says
after all of the bar's have finished updating
it seems to be a reasonable assumption that bar is asynchronous--perhaps it is making a network call. But currently it is missing any mechanism to report when it's done. Therefore, the first job is to redefine bar so it has an asynchronous interface. There are two basic kinds of asynchronous interfaces. The older, classical node approach is callbacks. A new approach is promises.
In either case, therefore, you need to start off by redefining bar in one way or another. You cannot use an interface asynchronously unless it is designed to be used that way.
After we have given bar an asynchronous interface, we can then use that from within foo to correctly report when the bars are "completely done".
Callbacks
In the callback case, we change the calling sequence for bar to bar(item, callback), giving it a way to report back when it is done. We will also need a callback on the foo routine, to be invoked when all the bar calls finish. Following your preference to not use libraries, the logic for this could look something like this:
function foo(items, callback) {
var count = items.length;
var results = [];
items.forEach(function(item, idx) {
bar(item, function(err, data) {
if (err) callback(err);
results[idx] = data;
if (!count--) callback(null, results);
});
});
}
This loops over the items, calling bar for each one. When each bar finishes, we place its result in a results array, and check if this is the final bar, in which case we call the top-level callback.
This would be used as
foo(items, function(err, data) {
if (err) console.log("Error is", err);
else console.log("All succeeded, with resulting array of ", data);
});
The above is essentially equivalent to using async.each, as suggested in another answer, as follows:
function foo(items, callback) {
async.each(items, bar, callback);
}
Waiting for each bar to finish
If you want to wait for each bar to finish before proceeding to the next one, with async it's pretty easy with eachSeries:
function foo(items, callback) {
async.eachSeries(items, bar, callback);
}
Non-library code would be a bit more complicated:
function foo(items, callback) {
var results = [];
function next(i) {
if (i >= items.length) return callback(results));
bar(items[i], function barCallback(err, data) {
if (err) return callback(err);
results[i] = data;
next(++i);
});
}
next(0);
}
Here, the callback reporting that each bar is completed (barCallback) calls the next routine with incremented i to kick off the call to the next bar.
Promises
However, you are likely to be better off using promises. In this case, we will design bar to return a promise, instead of invoking a callback. Assuming bar calls some database routine MYDB, the new version of bar is likely to look like this:
function bar(item) {
return MYDB.findRecordPromise(item.param);
}
If MYDB only provides old-fashioned callback-based interfaces, then the basic approach is
function bar(item) {
return new Promise(function(resolve, reject) {
MYDB.findRecord(item.param, function(err, data) {
if (err) reject(err);
else resolve(data);
});
});
}
Now that we have a promise-based version of bar, we will define foo to return a promise as well, instead of taking an extra callback parameter. Then it's very simple:
function foo(items) {
return Promise.all(items.map(bar));
}
Promise.all takes an array of promises (which we create from items by mapping over bar), and fulfills when all of them fulfill, with a value of an array of fulfilled values, and rejects when any of them reject.
This is used as
foo(items) . then(
function(data) { console.log("All succeeded, with resulting array of ", data); },
function(err) { console.log("Error is", err); }
);
Waiting for each bar to finish
If, with the promises approach, you want to wait for each bar to finish before the next one is kicked off, then the basic approach is:
function foo(items) {
return items.reduce(function(promise, item) {
return promise.then(function() { return bar(item); });
}, Promise.resolve());
}
This starts off with an "empty" (pre-resolved) promise, then each time through the loop, adds a then to the chain to execute the next bar when the preceding one is finished.
If you want to get fancy and use advanced async functions, with the new await keyword:
async function foo(items) {
for (var i = 0; i < items.length; i++) {
await bar(items[i]);
}
}
Aside
As an aside, the presumption in another answer that you might be looking for an asynchronous interface to a routine (foo) which loops over some calls to bar which are either synchronous, or may be asynchronous but can only be kicked off with no way to know when they completed, seems odd. Consider this logic:
var foo = function(items, fooCallback){
for (var i=0; i<items.length; i++){
var item = items[i];
bar(item);
}
fooCallback(true);
}
This does nothing different from the original code, except that instead of returning true, it calls the callback with true. It does not ensure that the bars are actually completed, since, to repeat myself, without giving bar a new asynchronous interface, we have no way to know that they are completed.
The code shown using async.each suffers from a similar flaw:
var async = require('async');
var foo = function(items){
async.each(items, function(item, callback){
var _item = item; // USE OF _item SEEMS UNNECESSARY
bar(_item);
callback(); // BAR IS NOT COMPLETED HERE
},
function(err){
return true; // THIS RETURN VALUE GOES NOWHERE
});
}
This will merely kick off all the bars, but then immediately call the callback before they are finished. The entire async.each part will finish more or less instantaneously, immediately after which the final function(err) will be invoked; the value of true it returns will simply go into outer space. If you wanted to write it this way, it should be
var foo = function(items, fooCallback){
async.each(items, function(item, callback){
bar(item);
callback();
},
function(err){
fooCallback(true); // CHANGE THIS
}
});
}
but even in this case fooCallback will be called pretty much instantaneously, regardless of whether the bars are finished or whether they returned errors.
There are many options to solve what you want, you just need to define the nature of your funcions foo and bar. Here are some options:
Your original code doesn't need it:
Your foo and bar functions doesn't have any callback in their parameters so they are not asynchronous.
Asuming foo function is asynchronous and bar function is synchronous:
You should rewrite your original code using callbacks, something like this:
var foo = function(items, fooCallback){
for (var i=0; i<items.length; i++){
var item = items[i];
bar(item);
}
fooCallback(true);
}
Using a Async:
You should install it at the beggining:
npm install async
You use async, like this.
var async = require('async');
var foo = function(items){
async.each(items, function(item, callback){
var _item = item;
bar(_item);
callback();
},
function(err){
return true;
}
});
}

Series flow with Async in NodeJS gets asynchrounously called

I need to call 3 functions in series x1() then x2() then x3(). But since x1 and x2 are time consuming operations x3 executes before them giving unexpected results. I have used the async library in NodeJS with the series method as below.
How can I solve this without using setTimeout for x3()
async.series([function(callback) { x1();callback(null, null);},
function(callback) { x2();callback(null, null);},
function(callback) {
x3();
callback(null, null); }],
function(err, results) { }
);
use recursive try catch..
a=true;
while(a){
try{
your work.....
a=false;
}catch(exp ){
a=true
}
}
You can use javascript's async nature to solve it. Here how it's done with callbacks. Call x3() instead of onMethodsDone().
Here, suppose x1() accepts an array. Forget it if you don't need any parameters. x3() gets the modified array from x2() which is also another time consuming function. setTimeout() is used to show the async behaviour since x1() and x2() are time consuming.
x1([1,2],function(results){
x2(results,onMethodsDone);
});
function onMethodsDone(results){
alert(results);
}
function x1(records,cb){
setTimeout(function (){
records.push(3,4,5);
cb(records); //parse updated records to callback
},2000);
}
function x2(records,cb){
setTimeout(function (){
records.push(6,7,8);
cb(records);
},2000);
}
javascripts Listeners or Promise/defers also can be used to handle it.
As the callback function had to be overloaded with different number of arguments the best option was to structure the method as follows
function callback(cbargs){
var arg1 = cbargs.cbarg1;
var arg2 = cbargs.cbarg2;
if(arg3!=null){
.....
}
}
function x1(arg1 , arg2, callback, cbargs){
.....
.....
callback(cbargs);
}
function x3(arg1 , arg2, callback, cbargs){
.....
.....
callback(cbargs);
}
x1(arg1, arg2,callback,{cbarg1, cbarg2});
x1(arg1, arg2,callback,{cbarg1, cbarg2, cbarg3});

Resources