I have situation where i have an array of employees data and i need to process something parallel for every employee.To implement it and achieve the task i broke the things to chunks to four methods and every method has a callback calling each other and returning callback.I am using
async.eachSeries
to start the process for each element of the employee array.
In the last method i have to set the setInterval to perform same task if required response is not achieved,this interval is not cleared till the process of repeating task continues to 5 times(if desired value is not received 5 times,but cleared after 5th time).
Now,the problem happening is that data which i am processing inside setInterval is getting overriden by values of last employees.
So i am not able to keep track of process happening for all the employee Array elements and also the details of processing for last employee are getting mixed up.
In between the four methods which i am using for performing the task are carrying out the process of saving data to redis , MongoDB , Outside Api's giving response in callback.
Can anyone suggest better way of doing this and also i feel that the problem is happening because i am not returning any callback from SetInterval method().But since that method itself is an asynchronous method so i am unware about how to handle the situation.
EmployeeArray
async.eachSeries() used to process EmployeeArray
for each i have Four callBack Medhods .
async.eachSeries() {
Callback1(){
Callback2(){
Callback3(){
Callback4(){
SetInterval(No CallBack inside this from my side)
}
}
}
}
}
As per I know the async each function does parallel processing. Also u can use async waterfall to make your code more clean. Try something like this.
async.each(openFiles, function(file, callback1) {
async.waterfall([
function(callback) {
callback(null, 'one', 'two');
},
function(arg1, arg2, callback) {
// arg1 now equals 'one' and arg2 now equals 'two'
callback(null, 'three');
},
function(arg1, callback) {
// arg1 now equals 'three'
callback(null, 'done');
}
], function (err, result) {
callback1(err);
});
}, function(err){
//if you come here without error your data is processed
});
Related
So I'm making a web application and I'm trying to send variables to an EJS file but when they are sent out of the mongo functions they come out as undefined because it's a different scope for some reason. It's hard to explain so let me try to show you.
router.get("/", function(req, res){
var bookCount;
var userCount;
Books.count({}, function(err, stats){
if(err){
console.log("Books count failed to load.");
}else{
bookCount = stats;
}
});
User.count({}, function(err, count){
if(err){
console.log("User count failed to load.")
}else{
userCount = count;
console.log(userCount);
}
});
console.log(userCount);
//Get All books from DB
Books.find({}, function(err, allbooks){
if(err){
console.log("Problem getting all books");
}else{
res.render("index", {allbooks: allbooks, bookCount: bookCount, userCount: userCount});
}
});
});
So in the User.Count and Books.count I'm finding the number of documents in a collection which works and the number is stored inside of the variables declared at the very top.
After assigning the numbers like userCount i did console.log(userCount) which outputs the correct number which is 3, If was to do console.log(userCount) out of the User.count function it would return undefined, which is a reference to the declaration at the very top.
What is really weird is that Book.Find() has the correct userCount even though its a totally different function. The whole goal im trying to accomplish is doing res.render("index", {userCount: userCount}); outside of the Books.find(). I can do it but of course for some reason it passes undefined instead of 3. I hope this made a shred of sense.
I seem to have found a solution. but if anyone knows a different way I would love to know. So basically all you need to do is move the User.Count function outside of the router.get() function. Not completely sure about the logic of that but it works...
This is a classic asynchronous-operation problem: Your methods (Books.count, Books.find, User.count) are called immediately, but the callback functions you pass to them are not. userCount is undefined in your log because console.log is called before the assignment in the callback function is made. Your code is similar to:
var userCount;
setTimeout(function() {
userCount = 3;
}, 1000);
console.log(userCount); // undefined
User.count takes time to execute before calling back with the result, just like setTimeout takes the specified time to execute before calling its callback. The problem is JS doesn't pause and wait for the timeout to complete before moving on and calling console.log below it, it calls setTimeout, calls console.log immediately after, then the callback function is called one second later.
To render a complete view, you need to be sure you have all of the data before you call res.render. To do so you need to wait for all of the methods to call back before calling res.render. But wait, I just told you that JS doesn't pause and wait, so how can this be accomplished? Promise is the answer. Multiple promises, actually.
It looks like you are using Mongoose models. Mongoose has been written so that if you don't pass a callback function to your methods, they return a promise.
Books.count({}) // returns a promise
JS promises have a method then which takes a callback function that is called when the promise has been resolved with the value of the asynchronous method call.
Books.count({}) // takes some time
.then(function(bookCount) { // called when Books.count is done
// use the bookCount here
})
The problem is, you want to wait for multiple operations to complete, and multiple promises, before continuing. Luckily JS has a utility just for this purpose:
Promise.all( // wait for all of these operations to finish before calling the callback
Books.count({}),
User.count({}),
Books.find({})
)
.then(function(array) { // all done!
// the results are in an array
bookCount = array[0];
userC0unt = array[1];
allBooks = array[2];
})
I'm trying to build a bot cluster, and I'm having some trouble here.
I'd like to call the following function, called BuildABot.
http://pastebin.com/zwP7rZay
Now after this call is finished, I'd like to call requestItems, which takes in the steamOffers object as a parameter.
http://pastebin.com/jsJ4fhwG
However, the object is null, because the call hasn't finished.
Is there any way to halt the call until buildABot is done?
There are various ways to handle your requirement, I will mention 2 options while I am sure you can find more. You may find many more examples all over the web.
Option 1: Using a callback function -
Pass a callback function to the async function, When async function is finished it will call the callback func. This way the code in the callback func will be executed only when async call is over.
Related topic - how-to-write-asynchronous-functions-for-node-js
Option 2: If you have more complicated logic and you want to perform one part after another you can use the async waterfall.
Code example:
async.waterfall([
function(callback){
callback(null, 'one', 'two');
},
function(arg1, arg2, callback){
// arg1 now equals 'one' and arg2 now equals 'two'
callback(null, 'three');
},
function(arg1, callback){
// arg1 now equals 'three'
callback(null, 'done');
}
], function (err, result) {
// result now equals 'done'
});
See async module site for more information.
I have to implement a program in node.js which looks like the following code snippet. It has an array though which I have to traverse and match the values with database table entries. I need to wait till the loop ends and send the result back to the calling function:
var arr=[];
arr=[one,two,three,four,five];
for(int j=0;j<arr.length;j++) {
var str="/^"+arr[j]+"/";
// consider collection to be a variable to point to a database table
collection.find({value:str}).toArray(function getResult(err, result) {
//do something incase a mathc is found in the database...
});
}
However, as the str="/^"+arr[j]+"/"; (which is actually a regex to be passed to find function of MongoDB in order to find partial match) executes asynchronously before the find function, I am unable to traverse through the array and get required output.
Also, I am having hard time traversing through array and send the result back to calling function as I do not have any idea when will the loop finish executing.
Try using async each. This will let you iterate over an array and execute asynchronous functions. Async is a great library that has solutions and helpers for many common asynchronous patterns and problems.
https://github.com/caolan/async#each
Something like this:
var arr=[];
arr=[one,two,three,four,five];
asych.each(arr, function (item, callback) {
var str="/^"+item+"/";
// consider collection to be a variable to point to a database table
collection.find({value:str}).toArray(function getResult(err, result) {
if (err) { return callback(err); }
// do something incase a mathc is found in the database...
// whatever logic you want to do on result should go here, then execute callback
// to indicate that this iteration is complete
callback(null);
});
} function (error) {
// At this point, the each loop is done and you can continue processing here
// Be sure to check for errors!
})
Using Node.js and the node-postgres module to communicate with a database, I'm attempting to write a function that accepts an array of queries and callbacks and executes them all asynchronously using the same database connection. The function accepts a two-dimensional array and calling it looks like this:
perform_queries_async([
['SELECT COUNT(id) as count FROM ideas', function(result) {
console.log("FUNCTION 1");
}],
["INSERT INTO ideas (name) VALUES ('test')", function(result) {
console.log("FUNCTION 2");
}]
]);
And the function iterates over the array, creating a query for each sub-array, like so:
function perform_queries_async(queries) {
var client = new pg.Client(process.env.DATABASE_URL);
for(var i=0; i<queries.length; i++) {
var q = queries[i];
client.query(q[0], function(err, result) {
if(err) {
console.log(err);
} else {
q[1](result);
}
});
}
client.on('drain', function() {
console.log("drained");
client.end();
});
client.connect();
}
When I ran the above code, I expected to see output like this:
FUNCTION 1
FUNCTION 2
drained
However, the output bizarrely appears like so:
FUNCTION 2
drained
FUNCTION 2
Not only is the second function getting called for both requests, it also seems as though the drain code is getting called before the client's queue of queries is finished running...yet the second query still runs perfectly fine even though the client.end() code ostensibly killed the client once the event is called.
I've been tearing my hair out about this for hours. I tried hardcoding in my sample array (thus removing the for loop), and my code worked as expected, which leads me to believe that there is some problem with my loop that I'm not seeing.
Any ideas on why this might be happening would be greatly appreciated.
The simplest way to properly capture the value of the q variable in a closure in modern JavaScript is to use forEach:
queries.forEach(function(q) {
client.query(q[0], function(err, result) {
if(err) {
console.log(err);
} else {
q[1](result);
}
});
});
If you don't capture the value, your code reflects the last value that q had, as the callback function executed later, in the context of the containing function.
forEach, by using a callback function isolates and captures the value of q so it can be properly evaluated by the inner callback.
A victim of the famous Javascript closure/loop gotcha. See my (and other) answers here:
I am trying to open 10 websocket connections with nodejs, but somehow my loop doesnt work
Basically, at the time your callback is executed, q is set to the last element of the input array. The way around it is to dynamically generate the closure.
It will be good to execute this using async module . It will help you to reuse the code also . and will make the code more readable . I just love the auto function provided by async module
Ref: https://github.com/caolan/async
i try to understand and use async library in node. What i am really not understand, how async.parallel function worked. The name parallel it seems for like multithreading, consider the following sample:
async.parallel([
function(callback){
setTimeout(function(){
console.log('1');
callback(null, 'one');
}, 200);
},
function(callback){
setTimeout(function(){
console.log('2');
callback(null, 'two');
}, 100);
}
],
// optional callback
function(err, results){
if(err){
console.log('Error');
} else {
console.log(results);
}
// the results array will equal ['one','two'] even though
// the second function had a shorter timeout.
});
i have got the result
[ 'one', 'two' ]
Do async.parallel execute on multithread? If not, what the name parallel express?
The placement of results within the array is gauranteed by your placement of those functions within the array you pass to parallel. Notice that functions you pass to parallel have to take the same arguments. Picture these functions being placed in a wrapper and then called like this
someArrayOfFunctions[index](arguments);
Using this method, async gaurantees that independent of when functions within parallel finishes, the results are placed in the array in the order expected, through use of callbacks for populating results, based on indices. The specific implementation does not matter, the fact is no matter when you finish, the results of respective runs will end up placed based on where their function was in the array, not based on timing.
As per your second question, whether async.parallel is truly parallel. It is not. Refer to the following example:
var async = require('async');
async.parallel([
function(callback){
while(true);//blocking!
},
function(callback){
console.log('Not Blocked');//Never get's called
}
]);
The second function will never get called. The parallel that async provides helps out, for exactly the reason you are confused about your example. The problem with asynchronous code is that, sometimes we have some series of callbacks for things that are actually parallel in node(disk I/O, network I/O, etc...) that need to complete, but that will end up completing at unpredictable intervals. Say for example we have configuration data to collect from multiple sources, and no sync methods are supplied. We don't want to run these serially, because this slows things down considerably, but we still want to collect the results in order. This is the prime example use for async.parallel. But, no, async.parallel cannot make synchronous code execute asynchronously, as is shown by this blocking example.
True parallelism in node comes within the V8 backend. The only way to provide this parallelism would be to release an alternate version of node or by developing native extensions.