I have a for-loop statement and an async MongoDB inside loop body. What I want to do is to make a find query from my MongoDB database, and push the result into an Array.
Here is the code:
function() arrResult() {
var arr = [];
for(...) {
collection.find({ foo: i }, function (err, cursor) {
arr.push(cursor);
}
}
return arr;
}
But it's obvious that the return value of the function would be an empty Array.
I want to tackle this problem using Q module. Is there any solutions?
I want to tackle this problem using Q module. Is there any solutions?
Yes, promises are a very easy abstraction to deal with this. You can execute the queries in parallel, and collect their results with all.
In particular, with Q it would look like this:
function arrResult(…) {
var promises = [];
for (…)
promises.push( Q.ninvoke(collection, "find", {foo: i}) );
return Q.all(promises);
}
arrResult(…).then(function(arr) {
…
}, function(err) {
// first error, if any occured
});
You need a sync mechanism that acts like a process gate.
Each returning query has to arrive at the gate, e.g. decrements some counter and deposit its result.
When all arrived at the gate, a final callback does return the collected results.
Related
I have a function that processes an array of data (first parameter) and, once the procesing is finished, it invokes only one time a callback function (second parameter). I'm using forEach to process data item by item, consisting the processing of each item in some checkings and storing the param in database. The function storeInDB() does the storing work and uses a callback (second parameter) when the item has been stored.
A first approach to the code is the following:
function doWork(data, callback) {
data.forEach(function (item) {
// Do some check on item
...
storeInDB(item, function(err) {
// check error etc.
...
callback();
});
});
}
However, it's wrong, as the the callback function will be invoked several times (as many as element in the data array).
I'd like to know how to refactor my code in order to achieve the desired behaviour, i.e. only one invocation to callback once the storing work is finished. I guess that async could help in this task, but I haven't find the right pattern yet to combine async + forEach.
Any help is appreciated!
You can use a library such as async to do this, although I would recommend using promises if possible. For your immediate problem you can use a counter to determine how many storage calls have completed and call the callback when the total number are completed.
let counter = 0;
data.forEach(function (item) {
// Do some check on item
...
storeInDB(item, function(err) {
// check error etc.
counter++
if (counter == data.length) {
callback();
}
});
});
you can also utilize the three parameters passed to the function to execute on each array method
function doWork(data, callback) {
data.forEach(function (value,idx,arr) {
// Do some check on item
...
storeInDB(arr[idx], function(err) {
// check error etc.
...
if ( (idx + 1) === arr.length ) {
callback();
}
});
});
}
If storeInDB function returns a promise, you can push all async functions to an array and use Promise.all. After all tasks run successfully, It will invokes callback function.
Hope this helps you.
function doWork(data, callback) {
let arr = [];
data.map(function(itm) {
// Do some check on item
...
arr.push(storeInDB(item));
});
Promise.all(arr)
.then(function(res) {
callback();
});
}
My goal is to insert the values gotten from a redis hash. I am using the redis package for node js.
My code is the following:
getFromHash(ids) {
const resultArray = [];
ids.forEach((id) => {
common.redisMaster.hget('mykey', id, (err, res) => {
resultArray.push(res);
});
});
console.log(resultArray);
},
The array logged at the end of the function is empty and res is not empty. What could i do to fill this array please ?
You need to use some control flow, either the async library or Promises (as described in reds docs)
Put your console.log inside the callback when the results return from the redis call. Then you will see more print out. Use one of the control flow patterns for your .forEach as well, as that is currently synchronous.
If you modify your code to something like this, it will work nicely:
var getFromHash = function getFromHash(ids) {
const resultArray = [];
ids.forEach((id) => {
common.redisMaster.hget('mykey', id, (err, res) => {
resultArray.push(res);
if (resultArray.length === ids.length) {
// All done.
console.log('getFromHash complete: ', resultArray);
}
});
});
};
In your original code you're printing the result array before any of the hget calls have returned.
Another approach will be to create an array of promises and then do a Promise.all on it.
You'll see this kind of behavior a lot with Node, remember it uses asynchronous calls for almost all i/o. When you're coming from a language where most function calls are synchronous you get tripped up by this kind of problem a lot!
Like the OP from this question, I want to do a for loop, and do something when all the actions have finished.
I checked the answer, and the async library, but all the solutions involve iterating over an array. I don't want to do something "forEach" element of an array, I don't have an array.
What if I just want to do an operation n times ? For example, say I want to insert n random entries in my database, and do something afterwards ? For now I'm stuck with something like :
function insertMultipleRandomEntries(n_entries,callback){
var sync_i=0;
for(var i=0;i<n_entries;i++){
insertRandomEntry(function(){
if(sync_i==(max-1)){
thingDoneAtTheEnd();
callback(); //watched by another function, do my stuff there
}
else{
sync_i++;
console.log(sync_i+" entries done successfully");
thingDoneEachTime();
}
});
}
}
Which is absolutely horrendous. I can't find anything like a simple for in async, how would you have done this ?
You can use Promises, supported without a library in node.js since version 4.0.
If the callback function of insertRandomEntry has a parameter, you can pass it to resolve. In the function given to then, you receive an array of parameters given to resolve.
function insertMultipleRandomEntries(n_entries,callback){
var promises = [];
for(var i=0;i<n_entries;i++) {
promises.push(new Promise(function (resolve, reject) {
insertRandomEntry(function (val) {
thingDoneEachTime(val);
resolve(val);
});
}));
}
Promise.all(promises).then(function (vals) {
// vals is an array of values given to individual resolve calls
thingDoneAtTheEnd();
callback();
});
}
I'm kinda new to this q stuff and I find it pretty awesome, but there's something I still can't figure out.
I managed to run some combined promises with q.all by passing q.all an array of promises. Something like this..
var promises = [promiseOne(), promiseTwo()];
q.all(promises).then(function (results) {
res.send(results);
} );
The thing with this is that I would actually want those promises to be named, so I don't have to rely in the order of the promises.
I read somewhere that you can actually pass an object to q.all, to have the results named. So that would be something like this:
var promises = { promiseOne: promiseOne(), promiseTwo: promiseTwo() }
q.all(promises).then(function(results){
res.send(results);
});
But I guess this just doesn't work the same way as sending an array as I'm not getting the results of my promises in there. The result I get is similar to this one:
{
promiseOne: {
source: {}
},
promiseTwo: {
source: {}
}
}
So how would you go about getting named results from q.all?
One thing to note is that the amount of promises I will have in the promises array is not fixed as I get that from a GET param sent by the user to my function.
Also, inside each of my promises I have another array (or object) of promises to be resolved and whose results I would like to be named as well.
Here is another way to write the code Roamer wrote with the functionality you asked for (return an object):
Q.props = obj => {
const ps = Q.all(Object.keys(obj).map(x => Q.all([x, obj[x]])));
return ps.then(x => x.reduce((p, c) => {p[c[0]] = c[1]; return p } , {}));
};
Which would let you do:
Q.props(promises).then(o => {
o.promiseOne
});
Although you should consider using bluebird if you want all these helper functions.
Q.js appears not to offer Q.all(object), therefore you will need to map your object to an array before passing to Q.all()
Something like this will be reusable and convenient :
Q.allObj = function (obj) {
var array = Object.keys(obj).map(function(key, i) {
try {
//expect a promise
return obj[key].then(function(value) {
return {key: key, value: value};
});
}
catch(e) {
// whoops it was a value
return {key: key, value: obj[key]};
}
});
return Q.all(array);
};
Use as follows :
Q.allObj(myObj).then(function(results) {
results.forEach(function(obj) {
var name = obj.key;
var value = obj.value;
...
});
});
I have a problem, but I have no idea how would one go around this.
I'm using loopback, but I think I would've face the same problem in mongodb sooner or later. Let me explain what am I doing:
I fetch entries from another REST services, then I prepare entries for my API response (entries are not ready yet, because they don't have id from my database)
Before I send response I want to check if entry exist in database, if it doesn't:
Create it, if it does (determined by source_id):
Use it & update it to newer version
Send response with entries (entries now have database ids assigned to them)
This seems okay, and easy to implement but it's not as far as my knowledge goes. I will try to explain further in code:
//This will not work since there are many async call, and fixedResults will be empty at the end
var fixedResults = [];
//results is array of entries
results.forEach(function(item) {
Entry.findOne({where: {source_id: item.source_id}}, functioN(err, res) {
//Did we find it in database?
if(res === null) {
//Create object, another async call here
fixedResults.push(newObj);
} else {
//Update object, another async call here
fixedResults.push(updatedObj);
}
});
});
callback(null, fixedResults);
Note: I left some of the code out, but I think its pretty self explanatory if you read through it.
So I want to iterate through all objects, create or update them in database, then when all are updated/created, use them. How would I do this?
You can use promises. They are callbacks that will be invoked after some other condition has completed. Here's an example of chaining together promises https://coderwall.com/p/ijy61g.
The q library is a good one - https://github.com/kriskowal/q
This question how to use q.js promises to work with multiple asynchronous operations gives a nice code example of how you might build these up.
This pattern is generically called an 'async map'
var fixedResults = [];
var outstanding = 0;
//results is array of entries
results.forEach(function(item, i) {
Entry.findOne({where: {source_id: item.source_id}}, functioN(err, res) {
outstanding++;
//Did we find it in database?
if(res === null) {
//Create object, another async call here
DoCreateObject(function (err, result) {
if (err) callback(err);
fixedResults[i] = result;
if (--outstanding === 0) callback (null, fixedResults);
});
} else {
//Update object, another async call here
DoOtherCall(function (err, result) {
if(err) callback(err);
fixedResults[i] = result;
if (--outstanding === 0) callback (null, fixedResults);
});
}
});
});
callback(null, fixedResults);
You could use async.map for this. For each element in the array, run the array iterator function doing what you want to do to each element, then run the callback with the result (instead of fixedResults.push), triggering the map callback when all are done. Each iteration ad database call would then be run in parallel.
Mongo has a function called upsert.
http://docs.mongodb.org/manual/reference/method/db.collection.update/
It does exactly what you ask for without needing the checks. You can fire all three requests asnc and just validate the result comes back as true. No need for additional processing.