I've got a problem with redis and nodejs. I have to loop through a list of phone numbers, and check if this number is present in my redis database. Here is my code :
function getContactList(contacts, callback) {
var contactList = {};
for(var i = 0; i < contacts.length; i++) {
var phoneNumber = contacts[i];
if(utils.isValidNumber(phoneNumber)) {
db.client().get(phoneNumber).then(function(reply) {
console.log("before");
contactList[phoneNumber] = reply;
});
}
}
console.log("after");
callback(contactList);
};
The "after" console log appears before the "before" console log, and the callback always return an empty contactList. This is because requests to redis are asynchronous if I understood well. But the thing is I don't know how to make it works.
How can I do ?
You have two main issues.
Your phoneNumber variable will not be what you want it to be. That can be fixed by changing to a .forEach() or .map() iteration of your array because that will create a local function scope for the current variable.
You have create a way to know when all the async operations are done. There are lots of duplicate questions/answers that show how to do that. You probably want to use Promise.all().
I'd suggest this solution that leverages the promises you already have:
function getContactList(contacts) {
var contactList = {};
return Promise.all(contacts.filter(utils.isValidNumber).map(function(phoneNumber) {
return db.client().get(phoneNumber).then(function(reply) {
// build custom object
constactList[phoneNumber] = reply;
});
})).then(function() {
// make contactList be the resolve value
return contactList;
});
}
getContactList.then(function(contactList) {
// use the contactList here
}, funtion(err) {
// process errors here
});
Here's how this works:
Call contacts.filter(utils.isValidNumber) to filter the array to only valid numbers.
Call .map() to iterate through that filtered array
return db.client().get(phoneNumber) from the .map() callback to create an array of promises.
After getting the data for the phone number, add that data to your custom contactList object (this is essentially a side effect of the .map() loop.
Use Promise.all() on the returned array of promises to know when they are all done.
Make the contactList object we built up be the resolve value of the returned promise.
Then, to call it just use the returned promise with .then() to get the final result. No need to add a callback argument when you already have a promise that you can just return.
The simplest solution may be to use MGET with a list of phone numbers and put the callback in the 'then' section.
You could also put the promises in an array and use Promise.all().
At some point you might want your function to return a promise rather than with callback, just to stay consistent.
Consider refactoring your NodeJS code to use Promises.
Bluebird is an excellent choice: http://bluebirdjs.com/docs/working-with-callbacks.html
you put async code into a for loop (sync operations). So, each iteration of the for loop is not waiting for the db.client(...) function to end.
Take a look at this stackoverflow answer, it explains how to make async loops :
Here
Related
I'm struggling with callbacks in Node.js. I simply want playerNumber to be set to the number of players in my collection of Players. The console.log works, but I can't get the variable out of the function and into the playerNumber variable.
And if there's a simpler way get this value for use in the rest of my backend code, I'm all ears. I'm clearly new at Node.js, but the code always seems more involved than I'm expecting.
Thanks in advance!
var playerNumber = function countPlayers(callback){
Player.count(function(err, numOfDocs) {
console.log('I have '+numOfDocs+' documents in my collection');
callback(err, numOfDocs);
});
}
It's probably async, and it's a typical first-timer experience to want to "get back to normal" on the call chain on the way back from async call. This can't be done, but it's not so bad to live with it. Here's how...
Step 1: Promises are better than callbacks. I'll leave the long story
to others.
Step 2: Callbacks can be made into promises
In the OP case...
// The promise constructor takes a function that has two functions as params
// one to call on success, and one to call on error. Instead of a callback
// call the 'resolve' param with the data and the 'reject' param with any error
// mark the function 'async' so callers know it can be 'await'-ed
const playerNumber = async function countPlayers() {
return new Promise((resolve, reject) => {
Player.count(function(err, numOfDocs) {
err ? reject(err) : resolve(numOfDocs);
});
});
}
Step 3: Yes, the callers must deal with this, and the callers of the callers, and so on. It's not so bad.
In the OP case (in the most modern syntax)...
// this has to be async because it 'awaits' the first function
// think of await as stopping serial execution until the async function finishes
// (it's not that at all, but that's an okay starting simplification)
async function printPlayerCount() {
const count = await playerNumber();
console.log(count);
}
// as long as we're calling something async (something that must be awaited)
// we mark the function as async
async function printPlayerCountAndPrintSomethingElse() {
await printPlayerCount();
console.log('do something else');
}
Step 4: Enjoy it, and do some further study. It's actually great that we can do such a complex thing so simply. Here's good reading to start with: MDN on Promises.
I'm still somewhat new to working with Node and def new to working asynchronously and with promises.
I have an application that is hitting a REST endpoint, then calling a chain of functions. The end of this chain is calling hgetall and I need to wait until I get the result and pass it back. I'm testing with Postman and I'm getting {} back instead of the id. I can console.log the id, so I know that this is because some of the code isn't waiting for the result of hgetall before continuing.
I'm using await to wait for the result of hgetall, but that's only working for the end of the chain. do I need to do this for the entire chain of functions, or is there a way to have everything wait for the result before continuing on? Here's the last bit of the logic chain:
Note: I've removed some of the logic from the below functions and renamed a few things to make it a bit easier to see the flow and whats going on with this particular issue. So, some of it may look a bit weird.
For this example, it will call GetProfileById().
FindProfile(info) {
var profile;
var profileId = this.GenerateProfileIdkey(info); // Yes, this will always give me the correct key
profile = this.GetProfileById(profileId);
return profile;
}
This checks with the Redis exists, to verify if the key exists, then tries to get the id with that key. I am now aware that the Key() returns true instead of what Redis actually returns, but I'll fix that once I get this current issue resolved.
GetProfileById(profileId) {
if ((this.datastore.Key(profileId) === true) && (profileId != null)) {
logger.info('GetProfileById ==> Profile found. Returning the profile');
return this.datastore.GetId(profileId);
} else {
logger.info(`GetProfileById ==> No profile found with key ${profileId}`)
return false;
}
}
GetId() then calls the data_store to get the id. This is also where I started to use await and async to try and wait for the result to come through before proceeding. This part does wait for the result, but the functions prior to this don't seem to wait for this one to return anything. Also curious why it only returns the key and not the value, but when I print out the result in hgetall I get the key and value?
async GetId(key) {
var result = await this.store.RedisGetId(key);
console.log('PDS ==> Here is the GetId result');
console.log(result); // returns [ 'id' ]
return result;
}
and finally, we have the hgetall call. Again, new to promises and async, so this may not be the best way of handling this or right at all, but it is getting the result and waiting for the result before it returns anything
async RedisGetId(key) {
var returnVal;
var values;
return new Promise((resolve, reject) => {
client.hgetall(key, (err, object) => {
if (err) {
reject(err);
} else {
resolve(Object.keys(object));
console.log(object); // returns {id: 'xxxxxxxxxxxxxx'}
return object;
}
});
});
}
Am I going to need to async every single function that could potentially end up making a Redis call, or is there a way to make the app wait for the Redis call to return something, then continue on?
Short answer is "Yes". In general, if a call makes an asynchronous request and you need to await the answer, you will need to do something to wait for it.
Sometimes, you can get smart and issue multiple calls at once and await all of them in parallel using Promise.all.
However, it looks like in your case your workflow is synchronous, so you will need to await each step individually. This can get ugly, so for redis I typically use something like promisify and make it easier to use native promises with redis. There is even an example on how to do this in the redis docs:
const {promisify} = require('util');
const getAsync = promisify(client.get).bind(client);
...
const fooVal = await getAsync('foo');
Makes your code much nicer.
I have an orientdb database. I want to use nodejs with RESTfull calls to create a large number of records. I need to get the #rid of each for some later processing.
My psuedo code is:
for each record
write.to.db(record)
when the async of write.to.db() finishes
process based on #rid
carryon()
I have landed in serious callback hell from this. The version that was closest used a tail recursion in the .then function to write the next record to the db. However, I couldn't carry on with the rest of the processing.
A final constraint is that I am behind a corporate proxy and cannot use any other packages without going through the network administrator, so using the native nodejs packages is essential.
Any suggestions?
With a completion callback, the general design pattern for this type of problem makes use of a local function for doing each write:
var records = ....; // array of records to write
var index = 0;
function writeNext(r) {
write.to.db(r, function(err) {
if (err) {
// error handling
} else {
++index;
if (index < records.length) {
writeOne(records[index]);
}
}
});
}
writeNext(records[0]);
The key here is that you can't use synchronous iterators like .forEach() because they won't iterate one at a time and wait for completion. Instead, you do your own iteration.
If your write function returns a promise, you can use the .reduce() pattern that is common for iterating an array.
var records = ...; // some array of records to write
records.reduce(function(p, r) {
return p.then(function() {
return write.to.db(r);
});
}, Promsise.resolve()).then(function() {
// all done here
}, function(err) {
// error here
});
This solution chains promises together, waiting for each one to resolve before executing the next save.
It's kinda hard to tell which function would be best for your scenario w/o more detail, but I almost always use asyncjs for this kind of thing.
From what you say, one way to do it would be with async.map:
var recordsToCreate = [...];
function functionThatCallsTheApi(record, cb){
// do the api call, then call cb(null, rid)
}
async.map(recordsToCreate, functionThatCallsTheApi, function(err, results){
// here, err will be if anything failed in any function
// results will be an array of the rids
});
You can also check out other ones to enable throttling, which is probablya good idea.
I am trying to run some database queries (using sails.js) on an array and upon the queries' return, do something. I figured the best way to do so would be to use a for loop and resolve the promises async, and once they've all resolved, continue on. However, only the last promise in my array is resolving, and it is resolving multiple times because in each 'User.findOne...' then function, the index is array.length-1.
My questions:
How does variable scope in asynchronous loops work? Best resources to explain this?
What is the best way to solve my problem? Why?
Are there any other patterns I should use or not use? I am fairly new to promises and async js, so any tips would be helpful!
Main tutorials I've checked
https://github.com/kriskowal/q
https://github.com/kriskowal/q/wiki/API-Reference
https://github.com/bellbind/using-promise-q/
Thank you for your help!
My simplified code:
functionWhichReturnsPromise()
.then(function(user){
var promises = [];
Q.try(function(){
for (var index in array) {
var fbid = array[index];// Get fbid from array
promises.push(Q.defer().promise); // Add promise to promise array
// Find userid from fbid; resolve respective promise when finished
User.findOne({facebook_id: fbid}).then(function(userSeen){
promises[index].resolve(userSeen.id);
sails.log('resolved where id=' + userSeen.id); // correct
sails.log('resolved where index=' + index); // PROBLEM: always last index
});
}
}).then(function(){
// For debugging purposes
Q.delay(1000).then(function(){
sails.log(promises[0]); // Unresolved
sails.log(promises[1]); // Unresolved
sails.log(promises[2]); // Only last promise in array is resolved
});
// When the userids have been extracted from above (promises fulfilled)...
Q.all(promises).then(function(seenids){
// Do stuff here (Doesn't get here)
});
});
});
In Javascript, variable's scope is function and not curly braces.
Therefore in the following code, the scope of var index is not the for loop's curly braces, the scope is actually the function in which the for loop exists.
Q.try(function(){
for (var index in array) {
var fbid = array[index];// Get fbid from array
promises.push(Q.defer().promise); // Add promise to promise array
// Find userid from fbid; resolve respective promise when finished
User.findOne({facebook_id: fbid}).then(function(userSeen){
promises[index].resolve(userSeen.id);
sails.log('resolved where id=' + userSeen.id); // correct
sails.log('resolved where index=' + index); // PROBLEM: always last index
});
}
})
Within for loop you call the async function, in your case its mongodb call (findOne).
You should always assume that these async function can take any number of milliseconds to run (depends on the function). But in general, usually the loop would have completed before the async functions run. Your for loop fires all those async functions even before those functions start running. The issue is that all those async functions which area pending are still pointing towards that variable index. And that variable is common to all of them because index was in scope of the outer function.
This is a problem created somewhat because of closures in Javascript. And to solve this, we need to use more closures.
There are many resources on the topic of closures that you can google. But go thru the MDN's description of it.
If you capture value of index within another function inside the loop, then you will be good to go.
Here is my suggested solution to your issue. I haven't tested it though, but you get the idea.
Q.try (function () {
array.forEach( function(ele, idx, array) {
(function(index) {
var fbid = array[index]; // Get fbid from array
promises.push(Q.defer().promise); // Add promise to promise array
// Find userid from fbid; resolve respective promise when finished
User.findOne({
facebook_id : fbid
}).then(function (userSeen) {
promises[index].resolve(userSeen.id);
sails.log('resolved where id=' + userSeen.id); // correct
sails.log('resolved where index=' + index); // PROBLEM: always last index
});
})(idx);
})
})
Hope this helps.
Also Note: it is incorrect to use for...in for iterating through arrays.
I have to implement a program in node.js which looks like the following code snippet. It has an array though which I have to traverse and match the values with database table entries. I need to wait till the loop ends and send the result back to the calling function:
var arr=[];
arr=[one,two,three,four,five];
for(int j=0;j<arr.length;j++) {
var str="/^"+arr[j]+"/";
// consider collection to be a variable to point to a database table
collection.find({value:str}).toArray(function getResult(err, result) {
//do something incase a mathc is found in the database...
});
}
However, as the str="/^"+arr[j]+"/"; (which is actually a regex to be passed to find function of MongoDB in order to find partial match) executes asynchronously before the find function, I am unable to traverse through the array and get required output.
Also, I am having hard time traversing through array and send the result back to calling function as I do not have any idea when will the loop finish executing.
Try using async each. This will let you iterate over an array and execute asynchronous functions. Async is a great library that has solutions and helpers for many common asynchronous patterns and problems.
https://github.com/caolan/async#each
Something like this:
var arr=[];
arr=[one,two,three,four,five];
asych.each(arr, function (item, callback) {
var str="/^"+item+"/";
// consider collection to be a variable to point to a database table
collection.find({value:str}).toArray(function getResult(err, result) {
if (err) { return callback(err); }
// do something incase a mathc is found in the database...
// whatever logic you want to do on result should go here, then execute callback
// to indicate that this iteration is complete
callback(null);
});
} function (error) {
// At this point, the each loop is done and you can continue processing here
// Be sure to check for errors!
})