I'm trying to retrieve products basis by categories, I'd like to parallel the process, first I'm not able to figure out how to write wait condition or ideal a call back method to let parent function know that all products have been retrieved from database.
I'd be open to all solution, ideally here I have used Async but want to prefer Q (https://github.com/kriskowal/q/wiki/Examples-Gallery) which seems to be much better choice with Mongoose and MongoDB operations.
var temp = []
Async.each([1,2,3,4,5,6,7,8,9...n],
function (item, callback) {
database.getProductsByTaxonomy(item, function (err, products) {
temp = new Object();
temp.TaxonomyID = item;
temp.Products = products;
results.push(temp);
callback(err, products);
});
},
function (err) {
console.log(err);
});
<<wait for all .each call completes>>
return temp; // or callback (err, temp); // or emit?
Any solutions?
You can use my Qx library, which simplifies working with Q and arrays:
return Qx.map(arr, function (item) {
return getPromiseFromDb(item).then(function(products) {
return { taxonmyId: item, products: products };
});
});
If your DB uses callbacks rather than promises, you can use Q.ninvoke() to turn it into a promise.
Related
I've tried a few approaches to this - what I have is all my modules in one file that interact with mongodb, and in another, the express route functions that call into those async functions looking for data. The problem is that the data is available in the async function, but is not returned to the calling function, and I'm not sure how to pass it back properly (not sure if it's an issue of not waiting for the async function to return and returning the array early, or if I'm actually returning it wrong).
Here is the code for the calling function
router.route('/').get((req, res) => {
db.getItemsFromCollection("Plants").then( (itemArr) => {
console.log(itemArr);
})
});
And the db function (two attempts, one commented out)
getItemsFromCollection: async function(collectionName) {
let itemArr = [];
const collection = client.db().collection(collectionName);
/*collection.find().forEach(function( doc) {
itemArr.push(doc);
//console.log(doc);
})
return itemArr;*/
collection.find({}).toArray(function(err, item) {
if (err) throw err;
console.log(item);
itemArr.push(item);
})
return itemArr;
},
If you are using mongodb then it already supports Promise syntax:
However, all async API calls support an optional callback as the final
argument, if a callback is provided a Promise will not be returned.
Then you can make getItemsFromCollection become:
getItemsFromCollection: async function (collectionName) {
// let itemArr = [];
const collection = client.db().collection(collectionName);
const items = await collection.find({}).toArray(); // don't pass callback param
console.log(items);
// I guess you want to get back an array of the collection item
return items;
},
Complete NodeJS testing noob here. Trying to individually test functions that are called through my API (meaning, rather than make an http request to a specific endpoint, which usually invokes several functions, which in turn make requests to different third party APIs, I want to test the functions themselves separately). The way they're called is I've built a class for each data source (data source = third party API), each class contains the same functions with the same exact signatures - getData and convertData, and return a callback with the results.
I've also created a module that creates many user mocks, since each user context returns different data (meaning, a user object is fed into getData, which uses certain user properties in order to determine what data should be returned).
The way I wanted to test this was to create numerous mocks, then run the functions for each. This is what I've got so far:
// Data sources to iterate over. Each is a class instance acquired through "require".
var dataSources = [
source1,
source2,
source3,
source4
];
describe('getData', function() {
this.timeout(10000);
describe('per data source,', function() {
context('standard call', function() {
// Associative array to hold the data returned, a key for each data source.
var finalResults = {};
// Iterate over all data sources
_.forEach(dataSources, function(dataSource) {
// Generate user mocks
var users = userMocks(10);
// Iterate over all users.
_.forEach(users, function (user) {
// Call each data source with each of the users.
// Numbers of calls to make - (users * data-sources), so in this case - 10*4.
dataSource.getData(user, function (err, data) {
if (err) return done(err);
// Convert the data returned to my format
dataSource.convertData(data, function (err, processedData) {
if (err) return done(err);
// Populate finalResults with converted data from each source
if (finalResults[dataSource.sourceName]) {
finalResults[dataSource.sourceName] = finalResults[dataSource.sourceName].concat(processedData);
} else {
finalResults[dataSource.sourceName] = processedData;
}
});
});
});
});
it('should return something', function(done) {
_.forEach(finalResults.keys, function(key) {
expect(finalResults[key]).to.not.be.empty;
expect(finalResults[key].length).to.be.greaterThan(0);
});
setTimeout(function() {
done();
}, 10000);
})
});
});
});
});`
This works (or at least the test passes when the query is valid, which is what I wanted), but it's cumbersome and (so very) far from elegant or effective, specifically the usage of timeout rather than using promises, async of some sort, or maybe a different alternative I'm not yet familiar with.
Since most of the resources I found (http://alanhollis.com/node-js-testing-a-node-js-api-with-mocha-async-and-should/, https://developmentnow.com/2015/02/05/make-your-node-js-api-bulletproof-how-to-test-with-mocha-chai-and-supertest/, https://justinbellamy.com/testing-async-code-with-mocha/, just to name a few) discuss direct API testing rather than specific async functions, I would love to get some input/best practices tips from more experienced Noders.
You need to know when bunch of asynchronous operations complete. Elegant way to test that is to use promises and promise aggregation:
Promise.all([ promise1, promise2, promise3 ]).then(function(results) {
// all my promises are fulfilled here, and results is an array of results
});
Wrap your dataSources into a promises using bluebird. You don't need to modify tested code self, bluebird provides convenience method:
var Promise = require('bluebird')
var dataSources = [
source1,
source2,
source3,
source4
].map(Promise.promisifyAll);
Use newly promisified functions to create promise for each call:
context('standard call', function() {
var finalResults = {};
var promiseOfResults = datasources.map(function(dataSource) {
var users = userMocks(10);
// Promise.all will take an array of promises and return a promise that is fulfilled then all of promises are
return Promise.all( users.map(function(user) {
// *Async functions are generated by bluebird, via Promise.promisifyAll
return dataSource.getDataAsync(user)
.then(dataSource.convertDataAsync)
.then(function(processedData) {
if (finalResults[dataSource.sourceName]) {
finalResults[dataSource.sourceName] = finalResults[dataSource.sourceName].concat(processedData);
} else {
finalResults[dataSource.sourceName] = processedData;
}
});
});
});
// promiseOfResults consists now of array of agregated promises
it('should return something', function(done) {
// Promise.all agregates all od your 'datasource' promises and is fulfilled when all of them are
// You don't need the promise result here, since you agegated finalResults yourself
return Promise.all( promiseOfResults ).then(function() {
_.forEach(finalResults.keys, function(key) {
expect(finalResults[key]).to.not.be.empty;
expect(finalResults[key].length).to.be.greaterThan(0);
});
done();
});
});
Rest of your test should use same Promise.all( promiseOfResults ), unless you need new set of results.
Sorry for my Title, I don't know what can I put.
Can you help me please, I would like to print data from a "then" in a "then" ?
Thank you
models.book.find()
.then( function (content) {
var i = 0;
while (content[i]) {
models.author.findOne({"_id": content[i].author_id}, function(err, data) {
console.log(data); //here, it' good
content[i] = data;
MY_DATA = content;
return MY_DATA;
});
i++;
};
})
.then(function (result) {
console.log(result); // here I would like to print MY_DATA
});
There are a number of problems with your code, and I don't think it's behaving as you're expecting it to.
Chaining Promises
In order to effectively chain promises how you're expecting, each promise callback needs to return another promise. Here's an example with yours changed around a bit.
var promise = models.book.find().exec(); // This returns a promise
// Let's hook into the promise returned from
var promise2 = promise.then( function (books) {
// Let's only get the author for the first book for simplicity sake
return models.author.findOne({ "_id": books[0].author_id }).exec();
});
promise2.then( function (author) {
// Do something with the author
});
In your example, you're not returning anything with your callback (return MY_DATA is returning within the models.author.findOne callback, so nothing happens), so it's not behaving as you're expecting it to.
model.author.findOne is asynchronous
model.author.findOne is asynchronous, so you can't expect to call it multiple times in the callback without handling them asynchronously.
// This code will return an empty array
models.book.find( function (err, books) {
var i = 0, results = [];
while (books[i]) {
models.author.findOne({ "_id": books[i].author_id}, function (err, data) {
// This will get called long after results is returned
results.push(data);
});
i++;
};
return results; // Returns an empty array
});
Handling multiple promises
Mongoose uses mpromise, and I don't see a method to handle multiple promises together, but here's a way your case could be done.
var Promise = require('mpromise');
models.book.find().exec()
.then( function (books) {
var i = 0,
count = 0,
promise = new Promise(),
results = [];
while (books[i]) {
models.author.findOne({ "_id": books[i].author_id }, function (err, author) {
results.push(author);
count++;
// Keep doing this until you get to the last one
if (count === books.length) {
// Fulfill the promise to get to the next then
return promise.fulfill(results);
}
return;
});
}
return promise;
})
.then( function (results) {
// Do something with results
});
I don't know if this will work exactly like it is, but it should give you an idea of what needs to be done.
I am having my code as
function updateData(data){
data.forEach(function(obj){
users.find({_id:obj.userId}).toArray(
function(e, res) {
obj.userData = res;
console.log(res)
});
});
return data;
}
The problem is I am unable to find the updated data, I am trying to update my data and adding one more field to it based on userId. The data parameter is an array containing the output from comments table. hope you understand the scenario.
It looks that users.find({...}).toArray(function(...){...}) is going to be asynchronous, so there is no way that you can be sure that the db call has been completed and that each data.obj has been updated before data is returned.
Instead of using the javascript Array.prototype.forEach() function, you could use the NodeJS async.each function from the async library by Caolan which would iterate through the array, update each object and then return the data object only when all functions calls have completed.
For example:
var async = require("async");
function updateData(data){
async.each(data, function(obj, callback) {
users.find({_id:obj.userId}).toArray(
function(e, res) {
obj.userData = res;
callback(e);
}
);
},
function(error){
return data;
}
}
I have a loop that I need to run inside of a callback, unfortunately accessing the array outside of the callback leaves me with a blank array. I know why this happens, but I want to know the best solution to tackle this.
Gallery.prototype.getGallery = function(cb) {
self = this;
var cos = new pb.CustomObjectService();
var ms = new pb.MediaService();
var s = [];
cos.loadTypeByName('Gallery Image', function(err, gallery){
cos.findByType(gallery._id.toString(), function(err, rpy){
for(var i = 0; i < rpy.length; i++){
ms.loadById(rpy[i].Image, function(e,r){
s.push(r.location);
console.log(r.location); /* <-- logs expected data */
});
}
console.log(s[0]); /* <-- this is undefined */
});
});
};
Replace your for loop with a call to async.*; in this case async.map seems right. Pass a callback to async.map; it will be invoked when all the individual calls to ms.loadById are done, with the array of results.
async.map(
rpy,
function(elt, callback) {
ms.loadById(elt.Image, callback);
},
function(err, data) {
// comes here after all individual async calls have completed
// check errors; array of results is in data
}
);
If you want to go into the promises world, then wrap the calls to ms.loadById in a promise. Here's a roll-your-own version, but various versions of what is usually called promisify are also out there.
function loadByIdPromise(elt) {
return new Promise(function(resolve, reject) {
ms.loadById(elt.image, function(err, data) {
if (err) return reject(err);
resolve(data);
});
});
}
Then do a Promise.all on the resulting promises:
Promise.all(rpy.map(loadByIdPromise))
.then(function(data) {
// comes here when all individual async calls complete successfully
// data is your array of results
});
Using the promises style, your entire code would look like:
loadTypeByNamePromise('Gallery Image') .
then(function(gallery) { return findByTypePromise(gallery._id.toString(); }) .
then(function(rpy) { return Promise.all(rpy.map(loadByIdPromise)); }) .
then(function(results) { /* do something with [results] */ });