I am using MEAN (Mongo Express Angulars NodeJs) for a project. The problem is I have to add one extra attribute to data received from query object. And make new data array with exactly old data array but have one extra attribute. I know how to add attribute and pass them into callback using waterfall model, as I am using multiple callback functions and for loops I am not able to get expected result.
code:
var fetchRevenue = function(restaurantArray, type, startDate, endDate, fn) {
_.forEach(restaurantArray, function(rest) {
fetchDateWiseReport(new Date('07/10/2015'), new Date('07/16/2015'), rest._id, type, function(orders) {
var newOrders = [];
async.waterfall([
function(callback) {
if(orders && orders.length > 0){
async.forEach(orders, function(order) {
getSellingPriceOfItems(order.orders, function(sp) {
order.sp = sp;
newOrders.push(order);
if (newOrders.length === orders.length)
callback(null, newOrders);
});
});
} else {
newOrders.push([]);
}
},
function(newOrders, callback) {
var restArr = []
//get sum of all orders of each restaurant and add into restArr
callback(null, restArr);
},
function(restArr, callback) {
callback(null, newOrders);
}
], function(err, result) {
fn(result);
});
});
});
};
where my functions:
fetchDateWiseReport = fetches restaurant record for given date and send result in callback
getSellingPriceOfItems = query to item model find price for each item and return selling price of given array and send result in callback.
my complete code including all functions is here.
now I want orders should be equal to newOrders with additional attibute 'sp'. But I am unable to get this. Will you suggest me something to proceed?
Use Express way to handle callback problem
in you route
app.get('you/route',fetchDateWiseReport(), second(),finalReturningREsult())
your first function will be doing first async loop function assiggn reult in req.body.firstResult and pass to second function. and so on
Related
In a post function, I am trying to retrieve the nth activity of a user (since I have a dropdown that return the index number of the activity). When I run the query
collection.find({'local.email':req.user.local.email},
{'local.activities':{$slice : [currActivity,1]}});
I receive the correct activity object in Robo3T.
But, when I call the same query in Node inside a post function, it returns an undefined.
app.post('/addlog',function(req,res){
var currActivity = req.body.curAct;
var score = req.body.score;
var comment = req.body.reason;
mongoose.connect('mongodb://****:****#ds044907.mlab.com:44907/intraspect',function (err, database) {
if (err)
throw err
else
{
db = database;
var collection = db.collection('users');
var retrievedAct = collection.find({'local.email':req.user.local.email},
{'local.activities':{$slice : [currActivity,1]}}).toArray().then(console.log(retrievedAct));
if (retrievedAct.length > 0) { printjson (retrievedAct[0]); }
console.log(currActivity);
console.log(retrievedAct[0]);
// console.log(req.body.newAct);
collection.update({'local.activities.name':retrievedAct[0]},
{$push: {'local.activities.log' : {
comments: comment,
score: score,
log_time: Date.now()
}}})
.then(function(){
res.redirect('/homepage');
})
.catch(function() {
console.log('Error');
});
}
});
});
I checked that the currActivity variable does infact contain the integer value for the nth activity.
If you want the result of collection.find().toArray(), as specified in the docs, you have two options:
Passing a callback to .toArray() like you did with mongoose.connect()
Using the Promise that it returns if you don't pass a callback
Now you are doing neither of them.
Also, you are mixing callback style and Promises in your code. I recommend you unificate your code. If you are using a Node.js version bigger than 8, using async/await could be nice, it makes it simpler.
My input is streamed from another source, which makes it difficult to use async.forEach. I am pulling data from an API endpoint, but I have a limit of 1000 objects per request to the endpoint, and I need to get hundreds of thousands of them (basically all of them) and I will know they're finished when the response contains < 1000 objects. Now, I have tried this approach:
/* List all deposits */
var depositsAll = [];
var depositsIteration = [];
async.doWhilst(this._post(endpoint_path, function (err, response) {
// check err
/* Loop through the data and gather only the deposits */
for (var key in response) {
//do some stuff
}
depositsAll += depositsIteration;
return callback(null, depositsAll);
}, {limit: 1000, offset: 0, sort: 'desc'}),
response.length > 1000, function (err, depositsAll) {
// check for err
// return the complete result
return callback(null, depositsAll);
});
With this code I get an async internal error that iterator is not a function. But in general I am almost sure the logic is not correct as well.
If it's not clear what I'm trying to achieve - I need to perform a request multiple times, and add the response data to a result that at the end contains all the results, so I can return it. And I need to perform requests until the response contains less than 1000 objects.
I also looked into async.queue but could not get the hang of it...
Any ideas?
You should be able to do it like that, but if that example is from your real code you have misunderstood some of how async works. doWhilst takes three arguments, each of them being a function:
The function to be called by async. Gets argument callback that must be called. In your case, you need to wrap this._post inside another function.
The test function (you would give value of response.length > 1000, ie. a boolean, if response would be defined)
The final function to be called once execution is stopped
Example with each needed function separated for readability:
var depositsAll = [];
var responseLength = 1000;
var self = this;
var post = function(asyncCb) {
self._post(endpoint_path, function(err, res) {
...
responseLength = res.length;
asyncCb(err, depositsAll);
});
}
var check = function() {
return responseLength >= 1000;
};
var done = function(err, deposits) {
console.log(deposits);
};
async.doWhilst(post, check, done);
In my node.js, express app, I am making an ajax call with the superagent middleware. The call fetches database data in a complex array using the node-mysql middleware through quite a few database queries.
Before pasting the code, I am trying to explain in words what I am trying to do although the code would suffice to say what it wants to do with the addition that all the asynchronous things inside the first callback should be done in the synchronous way.
Explanation:
Inside the callback of the first query , a for loop is executed to run the second query multiple times and after each loop, the next loop is to be called only after the callback of the second query is complete. Things are same for the next code lines as well.
Code:
You can however skip the innards( marked in comments) of the for loops to make things brief and easy if you want.
conn.query("SELECT * FROM `super_cats`",function(error, results, fields) {
if(error){console.log("erro while fetching products for homepage "+ error);}
for(var i in results) { // FIRST FOR LOOP INSIDE THE FIRST QUERY CALLBACK
/*Innards of for loop starts*/
var elem = new Object();
var supcat_id=results[i].id;
elem.super_id =supcat_id;
elem.cats=new Array();
var cat= '';
/*Innards of for loop ends*/
conn.query("SELECT * FROM `categories` WHERE `supcat_id`="+supcat_id,function(error_cats, results_cats, fields_cats) {
if (error_cats) {console.log("erro while fetching cats for menu " + error_cats);}
for(var j in results_cats) {
/*Innards of for loop starts*/
cat= new Object();
var cat_id=results_cats[j].id;
cat.cat_id=cat_id;
cat.cat_name=results_cats[j].cat_name;
cat.subcats=new Array();
/*Innards of for loop starts*/
conn.query("SELECT * FROM `subcategories` WHERE `category`="+cat_id,function(error_subcats, results_subcats, fields_subcats) {
if (error_subcats) {console.log("erro while fetching subcats for menu " + error_subcats);}
for(var k in results_subcats ){
/*Innards of for loop starts*/
var subcat=new Object();
var subcat_id=results_subcats[k].id;
subcat.subcat_id=subcat_id;
subcat.subcat_name=results_subcats[k].subcategory;
cat.subcats.push(subcat);
elem.cats.push(cat);
/*Innards of for loop starts*/
}// end of for loop for results_subcats
});
}// end of for loop for result_cats
});
super_cats.push(elem);
}// end of for supercat results
res.send(super_cats)
});
I tried with the async middleware but in vain as I just could not figure out which function to use in this case .
To be brief, requirements are :
1) All the asynchronous things inside the first callback should be done in the synchronous way.
2) the response should be sent to the ajax call only after all the calculations are done and not before that (as it would probably happen if things were asynchronous as they are in the existing code, wouldn't it ?)
It may be just semantics, but it's important to understand that you cannot run this in a synchronous way. You have to run it asynchronously, and manage the order of the processing to get the desired effect. I find it useful to think about these kinds of problems more in terms of how I want to transform the data (à la functional programming) rather than the imperative code I would write in a more synchronous environment.
From what I can tell by the code, you want to end up with a data structure in super_cats that looks something like this:
[
{
super_id: 1,
cats: [
{
cat_id: 2,
cat_name: "Category",
subcats: [
{
subcat_id: 3,
subcat_name: "Subcategory"
},
...
]
},
...
]
},
...
]
Let's start by extracting this into a single function call with a single callback.
function getCategoryTree(callback) {
}
Now, then, let's take it from the top. You want to run a single asynchronous function (an SQL query), and you want to produce an array with one entry per result. That sounds like a map operation to me. However, since we want one of the values (cats) to be determined asynchronously, we need to use an asynchronous map, which the async library provides.
Let's just fill in the async.map signature for now; we want to map over our results (this is the functional equivalent of our for loop), and for each one we want to turn the result into something—the asynchronous function that does the something is called the iterator. Finally, once we have all our transformed array elements, we want to call the callback given to our function.
function getCategoryTree(callback) {
conn.query("SELECT * FROM `super_cats`", function(error, results, fields) {
async.map(results, iterator, callback);
});
}
Let's create a new function for getting the top-level category information, and use its name in place of our iterator placeholder.
function getCategoryTree(callback) {
conn.query("SELECT * FROM `super_cats`", function(error, results, fields) {
async.map(results, getSuperCategory, callback);
});
}
function getSuperCategory(resultRow, callback) {
}
Now we need to decide what we want to give back for each resultRow. Based on our diagram above, we want an object with super_id equal to the row's ID, and cats equal to all the categories in the top-level category. However, since cats is also determined asynchronously, we need to run the next query and transform those results before we can move on.
Similar to last time, we want each item in our cats array to be an object with some information from the query's result, but we also want a subcats array, which is again determined asynchronously, so we'll use async.map again. This time, however, we'll use an anonymous function for the callback, since we want to do something with the results before we give them to the higher-level callback.
function getSuperCategory(resultItem, callback) {
var supcat_id = resultItem.id;
conn.query("SELECT * FROM `categories` WHERE supcat_id` = " + supcat_id, function(error, results, fields) {
async.map(results, getCategory, function(err, categories) {
callback(err, { super_id: supcat_id, cats: categories });
});
});
}
As you can see, once this async.map is done, it means we have all the categories under this this super-category; thus, we can call our callback with the object we want to be in the array.
Now that that's done, we just need to implement getCategory. It will look very similar to getSuperCategory, because we want to do basically the same thing—for each result, return an object that has some data from the query, but also an asynchronous component.
function getCategory(resultItem, callback) {
var cat_id = resultItem.id;
var cat_name = resultItem.cat_name;
conn.query("SELECT * FROM `subcategories` WHERE `category` = " + cat_id, function(error, results, fields) {
async.map(results, getSubCategory, function(err, subcategories) {
callback(err, { cat_id: cat_id, cat_name: cat_name, subcats: subcategories });
});
});
}
Now, we just need to implement getSubCategory.
function getSubCategory(resultItem, callback) {
callback(null, {
subcat_id: resultItem.id,
subcat_name: resultItem.subcategory
});
}
Oops! The data we need from getSubCategory doesn't have an asynchronous component! It turns out we didn't need that last async.map at all; we could have used a regular array map; let's change getCategory and getSubCategory to work that way.
function getCategory(resultItem, callback) {
var cat_id = resultItem.id;
var cat_name = resultItem.cat_name;
conn.query("SELECT * FROM `subcategories` WHERE `category` = " + cat_id, function(error, results, fields) {
var subcategories = results.map(getSubCategory);
callback(error, { cat_id: cat_id, cat_name: cat_name, subcats: subcategories });
});
}
function getSubCategory(resultItem) {
return {
subcat_id: resultItem.id,
subcat_name: resultItem.subcategory
};
}
It's worth noting that our original method worked fine; if there's a chance getSubCategory ever has an async component, you could just leave it as it was.
And that's it! Here's the code that I wrote as I was writing this answer; note that I had to fake out the SQL a bit, but I think the idea is there:
var async = require("async");
// fake out sql queries
queryNum = 0;
var conn = {
query: function(query, callback) {
queryNum++;
var results = [1, 2, 3, 4, 5].map(function(elem) {
return {
id: queryNum + "-" + elem,
cat_name: "catname-" + queryNum + "-" + elem,
subcategory: "subcategory-" + queryNum + "-" + elem
};
});
callback(null, results, null);
}
};
function getCategoryTree(callback) {
conn.query("SELECT * FROM `super_cats`", function(error, results, fields) {
async.map(results, getSuperCategory, callback);
});
}
function getSuperCategory(resultItem, callback) {
var supcat_id = resultItem.id;
conn.query("SELECT * FROM `categories` WHERE supcat_id` = " + supcat_id, function(error, results, fields) {
async.map(results, getCategory, function(err, categories) {
callback(err, { super_id: supcat_id, cats: categories });
});
});
}
function getCategory(resultItem, callback) {
var cat_id = resultItem.id;
var cat_name = resultItem.cat_name;
conn.query("SELECT * FROM `subcategories` WHERE `category` = " + cat_id, function(error, results, fields) {
var subcategories = results.map(getSubCategory);
callback(error, { cat_id: cat_id, cat_name: cat_name, subcats: subcategories });
});
}
function getSubCategory(resultItem) {
return {
subcat_id: resultItem.id,
subcat_name: resultItem.subcategory
};
}
getCategoryTree(function(err, result) {
console.log(JSON.stringify(result, null, " "));
});
There are some inefficiencies here, but for simplicity's sake I've glossed over them. For example, rather than running the second sub-query over and over, you could query at once for all the category IDs, then query all the categories at once, etc. Then, once you have all the data, you could loop over each array synchronously to pull out the pieces you need.
In addition, there are better ways to store tree structures in relational databases; in particular, take a look at Modified Preorder Tree Traversal.
I am using nodejs+express+mongoose. Assuming I have 2 schemas and models in place: "Fruits" and "Vegetables".
Assuming I have the following:
var testlist = ["Tomato", "Carrot", "Orange"];
var convertedList = [];
// Assume res is the "response" object in express
I wan to be able to check each item in the array against the "fruits" and "vegetables" collections respectively and insert them into a converted list where Tomato, Carrot, and Broccoli are replaced with their respective documents.
Below I have some pseudocode of what I think it would be, but know not how to do this.
for(var i = 0; i < testlist.length; i++) {
var fruitfind = Fruit.find({"name":testlist[i]});
var vegfind = Vegetables.find({"name":testlist[i]});
// If fruit only
if(fruitfind) {
convertedList.push(fruitfindresults);
}
// If vegetable only
else if(vegfind) {
convertedList.push(vegfindresults);
}
// If identified as a fruit and a vegetable (assume tomato is a doc listed under both fruit and vegetable collections)
else if (fruitfind && vegfind) {
convertedList.push(vegfindresults);
}
}
// Converted List should now contain the appropriate docs found.
res.send(convertedList) // Always appears to return empty array... how to deal with waiting for all the callbacks to finish for the fruitfind and vegfinds?
What is the best way to do this? Or is this even possible?
Assuming there's only one of each fruit/vegetable and that you intended to push a veggie that's found in both collections twice.
var async = require("async"),
testlist = ["Tomato", "Carrot", "Orange"];
async.map(testlist, function (plant, next) {
async.parallel([function (done) {
Fruit.findOne({"name": plant}, done);
},
function (done) {
Vegetables.findOne({"name": plant}, done);
}], function (err, plants) { // Edited: before it was (err, fruit, veggie) which is wrong
next(err, plants);
});
},
function (err, result) {
var convertedList = [].concat(result);
res.send(convertedList);
});
Note: haven't actually tested the code, but it should work. The async module is excellent for managing callbacks like this btw.
Update
To get each fruit only once, the async.parallel callback simply have to be rewritten like this:
function (err, plants) {
next(err, plants[0] || plants[1]);
}
And there's no concat needed anymore in the .map callback:
function (err, result) {
res.send(result);
}
find is an asynchronous function, and it makes a request to the mongo database. This means two things:
The functions will not return results immediately. The find function in mongoose follows a very common async pattern. It accepts a "callback" function which it will call with either the results, or an error. By node convention, if the first argument is not null, it is an error.
// So typically you'd call find like this
SomeModel.find({your: 'conditions'}, function (err, results) {
if (err) {
// some error has occurred which you must handle
} else {
res.send(results);
}
})
// note that if code existed on the lines following the find, it would
// be executed *before* the find completed.
As every query is firing another request off to the database, you typically want to limit the number if you can. In this case, instead of finding each fruit/veg by name, you could look for all the names at once by using mongo's $in.
With these two things in mind, your code might look something like this:
// here *first* we're finding fruits
Fruit.find({name: {$in: testlist}}, function (err, fruits) {
// when the fruit request calls back with results, we find vegetables
Vegetable.find({name: {$in: testlist}}, function (err, vegetables) {
// finally concat and send the results
res.send(fruits.concat(vegetables));
});
});
To have both requests happen in parallel, a little more work is required. You could use a library like async, or write something yourself like:
var fruits
, vegetables
, done = function () {
if (fruits && vegetables) {
res.send(fruits.concat(vegetables));
}
}
Fruit.find({name: {$in: testlist}}, function (err, docs) {
fruits = docs;
done();
});
Vegetable.find({name: {$in: testlist}}, function (err, docs) {
vegetables = docs;
done();
});
Note that both examples here simply concat the results and send them, as it's not clear how you want the results processed. This means that if a tomato, for example, was in both lists, it would appear in the results twice, both the Vegetable and Fruit documents.
You'll also need to handle any errors coming back from mongoose.
Edit: Uniquely named docs
In light of your comment, this is one way you might return only one doc for Tomato (or other records that are both fruit and vegetable)
// after retrieving fruits and vegetables, create a map which will
// serve to weed out docs with duplicate names
var map = {};
fruits.forEach(function (fruit) {
map[fruit.name] = fruit;
});
vegetables.forEach(function (vegetable) {
map[vegetable.name] = vegetable;
});
var results = [];
// this would sort by name
Object.keys(map).sort().forEach(function (key, i) {
results[i] = map[key];
});
res.send(results);
Note that this sort of thing becomes much more complicated if you need to sort and paginate or otherwise limit the result of the two queries, and if you need that you might rather consider keeping the documents in the same collection.
I want to create in the Server script a function that can return a collection plus some extra value.
For example:
Meteor.publish("users", function () {
var users;
users = Meteor.users.find();
users.forEach(function (user){
user.profile.image = "some-url";
});
return users;
});
But this don't work proper. My question is: What is the right way to add a value to a collection reponse in a publish function.
There are 2 ways you can implement a publish function:
By returning a cursor (or an array of cursors)
By using this.added(), this.changed() and this.removed().
Only method 2 allows to modify returned documents.
Please refer to Meteor documentation here. However, since the provided sample code might look complex, here is another one:
// server: publish the rooms collection
Meteor.publish("rooms", function () {
return Rooms.find({});
});
is equivalent to:
// server: publish the rooms collection
Meteor.publish("rooms", function () {
var self = this;
var handle = Rooms.find({}).observeChanges({
added: function(id, fields) { self.added("rooms", id, fields); },
changed: function(id, fields) { self.changed("rooms", id, fields); },
removed: function(id) { self.added("rooms", id); },
}
});
self.ready();
self.onStop(function () { handle.stop(); });
});
In the second sample, you can modify the 'field' parameter before sending it for publication, like this:
added: function(id, fields) {
fields.newField = 12;
self.added("rooms", id, fields);
},
Source: this post.
Is this important to do with the server? You could use the transform function on the client:
Client JS
//Somewhere where it can run before anything else (make sure you have access to the other bits of the document i.e services.facebook.id otherwise you'll get a services is undefined
Meteor.users._transform = function(doc) {
doc.profile.image = "http://graph.facebook.com/" + doc.services.facebook.id + "/picture";
return doc;
}
Now when you do:
Meteor.user().profile.image
=> "http://graph.facebook.com/55592/picture"
I have opened an issue before with regards to sharing a transform onto the client: https://github.com/meteor/meteor/issues/821