When I make an API call, I want to update 2 separate collections with MongoDB. The official way of thing this will be using transactions but because this is a prototype for now, I will just use 2 separate function calls to update each collection.
Here is what I am doing:
async.waterfall([
function(callback) {
callback(null);
}, function(callback) {
connectToDatabase(MONGODB_URI)
.then(db => updateDocument1(arguments))
.then(result => {
callback(null);
});
}
, function(callback) {
connectToDatabase(MONGODB_URI)
.then(db => updateDocument2(arguments))
.then(result => {
callback(null);
});
, function(err, result) {
callback(null, null)
}
])
But somehow, it does not execute updateDocument2. Also, updateDocument1 stops in the middle of the process, so none of documents are updated.
Do you know why? And Does anyone know a better way?
async parallel : do function execution parallelly without dependent of
other function
async series : do function execution in series (one by one)
async waterfall : do first function execution then pass output of
first function as input to next function
now in your case : if updating two collection not depending on each other you can use async parallel
Related
I have a basic CRUD application using html forms, nodejs/express and mongodb. I have been learning about synchronous vs asynchronous code via callbacks, promises, and async/await and to my understanding for a crud application you would want the operations to be asynchronous so multiple users can do the operations at the same time. I am trying to implement aync/await with my express crud operations and am not sure if they are executing synchronously or asynchronously.
Here is my update function, which allows a user to type in the _id of the blog they want to change, then type in a new title and new body for the blog and submit it. In its current state, to my knowledge it is executing synchronously:
app.post('/update', (req, res) => {
const oldValue = { _id: new mongodb.ObjectId(String(req.body.previousValue)) }
const newValues = { $set: { blogTitle: req.body.newValue, blogBody: req.body.newValue2 } }
db.collection("miscData").updateOne(oldValue, newValues, function (err, result) {
if (err) throw err;
console.log("1 document updated");
res.redirect('/')
});
})
The way in which I was going to change this to asynchronous was this way:
app.post('/update', async (req, res) => {
const oldValue = { _id: new mongodb.ObjectId(String(req.body.previousValue)) }
const newValues = { $set: { blogTitle: req.body.newValue, blogBody: req.body.newValue2 } }
await db.collection("miscData").updateOne(oldValue, newValues, function (err, result) {
if (err) throw err;
console.log("1 document updated");
res.redirect('/')
});
})
Both blocks of code work, however I am not sure if the second block of code is doing what I am intending it to do, which is allow a user to update a blog without blocking the call stack, or if the second block of code would only make sense if I was running more functions after the await. Does this achieve the intended purpose, if not how could/should I do that?
db.collection(...).updateOne is always asynchronous, so you need not worry that a long-running database operation might block your application. There are two ways how you can obtain the asynchronous result:
With a callback function
db.collection(...).updateOne(oldValues, newValues, function(err, result) {...});
console.log("This happens synchronously");
The callback function with the two parameters (err, result) will be called asynchronously, after the database operation has completed (and after the console.log). Either err contains a database error message or result contains the database result.
With promises
try {
var result = await db.collection(...).updateOne(oldValues, newValues);
// Do something with result
} catch(err) {
// Do something with err
}
console.log("This happens asynchronously");
The updateOne function without a callback function as third parameter returns a promise that must be awaited. The statements that do something with result will be executed asynchronously, after the database operation has successfully completed. If a database error occurs, the statements in the catch block are executed instead. In either case (success or error), the console.log is only executed afterwards.
(If updateOne does not have a two-parameter version, you can write
var result = await util.promisify(db.collection(...).updateOne)(oldValues, newValues);
using util.promisify.)
Your second code snippet contains a mixture of both ways (third parameter plus await), which does not make sense.
Currently i have a dashboard that lists a bunch of records in a table. users can select 1 record and hit execute and i send a AJAX POST request to my routes middleware which executes 3 functions inside async.waterfall and returns a 200 response back to my client if everything works correctly. this async waterfall usually takes about 40-55 seconds to finish executing (fn_1,fn_2 and fn_3) and works perfectly fine.
router.post('/url', function(req, res, next) {
try {
async.waterfall([
fn_1,
fn_2,
fn_3
], function (err, body) {
res.writeHead(200, {'Content-Type': 'application/json'});
res.end(JSON.stringify({"error":err, "result":body}));
});
function fn_1(callback) {
callback(null, response);
}
function fn_1(result, callback) {
callback(err, result);
}
function fn_2(result, callback) {
callback(null, result);
}
}
catch (err){
console.log(err)
}
});
But, If i were to give provision for users to select MULTIPLE records and send that as an array back to my route middleware. how can i execute multiple async.waterfall methods for each item in the array in parallel
i can run a loop and execute the waterfall inside the loop but it again will wait for each item to complete and only then start the next iteration. this is not what i want.
is this doable in node / express . whats the easiest way to achieve this ? or are there modules/plugins that can help solve this case ?
Here is an abbreviated version of your code and how it could be changed to suit your needs. If none of your calls need data from any of the other calls, you can just run them in parallel with promises and use Promise.all to capture the result.
function fn_1(callback) {
// See function fn_2 for structure
}
function fn_1(result, callback) {
// See function fn_2 for structure
}
function fn_2(result, callback) {
return new Promise(resolve, reject => {
resolve(result)
})
.then(d => {
// Instead of callbacks, use a "then"
// block/statement.
//
// Do something with D here.
})
}
Promise.all([fn_1(), fn_2(), fn_3()])
.then(v => {
// Do somthing with v;
})
.catch(e => {
// Do something with e
})
I tend to advocate the use of native Promises over libraries like async, however, since your already using async...
You can use parallel and map each item in the array to a waterfall handler e.g.
async.parallel(
myArray.map(val => cb => async.waterfall(fn_1, fn_2, fn_3, cb)
, (err, results) => {
// return consolidated response
})
You would need to rework your waterfall handlers to not send a response but instead just propagate any errors.
It should also be noted that parallel is only useful if you are infact running I/O bound code, if the code is anything like your example then you won't really gain anything from using parallel over something like async.each
Suppose I have the following source code in NodeJS using promise (connecting to mongodb with 3 collections, well, this is not important)
function getData(){
return new promise((resolve, reject) => {
var dataArray = [];
getCollection1().then((rows1) => {
getCollection2().then((rows2) => {
var val = someDataProcessingFunction(rows1, rows2);
dataArray.push(val);
resolve(dataArray);
}, (err) => {
reject(err);
}
}, (err) => {
reject(err);
});
getCollection3().then((rows3) => {
rows3.forEach((row3) => {
dataArray.push(row3);
});
resolve(dataArray);
}, (err) => {
reject(err);
});
});
}
The problem is, there is multiple places where dataArray is pushed, but I don't know when all the pushing is completed. so that I don't know when to resolve, to return the data to user, for example, by response.send in express. As to the code above while I am using resolves several times, I will only get part of the whole data returned.
I can think out two workarounds, neither is good. one is to surround all resolves in setTimeout, which means I must delay the response to user.
another is to count all the push times at the beginning, every time subtract 1 when one item is pushed. when the times goes 0, do resolve. but it is often difficult to calculate the number before doing the real loops.
Is there better solutions? Thank you and love you :)
First of all, avoid the Promise constructor antipattern and use return values instead! For multiple layers of asynchronous functions, just use multiple layers of promises. To await multiple promises at once, use Promise.all:
function getData(){
return Promise.all([
getCollection1(),
getCollection2(),
getCollection3()
]).then(([rows1, rows2, rows3]) => {
return [someDataProcessingFunction(rows1, rows2)].concat(rows3);
});
}
I am trying to use async module on my nodejs application with not lucky.
Suppose i have the following:
/**
* parents=['parent1Template','parent2TEmplate',...]
* children=[[{id: 'child1Parent1'}{id: 'child2Parent1'}],[{id: 'child1Parent2"}{id: 'child2Parent2'}],...]
*
*/
function createTemplate(parents,children){
var final=[]
async.each(Object.keys(parents), function(item,done){
if(children[item].length!==0) addChildsByParent(parents[item],children[item], function (result) {
final.push(result);
});
done();
});
console.log("Final results: "+final);
return final;
}
function addChildsByParent (parent,childs,callback){
var template=[];
//some operations..
async.each(childs,function(child){
Children.findone({"_id": child.id}, function (err,ch)){
template.push(ch);
}
});
return callback(template)
}
I need to get in final all results when all operations have finished.
I have seen also functions as parallel and waterfall on async module but the main problem is that i need to work always with two arrays and do the query when i get the single value.
What's the best way, maybe something like this?
async.waterfall([
each()...
async.waterfall([
each()...
])
])
I would use async map instead of each to build the response array using the map callback.
Also, I would use async parallel instead of waterfall to improve speed, since operations don't depend on each other and can be executed in parallel.
async.parallel({
final : function(next) {
async.map(Object.keys(parents), function(item,done){
if(children[item].length!==0) addChildsByParent(parents[item],children[item], function (result) {
done(null, result);
});
}, next);
},
template : function(next) {
async.map(childs, function(child, done) {
Children.findone({"_id" : child.id}, function (err, ch) {
done(err, ch);
});
});
}
}, function(error, results){
if (!error) {
console.log(results);
// This will be {final:[], template:[]}
}
});
I'm using a save middleware in Mongoose to create a log of activity in the DB whenever some action is taken. Something like
UserSchema.post("save", function (doc) {
mongoose.model("Activity").create({activity: "User created: " + doc._id});
});
This appears to work fine, but the problem is that I can't test it because there is no way to pass a callback to post (which probably would not make sense). I test this out using mocha with:
User.create({name: "foo"}, function (err, user) {
Activity.find().exec(function (err, act) {
act[0].activity.should.match(new RegExp(user._id));
done();
});
});
The problem is that the Activity.create apparently does not finish before .find is called. I can get around this by wrapping .find in setTimeout, but this seems hacky to me. Is there any way to test asynchronous mongoose middleware operations?
Unfortunately, there's not a way to reliably interleave these two asynchronous functions in the way you'd like (as there aren't threads, you can't "pause" execution). They can complete in an inconsistent order, which leaves you to solutions like a timeout.
I'd suggest you wire up an event handler to the Activity class so that when an Activity is written/fails, it looks at a list of queued (hashed?) Activities that should be logged. So, when an activity is created, add to list ("onactivitycreated"). Then, it will eventually be written ("onactivitywritten"), compare and remove successes maybe (not sure what makes sense with mocha). When your tests are complete you could see if the list is empty.
You can use util.inherits(Activity, EventEmitter) for example to extend the Activity class with event functionality.
Now, you'll still need to wait/timeout on the list, if there were failures that weren't handled through events, you'd need to handle that too.
Edit -- Ignore the suggestion below as an interesting demo of async that won't work for you. :)
If you'd like to test them, I'd have a look at a library like async where you can execute your code in a series (or waterfall in this case) so that you can first create a User, and then, once it completes, verify that the correct Activity has been recorded. I've used waterfall here so that values can be passed from one task to the next.
async.waterfall([
function(done) {
User.create({name: "foo"}, function (err, user) {
if (err) { done(err); return; }
done(null, user._id); // 2nd param sent to next task as 1st param
});
},
function(id, done) { // got the _id from above
// substitute efficient method for finding
// the corresponding activity document (maybe it's another field?)
Activity.findById(id, function (err, act) {
if (err) { done(err); return; }
if (act) { done(null, true);
done(null, false); // not found?!
});
}
], function(err, result) {
console.log("Success? " + result);
});
Async post-save middleware will apparently be available in Mongoose 4.0.0:
https://github.com/LearnBoost/mongoose/issues/787
https://github.com/LearnBoost/mongoose/issues/2124
For now, you can work around this by monkey-patching the save method on the document so that it supports async post-save middleware. The following code is working for me in a similar scenario to yours.
// put any functions here that you would like to run post-save
var postSave = [
function(next) {
console.log(this._id);
return next();
}
];
var Model = require('mongoose/lib/model');
// monkey patch the save method
FooSchema.methods.save = function(done) {
return Model.prototype.save.call(this, function(err, result) {
if (err) return done(err, result);
// bind the postSave functions to the saved model
var fns = postSave.map(function(f) {return f.bind(result);});
return async.series(fns,
function(err) {done(err, result);}
);
});
};