I'm relatively new to node.js, wondering how to make my asynchronous code more reliable. Focusing on code that uses 'async' (no promises yet, due to legacy reasons).
One of my reliability concerns is failing to invoke callbacks. E.g.
async.series([
function(callback){
if(..)
console.log("invalid input, try again"); // BUG: NO CALLBACK!
else
callback(null, 'ok');
},
function(callback){
...
}
],
function(err,result){ handleErrorOrResult(err,result);} // MIGHT NOT BE REACHED
);
This code may never complete (never reach 'handleErrorOrResult') and I won't even know that anything went wrong. Since this pit is easy to fall into, I wonder whether there are ready-made library solutions for it? Any ideas would be welcome, but one direction might be timeouts: invoke some error handler if the whole thing isn't completed within, say, 5 minutes.
Thanks very much
Make your own wrapper for it that implements a timeout.
var myAsync = {
series: function (callbacks, timeout, done) {
var timeoutId = setTimeout(function () {
done("Maximum Time Exceeded for method!");
}, timeout);
async.series(callbacks, function () {
clearTimeout(timeoutId);
done.apply(this, arguments);
});
}
}
Now you can use it like:
myAsync.series(functions, functions.length * 500, callback);
Related
How do I make a chained function wait for the function before it, to execute properly?
I have this excerpt from my module:
var ParentFunction = function(){
this.userAgent = "SomeAgent";
return this;
}
ParentFunction.prototype.login = function(){
var _this = this;
request.post(
url, {
"headers": {
"User-Agent": _this.userAgent
}
}, function(err, response, body){
return _this;
})
}
ParentFunction.prototype.user = function(username){
this.username = username;
return this;
}
ParentFunction.prototype.exec = function(callback){
request.post(
anotherURL, {
"headers": {
"User-Agent": _this.userAgent
}
}, function(err, response, body){
callback(body);
})
}
module.exports = parentFunction;
And this is from within my server:
var pF = require("./myModule.js"),
parentFunction = new pF();
parentFunction.login().user("Mobilpadde").exec(function(data){
res.json(data);
});
The problem is, that the user-function won't wait for login to finish (Meaning, it executes before the login returns _this). So how do I make it wait?
You can't make Javascript "wait" before executing the next call in the chain. The whole chain will execute immediately in sequential order without waiting for any async operations to complete.
The only way I can think of to make this structure work is to create a queue of things waiting to execute and then somehow monitor the things in that queue that are async so you know when to execute the next thing in the queue. This requires making each method follow some sort of standard convention for knowing both whether the method is async and if it is async when the async operation is done and what to do if there's an error in the chain.
jQuery does something like this for jQuery animations (it implements a queue for all chained animations and calls each animation in turn when the previous animation is done). But, its implementation is simpler than what you have here because it works only with jQuery animations (or manually queued functions that follow the proper convention), not with other jQuery methods. You are proposing a mix of three different kinds of methods, two of which are not even async.
So, to make a long story shorter, what you are asking for could likely be done if you make all methods follow a set of conventions, but it's not easy. Since you appear to only have one async operation here, I'm wondering if you could do something like this instead. You don't show what the .exec() method does, but if all it does is call some other function at the end of the chain, then you'ld only have one async method in the chain so you could just let it take a callback and do this:
parentFunction.user("Mobilepadde").login(function(data) {
res.json(data);
});
I was working on a queued means of doing this, but it is not something I can write and test in less than an hour and you'd have to offer some ideas for what you want to do with errors that occur anywhere in the chain. Non-async errors could just throw an exception, but async errors or even non-async errors that occur after an async operation completes can't just throw because there's no good way to catch them. So, error handling becomes complex and you really shouldn't embark on a design path that doesn't anticipate appropriate error handling.
The more I think about this, the more I think you either want to get a library designed to handle the sequencing of async operations (such as the async module) and use that for your queueing of operations or you want to just give up on the chaining and use promises to support sequencing of your operations. As I was thinking about how to do error handling in the task queue with async operations, it occurred to me that all these problems have already been dealt with in promises (propagation of errors through reject and catching of exceptions in async handlers that are turned into promise rejections, etc...). Doing error handling right with async operations is difficult and is one huge reason to build off of promises for sequencing async operations.
So, now thinking about solving this using promises, what you could do is to make each async method return a promise (you can promisfy the entire request module with one call with many promise libraries such as Bluebird). Then, one .login() and .exec() return promises, you can do this:
var Promise = require('bluebird');
var request = Promise.promisifyAll(require('request'));
ParentFunction.prototype.login = function(){
return request.postAsync(
url, {
"headers": {
"User-Agent": this.userAgent
}
});
}
ParentFunction.prototype.exec = function(){
return request.postAsync(
anotherURL, {
"headers": {
"User-Agent": this.userAgent
}
}).spread(function(response, body) {
return body;
})
}
parentFunction.login().then(function() {
parentFunction.user("Mobilpadde");
return parentFunction.exec().then(function(data) {
res.json(data);
});
}).catch(function(err) {
// handle errors here
});
This isn't chaining, but it gets you going in minutes rather than something that probably takes quite awhile to write (with robust error handling).
Try this and see if it works:
ParentFunction.prototype.login = function(callback){
var _this = this;
request.post(
url, {
"headers": {
"User-Agent": _this.userAgent
}
}, function(err, response, body){
return callback(_this);
})
}
}
On the server side:
parentFunction.login(function(loggedin){
loggedin.user("Mobilpadde").exec(function(data){
res.json(data);
});
});
As your project grows, we started to have this much appreciated, defensive code snippet pretty much everywhere :
func(err, result){
if(err){
console.log('An error occurred!, #myModule :' + err);
return callback(err);
}
//then the rest..
}
A quick google search reveals some libs that attempt to overcome this common concern, e.g. https://www.npmjs.com/package/callback-wrappers.
But what is the best approach to minimize the boilerplate coding without compromising the early error handling mechanism we have?
There are a couple of ways you can help to alleviate this issue, both use external modules.
Firstly, and my preferred method, is to use async, and in particular, async.series, async.parallel or async.waterfall. Each of these methods will skip straight to the last function if an error occurs in any of your async calls, thus preventing the splattering of if(err) conditions throughout your callbacks.
For example:
async.waterfall([
function(cb) {
someAsyncOperation(cb);
},
function(result, cb) {
doSomethingAsyncWithResult(result, cb);
}
], function(err, result) {
if(err) {
// Handle error - could have come from any of the above function blocks
} else {
// Do something with overall result
}
});
The other option is to use a promise library, such as q. This has a function Q.denodeify to help you wrap callback-style code into promise-style. With promises, you use .then., .catch and .done:
var qSomeAsyncOperation = Q.denodeify(someAsyncOperation);
var qDoSomethingAsyncWithResult = Q.denodeify(doSomethingAsyncWithResult);
Q()
.then(qSomeAsyncOperation)
.then(qDoSomethingAsyncWithResult)
.done(function(result) {
// Do something with overall result
}, function(err) {
// Handle error - could have come from any of the above function blocks
});
I prefer using async because it is easier to understand what is going on, and it is closer to the true callback-style that node.js has adopted.
Assume that I have a Collection called Tasks which has few tasks in it.I call a method to return a task array to the user but for some reason it doesn't return anything.
Here is a code for example:
if (Meteor.isClient) {
// This code only runs on the client
Template.body.helpers({
tasks: function () {
// Show newest tasks first
Meteor.call("getTasks", function(error, result) {
return result; // Doesn't do anything..
});
}
});
}
Meteor.methods({
getTasks: function() {
return Tasks.find({}, {sort: {createdAt: -1}});
}
});
Any ideas why when I call the method it doesn't return anything?
Tasks.find() returns a cursor, which makes no sense to transmit to the client via DDP.
You probably mean to return Tasks.find().fetch(), but that defeats the purpose of Meteor's very nice data synchronization mechanism.
Have you read Understanding Meteor's publish/subscribe?
I have a codebase which contains code similar to the code below many times:
function(doc, callback) {
doSomething(function(err) {
if(err) return callback(err);
callback(null, doc);
});
}
I'm wondering if there are any downsides to just combining the explicit error check into:
function(doc, callback) {
doSomething(function(err) {
callback(err, doc);
});
}
I understand that callback handlers are expected to check the err on callback, but in this case it's just bubbling up.
I suppose I'm wondering if based on the way callbacks are generally used, if this is an issue?
There is no difference, the code is doing the same thing. First one is just easier to edit later if you want to add some postprocessing.
Technically, second example provides a "doc" and first don't, but if somebody rely on that, they're doing it very wrong.
I'm using Mongoose with Node.js and have the following code that will call the callback after all the save() calls has finished. However, I feel that this is a very dirty way of doing it and would like to see the proper way to get this done.
function setup(callback) {
// Clear the DB and load fixtures
Account.remove({}, addFixtureData);
function addFixtureData() {
// Load the fixtures
fs.readFile('./fixtures/account.json', 'utf8', function(err, data) {
if (err) { throw err; }
var jsonData = JSON.parse(data);
var count = 0;
jsonData.forEach(function(json) {
count++;
var account = new Account(json);
account.save(function(err) {
if (err) { throw err; }
if (--count == 0 && callback) callback();
});
});
});
}
}
You can clean up the code a bit by using a library like async or Step.
Also, I've written a small module that handles loading fixtures for you, so you just do:
var fixtures = require('./mongoose-fixtures');
fixtures.load('./fixtures/account.json', function(err) {
//Fixtures loaded, you're ready to go
};
Github:
https://github.com/powmedia/mongoose-fixtures
It will also load a directory of fixture files, or objects.
I did a talk about common asyncronous patterns (serial and parallel) and ways to solve them:
https://github.com/masylum/i-love-async
I hope its useful.
I've recently created simpler abstraction called wait.for to call async functions in sync mode (based on Fibers). It's at an early stage but works. It is at:
https://github.com/luciotato/waitfor
Using wait.for, you can call any standard nodejs async function, as if it were a sync function, without blocking node's event loop. You can code sequentially when you need it.
using wait.for your code will be:
//in a fiber
function setup(callback) {
// Clear the DB and load fixtures
wait.for(Account.remove,{});
// Load the fixtures
var data = wait.for(fs.readFile,'./fixtures/account.json', 'utf8');
var jsonData = JSON.parse(data);
jsonData.forEach(function(json) {
var account = new Account(json);
wait.forMethod(account,'save');
}
callback();
}
That's actually the proper way of doing it, more or less. What you're doing there is a parallel loop. You can abstract it into it's own "async parallel foreach" function if you want (and many do), but that's really the only way of doing a parallel loop.
Depending on what you intended, one thing that could be done differently is the error handling. Because you're throwing, if there's a single error, that callback will never get executed (count won't be decremented). So it might be better to do:
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
And handle the error in the callback. It's better node-convention-wise.
I would also change another thing to save you the trouble of incrementing count on every iteration:
var jsonData = JSON.parse(data)
, count = jsonData.length;
jsonData.forEach(function(json) {
var account = new Account(json);
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
});
If you are already using underscore.js anywhere in your project, you can leverage the after method. You need to know how many async calls will be out there in advance, but aside from that it's a pretty elegant solution.