I'm a little confused to as how to properly use the async module. Say I have:
result = long_sync_io_function();
short_sync_function(result);
... //other stuff dependent on result
Generally, in Node you convert long_sync_io_function() to its asynchronous counterpart long_async_io_function(callback(err, result)) and do something like:
long_async_io_function(function(err, result) {
if (!err) {
short_sync_function(result);
...//other stuff dependent on result
}
});
But the constant embedding of callbacks quickly means lots and lots of indentation. Is the correct way to use async:
//1
async.waterfall([
function(callback) {
result = long_sync_io_function();
callback(null, result);
},
function(result, callback) {
short_sync_function(result);
//other stuff dependent on result
}
]);
or
//2
async.waterfall([
function(callback) {
long_async_io_function(callback);
},
function(result, callback) {
short_sync_function(result);
...//other stuff dependent on result
}
]);
Are these equivalent? I can't tell if aysnc helps create asynchronous code as in 1, or just aids in structuring existing asynchronous code as in 2.
The async library has no ability to create asynchronous functions/code. Instead it is meant as helpers for higher order structure/organisation of code that's already asynchronous.
So it's number 2.
Additional answer
For simply avoiding nesting callbacks you don't need to use the async library. Simply name your functions instead of declaring them inline. So, instead of:
long_async_io_function(function(err, result) {
if (!err) {
//..do stuff dependent on result
another_async_function(function(err,result) {
//..do other stuff
});
}
});
You can write it like this:
function do_other_stuff (err, result) {
//..
}
function do_stuff_with_io_result (err, result) {
//..
another_async_function(do_other_stuff);
}
long_async_io_function(do_stuff_with_io_result );
This makes the code self documenting and much easier to debug (especially if you're stepping through in a debugger or looking at a stack trace) and you can therefore remove the redundant comments like do stuff with result and do other stuff.
Related
As your project grows, we started to have this much appreciated, defensive code snippet pretty much everywhere :
func(err, result){
if(err){
console.log('An error occurred!, #myModule :' + err);
return callback(err);
}
//then the rest..
}
A quick google search reveals some libs that attempt to overcome this common concern, e.g. https://www.npmjs.com/package/callback-wrappers.
But what is the best approach to minimize the boilerplate coding without compromising the early error handling mechanism we have?
There are a couple of ways you can help to alleviate this issue, both use external modules.
Firstly, and my preferred method, is to use async, and in particular, async.series, async.parallel or async.waterfall. Each of these methods will skip straight to the last function if an error occurs in any of your async calls, thus preventing the splattering of if(err) conditions throughout your callbacks.
For example:
async.waterfall([
function(cb) {
someAsyncOperation(cb);
},
function(result, cb) {
doSomethingAsyncWithResult(result, cb);
}
], function(err, result) {
if(err) {
// Handle error - could have come from any of the above function blocks
} else {
// Do something with overall result
}
});
The other option is to use a promise library, such as q. This has a function Q.denodeify to help you wrap callback-style code into promise-style. With promises, you use .then., .catch and .done:
var qSomeAsyncOperation = Q.denodeify(someAsyncOperation);
var qDoSomethingAsyncWithResult = Q.denodeify(doSomethingAsyncWithResult);
Q()
.then(qSomeAsyncOperation)
.then(qDoSomethingAsyncWithResult)
.done(function(result) {
// Do something with overall result
}, function(err) {
// Handle error - could have come from any of the above function blocks
});
I prefer using async because it is easier to understand what is going on, and it is closer to the true callback-style that node.js has adopted.
In my gulp task, I just need to copy a file obtained from a previous task to a dest path that I have to determine at runtime using an asynchronous function (because I use fs functions).
How do I do this? How do I feed the result of an asynchronous function to a gulp.dest directive?
Sample code:
function getDest() {
fs.readdir('my/path', function(error, contents) {
// pick the right one
return the_right_one;
})
}
gulp.task('copy-stuff', ['previous-task'], function() {
return gulp.src('some/file').pipe(gulp.dest(getDest()));
})
You can do this as with any other async node/javascript code, by using callbacks.
For instance:
function getDest(cb) {
fs.readdir('my/path', function(error, contents) {
// pick the right one
cb(the_right_one);
})
}
gulp.task('copy-stuff', ['previous-task'], function() {
getDest(function(properDest) {
gulp.src('some/file')
.pipe(gulp.dest(properDest));
})
})
Note, in this example we don't return the stream or a promise, so you'll have some troubles using dependencies (this would finish before the a dependant would start). To solve this we can either return a promise that is resolved on end or simply use the callback given to the task.
Something like this should work (not tested):
gulp.task('copy-stuff', ['previous-task'], function(callback) {
getDest(function(properDest) {
gulp.src('some/file')
.pipe(gulp.dest(properDest))
.on('end', callback)
})
})
You can see more of the Orchestrator (the task runner gulp is using) API in the documentation or see the gulp documentation with async examples.
In 3.8 you can pass a function to gulp.dest that returns the destination for the file. This function has to be synchronous, so you could do
gulp.src('some/file')
.pipe(gulp.dest(function(file){
return 'build/' + fs.readFileSync('file_containing_target_dir');
}));
I have a codebase which contains code similar to the code below many times:
function(doc, callback) {
doSomething(function(err) {
if(err) return callback(err);
callback(null, doc);
});
}
I'm wondering if there are any downsides to just combining the explicit error check into:
function(doc, callback) {
doSomething(function(err) {
callback(err, doc);
});
}
I understand that callback handlers are expected to check the err on callback, but in this case it's just bubbling up.
I suppose I'm wondering if based on the way callbacks are generally used, if this is an issue?
There is no difference, the code is doing the same thing. First one is just easier to edit later if you want to add some postprocessing.
Technically, second example provides a "doc" and first don't, but if somebody rely on that, they're doing it very wrong.
I have the following example code - the first part can result in an async call or not - either way it should continue. I cannot put the rest of the code within the async callback since it needs to run when condition is false. So how to do this?
if(condition) {
someAsyncRequest(function(error, result)) {
//do something then continue
}
}
//do this next whether condition is true or not
I assume that putting the code to come after in a function might be the way to go and call that function within the async call above or on an else call if condition is false - but is there an alternative way that doesn't require me breaking it up in functions?
A library I've used in Node quite often is Async (https://github.com/caolan/async). Last I checked this also has support for the browser so you should be able to npm / concat / minify this in your distribution. If you're using this on server-side only you should consider https://github.com/continuationlabs/insync, which is a slightly improved version of Async, with some of the browser support removed.
One of the common patterns I use when using conditional async calls is populate an array with the functions I want to use in order and pass that to async.waterfall.
I've included an example below.
var tasks = [];
if (conditionOne) {
tasks.push(functionOne);
}
if (conditionTwo) {
tasks.push(functionTwo);
}
if (conditionThree) {
tasks.push(functionThree);
}
async.waterfall(tasks, function (err, result) {
// do something with the result.
// if any functions in the task throws an error, this function is
// immediately called with err == <that error>
});
var functionOne = function(callback) {
// do something
// callback(null, some_result);
};
var functionTwo = function(previousResult, callback) {
// do something with previous result if needed
// callback(null, previousResult, some_result);
};
var functionThree = function(previousResult, callback) {
// do something with previous result if needed
// callback(null, some_result);
};
Of course you could use promises instead. In either case I like to avoid nesting callbacks by using async or promises.
Some of the things you can avoid by NOT using nested callbacks are variable collision, hoisting bugs, "marching" to the right > > > >, hard to read code, etc.
Just declare some other function to be run whenever you need it :
var otherFunc = function() {
//do this next whether condition is true or not
}
if(condition) {
someAsyncRequest(function(error, result)) {
//do something then continue
otherFunc();
}
} else {
otherFunc();
}
Just an another way to do it, this is how I abstracted the pattern. There might be some libraries (promises?) that handle the same thing.
function conditional(condition, conditional_fun, callback) {
if(condition)
return conditional_fun(callback);
return callback();
}
And then at the code you can write
conditional(something === undefined,
function(callback) {
fetch_that_something_async(function() {
callback();
});
},
function() {
/// ... This is where your code would continue
});
I would recommend using clojurescript which has an awesome core-async library which makes life super easy when dealing with async calls.
In your case, you would write something like this:
(go
(when condition
(<! (someAsyncRequest)))
(otherCodeToHappenWhetherConditionIsTrueOrNot))
Note the go macro which will cause the body to run asynchronously, and the <! function which will block until the async function will return. Due to the <! function being inside the when condition, it will only block if the condition is true.
I'm using Mongoose with Node.js and have the following code that will call the callback after all the save() calls has finished. However, I feel that this is a very dirty way of doing it and would like to see the proper way to get this done.
function setup(callback) {
// Clear the DB and load fixtures
Account.remove({}, addFixtureData);
function addFixtureData() {
// Load the fixtures
fs.readFile('./fixtures/account.json', 'utf8', function(err, data) {
if (err) { throw err; }
var jsonData = JSON.parse(data);
var count = 0;
jsonData.forEach(function(json) {
count++;
var account = new Account(json);
account.save(function(err) {
if (err) { throw err; }
if (--count == 0 && callback) callback();
});
});
});
}
}
You can clean up the code a bit by using a library like async or Step.
Also, I've written a small module that handles loading fixtures for you, so you just do:
var fixtures = require('./mongoose-fixtures');
fixtures.load('./fixtures/account.json', function(err) {
//Fixtures loaded, you're ready to go
};
Github:
https://github.com/powmedia/mongoose-fixtures
It will also load a directory of fixture files, or objects.
I did a talk about common asyncronous patterns (serial and parallel) and ways to solve them:
https://github.com/masylum/i-love-async
I hope its useful.
I've recently created simpler abstraction called wait.for to call async functions in sync mode (based on Fibers). It's at an early stage but works. It is at:
https://github.com/luciotato/waitfor
Using wait.for, you can call any standard nodejs async function, as if it were a sync function, without blocking node's event loop. You can code sequentially when you need it.
using wait.for your code will be:
//in a fiber
function setup(callback) {
// Clear the DB and load fixtures
wait.for(Account.remove,{});
// Load the fixtures
var data = wait.for(fs.readFile,'./fixtures/account.json', 'utf8');
var jsonData = JSON.parse(data);
jsonData.forEach(function(json) {
var account = new Account(json);
wait.forMethod(account,'save');
}
callback();
}
That's actually the proper way of doing it, more or less. What you're doing there is a parallel loop. You can abstract it into it's own "async parallel foreach" function if you want (and many do), but that's really the only way of doing a parallel loop.
Depending on what you intended, one thing that could be done differently is the error handling. Because you're throwing, if there's a single error, that callback will never get executed (count won't be decremented). So it might be better to do:
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
And handle the error in the callback. It's better node-convention-wise.
I would also change another thing to save you the trouble of incrementing count on every iteration:
var jsonData = JSON.parse(data)
, count = jsonData.length;
jsonData.forEach(function(json) {
var account = new Account(json);
account.save(function(err) {
if (err) return callback(err);
if (!--count) callback();
});
});
If you are already using underscore.js anywhere in your project, you can leverage the after method. You need to know how many async calls will be out there in advance, but aside from that it's a pretty elegant solution.