I have been using async module to manage my code flow and it's indeed a very powerful lib. However, as the code grows bigger and logical flow become more complex, I run into a problem with too many async_callback to manage. Some pseudo code to demo the issue as below
async.auto({
getjobs:function(cb){},
loopjobs:['getjobs']{
async.map(jobs,dojob(job, cb_map),cb);
},
}, function(err, result) {
//handle
});
function dojob(job, callback) {
async.auto({
step1:function(cb){},
step2:['step1', function(cb) {}],
step3:['step1', function(cb) {
moreloops(params, cb);
}]
}, callback)
}
function moreloops(params, cb) {
async.map(params, doloop(param, cb_map), cb);
}
function dbloop(param, cb) {
//dosomething
cb();
}
In this flow, a combination of loops, sequential, parallel and conditional async callback is being used due to the need of the business logic. It become quite difficult to troubleshoot if any function is not returning a callback for any reason. I tried to debug using logging framework but obviously it's not very efficient due to the asynchronous nature of node.js.
Is there any good way to
Improve the flow in coding
Identify in which function the flow stops.
Thanks!
Few ways to improve debugging
Use vasync module It's like async but with better debugging
vasync may not have all the tools you need so promisify (bluebird) all callbacks and use promises so that functions will return at least .then and .catch
For example, in request module:
var request = require('request');
function makeRequest(cb) {
request(url, function(err, res, body) {
cb(err, body);
});
}
becomes
var Promise = require('bluebird');
var request = Promise.promisify(require("request"));
Promise.promisifyAll(request);
function makeRequest(cb) {
request(url)
.then(function(response) {
var body = response.body;
cb(null, body);
})
.catch(cb);
}
Just pass all the success in .then and errors in .catch and pass errors back to the the highest level.
Related
after some freecodecamp I started doing the Express js tutorial from MDN (https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/Displaying_data/Home_page 2) for some backend.
I am stuck at understanding where the callback in the async.parallel is coming from and what is represents.
If I delete the callback the site wont load, so it must have some important meaning but unfortunately I have no glue. Is it calling the function(err, results) { res.render(‘index’, […] }) to make the result availalble for data?
var Book = require(’…/models/book’);
var async = require(‘async’);
exports.index = function(req, res) {
async.parallel({
book_count: function(callback) {
Book.countDocuments({}, callback);
},
[...]
[...]
function(err, results) {
res.render('index', {
title: 'Local Library Home',
error: err, data: results
});
});
};
A Callback is a generic function invoked upon the completion of an asynchronous request. In this particular instance, the Callback is being utilized as a method of getting the data out of the asynchronous request to fill the number of books on your page. These are required because these query's are non-blocking, meaning Javascript will keep executing other surrounding code until the Callback is invoked. If you want more detail on how they work in general look here as previously mentioned by #dnp1204. I hope this answered you question.
I'm looking for a solution to waiting for an event to happen before sending a HTTP response.
Use Case
The idea is I call a function in one of my routes: zwave.connect("/dev/ttyACM5"); This function return immediately.
But there exists 2 events that notice about if it succeed or fail to connect the device:
zwave.on('driver ready', function(){...});
zwave.on('driver failed', function(){...});
In my route, I would like to know if the device succeed or fail to connect before sending the HTTP response.
My "solution"
When an event happen, I save the event in a database:
zwave.on('driver ready', function(){
//In the database, save the fact the event happened, here it's event "CONNECTED"
});
In my route, execute the connect function and wait for the event to
appear in the database:
router.get('/', function(request, response, next) {
zwave.connect("/dev/ttyACM5");
waitForEvent("CONNECTED", 5, null, function(){
response.redirect(/connected);
});
});
// The function use to wait for the event
waitForEvent: function(eventType, nbCallMax, nbCall, callback){
if(nbCall == null) nbCall = 1;
if(nbCallMax == null) nbCallMax = 1;
// Looking for event to happen (return true if event happened, false otherwise
event = findEventInDataBase(eventType);
if(event){
waitForEvent(eventType, nbCallMax, nbCall, callback);
}else{
setTimeout(waitForEvent(eventType, callback, nbCallMax, (nbCall+1)), 1500);
}
}
I don't think it is a good practice because it iterates calls over the database.
So what are your opinions/suggestions about it?
I've gone ahead and added the asynchronous and control-flow tags to your question because at the core of it, that is what you're asking about. (As an aside, if you're not using ES6 you should be able to translate the code below back to ES5.)
TL;DR
There are a lot of ways to handle async control flow in JavaScript (see also: What is the best control flow module for node.js?). You are looking for a structured way to handle it—likely Promises or the Reactive Extensions for JavaScript (a.k.a RxJS).
Example using a Promise
From MDN:
The Promise object is used for asynchronous computations. A Promise represents a value which may be available now, or in the future, or never.
The async computation in your case is the computation of a boolean value describing the success or failure to connect to the device. To do so, you can wrap the call to connect in a Promise object like so:
const p = new Promise((resolve) => {
// This assumes that the events are mutually exclusive
zwave.connect('/dev/ttyACM5');
zwave.on('driver ready', () => resolve(true));
zwave.on('driver failed', () => resolve(false));
});
Once you have a Promise representing the state of the connection, you can attach functions to its "future" value:
// Inside your route file
const p = /* ... */;
router.get('/', function(request, response, next) {
p.then(successful => {
if (successful) {
response.redirect('/connected');
}
else {
response.redirect('/failure');
}
});
});
You can learn more about Promises on MDN, or by reading one of many other resources on the topic (e.g. You're Missing the Point of Promises).
Have you tried this? From the look of it, your zwave probably have already implemented an EventEmmiter, you just need to attach a listener to it
router.get('/', function(request, response, next) {
zwave.connect("/dev/ttyACM5");
zwave.once('driver ready', function(){
response.redirect(/connected);
});
});
There is a npm sync module also. which is used for synchronize the process of executing the query.
When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.
Sample code
/*require sync module*/
var Sync = require('sync');
app.get('/',function(req,res,next){
story.find().exec(function(err,data){
var sync_function_data = find_user.sync(null, {name: "sanjeev"});
res.send({story:data,user:sync_function_data});
});
});
/*****sync function defined here *******/
function find_user(req_json, callback) {
process.nextTick(function () {
users.find(req_json,function (err,data)
{
if (!err) {
callback(null, data);
} else {
callback(null, err);
}
});
});
}
reference link: https://www.npmjs.com/package/sync
So im completely stumped and hope someone can help with the combination of Node JS Async and Request modules. I'm attempting to build a list of file to download which I pass to Async, as an array of object contain all the information I need to download and store said file. After tons of debugging I discovered that Request are not even making there way out and I cant figure out why.
async.each(missingFiles,
function (obj, cb) {
console.log(obj.url);
//var file = nfs.createWriteStream(obj.fullPath);
request(obj.url, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(response)
}
cb();
})
},
function (err) {
if (err) {
console.log("Async failed");
}
}
);
I came across similar issues before. If you send response outside the async block, http request/response cycle ends before your async tasks complete. The fix is to have send response in the done() callback.
app.post("/download", function(req, res) {
async.eachSeries(missingFiles, function (obj, cb) {
...
//do your work here
cb()
}, function done() {
res.json({success: true});
});
}
Recently I switched from using callbacks to using promise in my rest api express app.
But I'm having trouble with unit testing routes/controller with async behaviour of the promise. Here is the sample code that needs to be unit tested.
var handler = function (req, res, next) {
var query = {},
var options = {
sort: { updatedAt: -1 },
limit: 10
};
if (req.query.before) {
query.updatedAt = { $lt: req.query.before };
}
// User.findAsync returns bluebird promise
User.findAsync(query, null, options).then(function (user) {
res.json(user);
}).catch(function (e) {
next(e);
});
}
router.get('/api/users', handler);
My approach to test above code was to spy on req, next, and User.findAsync and check if they are called with correct arguments. But because of async behaviour of the promise, I was having trouble to check if res.json or next are get called.
I've tried to stub findAsync to return resolved promise (Promise.resolve(user)). but still then callback is executed asynchronously.
I'm not sure if I'm on the right track for testing express application.
What is good strategy to test this kind of code in good separation?
I've also heard about using supertest.
But for me, Using supertest to test from http end point feels like more of integration testing which is not unit testing and is quite expensive.
Also, In general, I would like to know if it is good practice to try to cover all of the code with unit testing (models, controller, middleware, etc) and what's good strategies or techniques of doing that. Or If it is just good enough to test http end points with super test.
If your method being tested doesn't return a promise then you can't use the promise syntax in Mocha. You can test your method the same way you'd test any other asynchronous method - with done as a parameter of it. Let's say we want to test your handler function:
var handler = function (req, res, next) {
//...
User.findAsync(query, null, options).then(function (user) {
res.json(user);
}).catch(function (e) {
next(e);
});
}
We can write a test as such:
describe("The handler", function(){
it("calls res.json", function(done){ // note the done argument
handler({query: {before: 5}, // mock request
{json: done} // res.json calls our `done` param of this function
function(){ throw new Error("error called"); });
});
});
Note that we mocked the request, response and the next handler. Our mocked response has a json method that lets the test know it is complete (this can be a function if you want to make assertions inside it) and if next is called instead we throw to signal it's not something that was supposed to happen.
I would like to launch asynchronous http calls on my server node, i saw the async node module and i guess the async.parallel enables us to do that.
The documented example is pretty clear, but i don't know how i could manage multiple http calls.
I tried the example bellow but it doesn't even launch the http calls:
var http = require('http');
var Calls = [];
Calls.push(function(callback) {
// First call
http.get('http://127.0.0.1:3002/first' callback);
});
Calls.push(function(callback) {
// Second call
http.get('http://127.0.0.1:3002/second' callback);
});
var async = require('async');
async.parallel(Calls, function(err, results) {
console.log('async callback: '+JSON.stringify(results));
res.render('view', results);
});
If i launch the http requests separately, i do have a result, but but calling the async callback i get async callback: [null,null]
Have a look at the documentation:
With http.request() one must always call req.end() to signify that
you're done with the request - even if there is no data being written
to the request body.
You are creating a request, but you are not finalizing it. In your calls you should do:
var req = http.request(options, function(page) {
// some code
});
req.end();
This is assuming you are doing a normal GET request without body.
You should also consider using http.get which is a nice shortcut:
http.get("http://127.0.0.1:3002/first", function(res) {
// do something with result
});
Update The other thing is that callbacks in async have to be of the form
function(err, res) { ... }
The way you are doing it now won't work, because callback to http.get accepts only one argument res. What you need to do is the following:
http.get('http://127.0.0.1:3002/second', function(res) {
callback(null, res);
});
Ok the thing is to call the callback this way callback(null, res); instead callback(res);, i think the first parameter is interpreted as an error and the second one is the real result.
dont use capital names for other purpouses than types/classes
below is your code with corrected obvious mistakes
var http = require('http');
var calls = [];
calls.push(function(callback) {
// First call
http.get('http://127.0.0.1:3002/first', function (resource) {
resource.setEncoding('utf8');
resource.on('data', function (data) {
console.log('first received', data);
callback();
});
});
});
calls.push(function(callback) {
// Second call
http.get('http://127.0.0.1:3002/second', function (resource) {
resource.setEncoding('utf8');
resource.on('data', function (data) {
console.log('second received', data);
callback();
});
});
});
var async = require('async');
async.parallel(calls, function(err, results) {
console.log('async callback ', results);
res.render('view', results);
});