How to do good async? - node.js

I'm a PHP developer and currently i'm working on a node.js project, i've never experienced async before so it confusing me.
Am I really have to do it like this?
// user model
exports.getRandomUser = function(cb){
db.query('...query...', cb);
}
// post model
exports.getRandomPost = function(uid, cb){
db.query('...query...', cb);
}
// router
router.get('/', function(req, res) {
user.getRandomUser(function(userInfo){
post.getRandomPost(userInfo.id, function(postInfo){
res.render('post', {data: postInfo});
});
});
});
Is there any way to make it less confusing?

Great question, yes there is a more simplified manner. What you are doing here is callback after callback, which makes the code look 'confusing'
Commonly referred to as callback hell
It looks pretty now for standards, but in time it will grow into a hungry programming power beast that needs your attention all the time.
For a javascript pro it's not really hard to deal with it, but if you want to have a more relaxed style at approaching callbacks; you can use promises. Promises are a way into the future for JS, but I think it's good to understand BOTH
a callback mostly has two parameters it looks like this:
dothis(function (error, data) {
if (error) {
throw new Error(error)
}
console.log('we have data', data)
})
With promises, this becomes much more easy to understand in the semantic terms
dothis.then(function(data) {
console.log('we have data', data)
}).catch(function(error) {
throw new Error(error)
})
This of course only works if your functions are promise compatible, if you want to learn more about promises check out this tutorial github: https://github.com/then/promise
You can even chain promises and create a really clean codebase!

Related

Where does the callback within this express app coming from?

after some freecodecamp I started doing the Express js tutorial from MDN (https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/Displaying_data/Home_page 2) for some backend.
I am stuck at understanding where the callback in the async.parallel is coming from and what is represents.
If I delete the callback the site wont load, so it must have some important meaning but unfortunately I have no glue. Is it calling the function(err, results) { res.render(‘index’, […] }) to make the result availalble for data?
var Book = require(’…/models/book’);
var async = require(‘async’);
exports.index = function(req, res) {
async.parallel({
book_count: function(callback) {
Book.countDocuments({}, callback);
},
[...]
[...]
function(err, results) {
res.render('index', {
title: 'Local Library Home',
error: err, data: results
});
});
};
A Callback is a generic function invoked upon the completion of an asynchronous request. In this particular instance, the Callback is being utilized as a method of getting the data out of the asynchronous request to fill the number of books on your page. These are required because these query's are non-blocking, meaning Javascript will keep executing other surrounding code until the Callback is invoked. If you want more detail on how they work in general look here as previously mentioned by #dnp1204. I hope this answered you question.

Node.JS async callback

I have been using async module to manage my code flow and it's indeed a very powerful lib. However, as the code grows bigger and logical flow become more complex, I run into a problem with too many async_callback to manage. Some pseudo code to demo the issue as below
async.auto({
getjobs:function(cb){},
loopjobs:['getjobs']{
async.map(jobs,dojob(job, cb_map),cb);
},
}, function(err, result) {
//handle
});
function dojob(job, callback) {
async.auto({
step1:function(cb){},
step2:['step1', function(cb) {}],
step3:['step1', function(cb) {
moreloops(params, cb);
}]
}, callback)
}
function moreloops(params, cb) {
async.map(params, doloop(param, cb_map), cb);
}
function dbloop(param, cb) {
//dosomething
cb();
}
In this flow, a combination of loops, sequential, parallel and conditional async callback is being used due to the need of the business logic. It become quite difficult to troubleshoot if any function is not returning a callback for any reason. I tried to debug using logging framework but obviously it's not very efficient due to the asynchronous nature of node.js.
Is there any good way to
Improve the flow in coding
Identify in which function the flow stops.
Thanks!
Few ways to improve debugging
Use vasync module It's like async but with better debugging
vasync may not have all the tools you need so promisify (bluebird) all callbacks and use promises so that functions will return at least .then and .catch
For example, in request module:
var request = require('request');
function makeRequest(cb) {
request(url, function(err, res, body) {
cb(err, body);
});
}
becomes
var Promise = require('bluebird');
var request = Promise.promisify(require("request"));
Promise.promisifyAll(request);
function makeRequest(cb) {
request(url)
.then(function(response) {
var body = response.body;
cb(null, body);
})
.catch(cb);
}
Just pass all the success in .then and errors in .catch and pass errors back to the the highest level.

Node.js: What is the guarantee that we reply to the right req in asynchronous programming?

In all most all node.js tutorial I can see this pattern of coding.
app.get('/user', function (req, res) {
User.findById(req.body.id, function(err, user) {
if (err) throw err;
res.send(user);
});
});
Now this line res.send(user) is scaring me. User.findById is asynchronous and this api end point may get simultaneous hits and what is the guarantee that we have right res at the time of we reply to the req.
As of now I am creating closure to maintain the variable value.
app.get('/user', function (req, res) {
User.findById(req.body.id, function(res){
return function(err, user) {
if (err) throw err;
res.send(user);
}
}(res));
});
What is the best way to solve this problem?
You do not need to add closure protection for req and res. They are already properly scoped to one callback and are not shared or accessible with code running from other requests.
The app.get() function call has already created a closure for the req and res objects. So, any code within that callback or called from that callback and passed those arguments is guaranteed to get the right objects, no matter how much async coding is going on anywhere.
app.get('/user', function (req, res) {
// This is already a unique closure right here with unique copies of
// the req and res arguments. These arguments cannot be accessed
// by any other request or by any code out of this scope
// unless you copy them yourself to some shared location
// with concurrency access issues
User.findById(req.body.id, function(err, user) {
if (err) throw err;
res.send(user);
});
});
The only other thing that has to happen correctly is for node.js to never reuse those req and res objects for other requests that might happen while this one is being processed asynchronously. And, it DOES guarantee that. req and res are created new for each request.
Furthermore, there isn't even a way from one request to get access to the req or res from another request unless YOU somehow share it inappropriately (like stuffing it in a shared module level variable). So, as long as you only use the req or res directly from the callback in the lexical scope or you pass them to a function you are calling and none of your code actually saves them someplace that has concurrency access issues, then there are no issues.
So, to summarize. node.js itself does not have any concurrency issues with the req and res objects. So, as long as you don't create a problem by storing req or res in some shared location that has concurrency access issues from your own code, this is not a problem.
Wrong Code Example
Just to show you what wrong code would look like that does cause a problem:
var _res;
app.get('/user', function (req, res) {
_res = res;
User.findById(_req.body.id, function(err, user) {
if (err) throw err;
_res.send(user);
});
});
app.get('/account', function (req, res) {
_res = res;
User.findById(req.body.account, function(err, account) {
if (err) throw err;
_res.send(account);
});
});
Here, the code is assigning res to a higher scoped and shared variable and then attempting to use it after an async call. This would be bad and does create a concurrency issue. But, this isn't an issue with node.js per se, but an issue with bad code that isn't aware of proper concurrency design. And, there's no reason to copy a variable being used in an async operation to a higher and shared scope either. If it needs to be elsewhere, it should be passed as an argument as that will keep it unique per request.
Here are some reading references on "Javascript lexical scope".
What You Should Know About Javascript Scope
MDN: Lexical Scope and Closures
You Don't Know JS: Scopes & Closures
Everything You Wanted to Know About Javascript Scope
Your attempt is both incorrect and unnecessary. In this case the lexical scoping takes care of what you think is an issue.
app.get('/user', function (req, res) { // <- the `res` here
User.findById(req.body.id, function(err, user) {
if (err) throw err;
res.send(user); // is the same `res` here
});
});
due to the already existing closure created by definition of the function passed as the callback.
I think your scare is unfounded. When a request comes in each one gets its own reference to the callback function you pass to it. Which in this case is the one that handles the response. This function gets the res object, which is then used in the next callback function passed to the findById due to lexical scoping in JavaSript.
app.get('/user', function (req, res) {
User.findById(req.body.id, function(err, user) {
if (err) throw err;
res.send(user); // res here is unique for every new request,no need to worry :)
});
});

How can I reduce this code duplication in NodeJS/Mongoose

I am using NodeJS and Mongoose for an application that has users. And I have a large number of actions the server does on a particular user, depending on the request.
That means, I have this particular code fragment appearing in a lot of functions:
User.findOne({'email':req.user.email}, function (err, user) {
if (err) {
console.log('err');
res.send('Error');
}
if(!user){
console.log('err');
res.send('Error');
}
// do something with returned user
user.data = ....
...
user.save(function(err) {
if(err) {
console.log('err');
res.send('Error');
}
else {
console.log('success');
res.send('Success');
}
}
As you can see, there is a lot of code that replicates. The code that changes is the part 'do something with returned user'. Almost everything else (error messages, etc.) remains same.
So, how can I extract this part out? Since this is working on callback mechanism, is there a certain way to achieve this?
One way is to use Promises. It would involve finding a way to convert the Mongooose api to return Promises instead of using callbacks. After that, you could create code that would follow the lines of
User.findOne(...)
.then((user) => {
// do something with the returned user
return user.save();
}).then(() => {
console.log('success');
res.send('Success');
}).catch(() => {
console.log('err');
res.send('Error');
});
Promises resemble traditional synchronous coding where you can propagate errors akin to try-catch block and thus needing only one error handling location. This way you wouldn't have to replicate the console.log('err'); res.send('Error'); lines in multiple places.
You can read an introduction to Promises for example in "Promises - A Gentle Introduction". For the part on converting Mongoose to Promises there may be an existing module for this, or another approach that APIs use is to not give callback function as the last argument and then a Promise is returned instead. Unfortunately, I don't have exact knowledge in this particular Mongoose API.

Proper Handling of fetch errors for Mongoose?

This is a pure best practice question. I am pretty new to Node and Mongoose. I absolutely love the technology and have been cranking away on a project to build a JSON-backed API for an app that I'm building.
I am finding that I am continuously repeating code when I fetch objects from my database. For example:
Playlist.findById(req.params.id, function(err,playlist){
if (err)
return res.json({error: "Error fetching playlist"});
else if (!playlist)
return res.json({error: "Error finding the playlist"});
//Actual code being performed on the playlist that I'm fetching
});
The error handling at the top of the function call is annoying because I have to repeat that code for every call to the database... or so I think.
I thought about using a callback like:
var fetchCallback = function(err,objOrDoc,callback){
//Handle the error messages
callback(objOrDoc);
};
However, this approach would mess up my sequential flow since I would have to define the callback function before I performed the fetch. So, if I had a lot of database queries chained together, I would have to place the callbacks in reverse order, which is far from ideal in a clean-coding perspective.
I'm wondering if anyone has run into this issue and has any best practices for cutting down on the repetition.
I'm also using the express framework, so if there's a helpful way to handle it in express, I'd be interested to know, too.
There are a couple interesting approaches you could try here.
At the most simple, you could simply have a function that loads up an object and handles the output in an error condition.
fetchResource = function(model, req, res, callback) {
model.findById(req.params.id, function(err, resource) {
if (err)
return res.json({error: "Error fetching " + model.toString()});
else if (!playlist)
return res.json({error: "Error finding the " + model.toString()});
callback(resource);
});
};
app.on('/playlists/1', function(req, res) {
fetchResource(Playlist, req, res, function(playlist) {
// code to deal with playlist.
});
});
That's still quite a bit of duplication, so I might try to move this out into a middleware.
Route Middleware
Routes may utilize route-specific middleware by passing one or more additional callbacks (or arrays) to the method. This feature is extremely useful for restricting access, loading data used by the route etc.
Now I haven't tested this and it's a bit hand-wavey (read: pseudocode), but I think it should serve as a decent example.
// assuming a URL of '/playlist/5' or '/user/10/edit', etc.
function loadResource(model) {
return function(req, res, next) {
model.findById(req.params.id, function(err, resource) {
if (err)
return res.json({error: "Error fetching " + model.toString()});
else if (!resource)
return res.json({error: "Error finding the " + model.toString()});
req.resource = resource;
next();
});
}
}
app.get('/playlist/:id', loadResource(Playlist), function(req, res) {
var playlist = req.resource;
...
});
app.get('/user/:id', loadResource(User), function(req, res) {
var user = req.resource;
...
});
The express source contains a pretty good example of this pattern, and the middleware section in the docs (specifically under 'Route Middleware') details it further.

Resources