I spent too much time trying to figure out why my express.js controller did not respond to a simple query, and figured out that runtime errors fired from a Mongoose-promise callback were silently interrupting the callback process.
Here is a simplified version of my code:
server.get('/api/test', function (req, res, next) {
User.find({}).exec().then(function success(users){
console.log('SUCCESS');
typo[0] = 1; // throws a runtime error
res.json(users);
}, function error(err){
console.log('ERROR');
res.json({error: err});
});
});
This results in SUCCESS showing up in my console, but nothing happens then. No response is given to the user, the error caused by my typo is not appearing in my console, and the error callback is not called either.
I am aware that one should not throw exceptions from a callback function, but in that case, this was just a typo, and it would make sense to me to be warned (e.g. a stack trace in my standard output) whenever one makes this kind of mistake. (we're humans, after all...)
In your opinion, what's the best way to get feedback whenever this kind of mistakes are made in promise callbacks?
This is Mongoose's fault for using a bad promise implementation. Promises are throw-safe so exceptions are caught (so they can be later handled by future code) - the future code never comes and Mongoose never reports that it did not. Good promise implementations do not suffer from this issue.
Your options are two:
Use a library like Bluebird:
var Promise = require("bluebird");
var mongoose = Promise.promisifyAll(require("mongoose"));
User.findAsync({}).then(function(data){
JSON.prase("dsa"); // not a silent failure, will show up, easy debugging
});
This has the advantage of being faster than mongoose promises so there is no performance penalty. Alternatively, if you're super conservative and don't want the performance and API gains of bluebird - you can use native promises:
// Promise is the native promise
Promise.resolve(User.find({}).exec()).then(function(data){
JSON.prase("dsa");
});
And then, assuming you're running a modern variant of nodejs (read: io.js v 1.4.1 or upper), you can subscribe to promise rejections:
process.on("unhandledRejection", function(p, why){
console.log("FOUND ERROR!!!!", p , why);
});
So exceptions are not silently suppressed.
The exec() has two promises
.then(function)
.then(null , function)
try this, I think it will help
server.get('/api/test', function(req, res, next) {
User.find({}).exec()
.then(function success(users) {
console.log('SUCCESS');
typo[0] = 1; // throws a runtime error
res.json(users);
})
.then(null, function error(err) {
console.log('ERROR');
res.json({
error: err
});
});
});
Related
Assume an Express route that makes a call to Mongoose and has to be async so it can await on the mongoose.find(). Also assume we are receiving XML but we have to change it to JSON, and that also needs to be async so I can call await inside of it.
If I do this:
app.post('/ams', async (req, res) => {
try {
xml2js.parseString(xml, async (err, json) => {
if (err) {
throw new XMLException();
}
// assume many more clauses here that can throw exceptions
res.status(200);
res.send("Data saved")
});
} catch(err) {
if (err instanceof XML2JSException) {
res.status(400);
message = "Malformed XML error: " + err;
res.send(message);
}
}
}
The server hangs forever. I'm assuming the async/await means that the server hits a timeout before something concludes.
If I put this:
res.status(200);
res.send("Data saved")
on the line before the catch(), then that is returned, but it is the only thing every returned. The client gets a 200, even if an XMLException is thrown.
I can see the XMLException throw in the console, but I cannot get a 400 to send back. I cannot get anything I that catch block to execute in a way that communicates the response to the client.
Is there a way to do this?
In a nutshell, there is no way to propagate an error from the xml2js.parseString() callback up to the higher code because that parent function has already exited and returned. This is how plain callbacks work with asynchronous code.
To understand the problem here, you have to follow the code flow for xml2js.parseString() in your function. If you instrumented it like this:
app.post('/ams', async (req, res) => {
try {
console.log("1");
xml2js.parseString(xml, async (err, json) => {
console.log("2");
if (err) {
throw new XMLException();
}
// assume many more clauses here that can throw exceptions
res.status(200);
res.send("Data saved")
});
console.log("3");
} catch (err) {
if (err instanceof XML2JSException) {
res.status(400);
message = "Malformed XML error: " + err;
res.send(message);
}
}
console.log("4");
});
Then, you would see this in the logs:
1 // about to call xml2js.parseString()
3 // after the call to xml2js.parseString()
4 // function about to exit
2 // callback called last after function returned
The outer function has finished and returned BEFORE your callback has been called. This is because xml2js.parseString() is asynchronous and non-blocking. That means that calling it just initiates the operation and then it immediately returns and the rest of your function continues to execute. It works in the background and some time later, it posts an event to the Javascript event queue and when the interpreter is done with whatever else it was doing, it will pick up that event and call the callback.
The callback will get called with an almost empty call stack. So, you can't use traditional try/catch exceptions with these plain, asynchronous callbacks. Instead, you must either handle the error inside the callback or call some function from within the callback to handle the error for you.
When you try to throw inside that plain, asynchronous callback, the exception just goes back into the event handler that triggered the completion of the asynchronous operation and no further because there's nothing else on the call stack. Your try/catch you show in your code cannot catch that exception. In fact, no other code can catch that exception - only code within the exception.
This is not a great way to write code, but nodejs survived with it for many years (by not using throw in these circumstances). However, this is why promises were invented and when used with the newer language features async/await, they provide a cleaner way to do things.
And, fortunately in this circumstance xml2js.parseString() has a promise interface already.
So, you can do this:
app.post('/ams', async (req, res) => {
try {
// get the xml data from somewhere
const json = await xml2js.parseString(xml);
// do something with json here
res.send("Data saved");
} catch (err) {
console.log(err);
res.status(400).send("Malformed XML error: " + err.message);
}
});
With the xml2js.parseString() interface, if you do NOT pass it a callback, it will return a promise instead that resolves to the final value or rejects with an error. This is not something all asynchronous interfaces can do, but is fairly common these days if the interface had the older style callback originally and then they want to now support promises. Newer interfaces are generally just built with only promise-based interfaces. Anyway, per the doc, this interface will return a promise if you don't pass a callback.
You can then use await with that promise that the function returns. If the promise resolves, the await will retrieve the resolved value of the promise. If the promise rejects, because you awaiting the rejection will be caught by the try/catch. FYI, you can also use .then() and .catch() with the promise, but in many cases, async and await are simpler so that's what I've shown here.
So, in this code, if there is invalid XML, then the promise that xml2js.parseString() returns will reject and control flow will go to the catch block where you can handle the error.
If you want to capture only the xml2js.parseString() error separately from other exceptions that could occur elsewhere in your code, you can put a try/catch around just it (though this code didn't show anything else that would likely throw an exception so I didn't add another try/catch). In fact, this form of try/catch can be used pretty much like you would normally use it with synchronous code. You can throw up to a higher level of try/catch too.
A few other notes, many people who first start programming with asynchronous operations try to just put await in front of anything asynchronous and hope that it solves their problem. await only does anything useful when you await a promise so your asynchronous function must return a promise that resolves/rejects when the asynchronous operation is complete for the await to do anything useful.
It is also possible to take a plain callback asynchronous function that does not have a promise interface and wrap a promise interface around it. You pretty much never want to mix promise interface functions with plain callback asynchronous operations because error handling and propagation is a nightmare with a mixed model. So, sometimes you have to "promisify" an older interface so you can use promises with it. In most cases, you can do that with util.promisify() built into the util library in nodejs. Fortunately, since promises and async/await are the modern and easier way to do asynchronous things, most newer asynchronous interfaces in the nodejs world come with promise interfaces already.
You are throwing exceptions inside the callback function. So you cant expect the catch block of the router to receive it.
One way to handle this is by using util.promisify.
try{
const util = require('util');
const parseString = util.promisify(xml2js.parseString);
let json = await parsestring(xml);
}catch(err)
{
...
}
First of all, I know one way to catch uncaught errors is use process.on('uncaughtException', err, function() {});.
I want to know how I can pass more details, or more context to the error. I want to be able to get the variables used. I'm not trying to recover from the error, only get more details of my environment when the error happened before shutting down the process. Yeah, the stack trace is nice, but I'd like to know how to replicate the error.
For example, I have this function:
function doTheBatman(var1) {
var var2 = 'whatever';
// this causes an uncaught exception later in the code
}
On process.on, I want to be able to access var1 and var2.
process.on('uncaughtException', function(err) {
// process.whatever doesn't contain any active variables
});
A synchronous exception in an Express route handler will be caught by Express itself and the exception will be passed off the default Express error handler where you can catch it yourself and the exception context is passed to that default express error handler.
You can see this code inside of Express where a route handler gets called:
Layer.prototype.handle_request = function handle(req, res, next) {
var fn = this.handle;
if (fn.length > 3) {
// not a standard request handler
return next();
}
try {
fn(req, res, next);
} catch (err) {
next(err);
}
};
The call to next(err) will pass the exception object off to the default Express error handler (which you can install a handler for).
If your code is throwing an exception inside of an asynchronous callback, then that is more complicated to catch in action. If you're using regular async callbacks (not promises), then the only way I know of to catch those at a meaningful spot is to put a try/catch inside of every async callback so you can capture the local stack info.
If you use promises at the lowest level and only program your logic and asynchronous code flow use promise functionality, then an exception in a promise handler will automatically turn into a rejected promise and certain promise libraries (like the Bluebird library can be configured to give you a pretty full stack trace of where things went wrong). Promises have this advantage because every promise .then() or .catch() handler is automatically wrapped in a try/catch handler and those are turned into promise rejections which propagate up the chain to the first .catch() handler they find.
For the non-Express example you just added to your question, you will just have to put a try/catch somehwere in the local neighborhood to catch a local synchronous exception:
function doTheBatman(var1) {
try {
var var2 = 'whatever';
// this causes an uncaught exception later in the code
} catch(e) {
console.log(e);
}
}
You can even set a debugger breakpoint in the catch handler and then examine variables in that scope when it is hit. I don't know of any way to examine the local variables at the point of the exception from the uncaughtException handler. err.stack should give you the stack trace, but not variable values.
Express/frameworks may offer more elegant/robust solutions, but if you're truly after what your question asks for... why not just capture variables outside of the functions scope? This is typically nasty and not considered best practice, but if you have a function that you know could be susceptive to failing often, perhaps the quick and dirty solution is what you need. You could always refine this later, but hopefully this demonstrates the approach...
var transactionVars = {};
function clearTransaction() {
transactionVars = {};
}
function doTheBatman(var1) {
transactionVars['var2'] = 'whatever';
// [...] bunch of stuff, possibly blowing up...
clearTransaction(); // we made it this far? cool, reset...
}
process.on('uncaughtException', function(err) {
console.log(transactionVars['var2']); // whatever
});
Furthermore, if you want to really dirty (in case these two functions are in two files) you can always tack transactionVars on the global object.
This is essentially a poor mans event emitter pattern, which I would highly recommend refactoring into once you grasp the general flow of how this works...
I am new to nodejs/expressjs and mongodb. I am trying to create an API that exposes data to my mobile app that I am trying to build using Ionic framework.
I have a route setup like this
router.get('/api/jobs', (req, res) => {
JobModel.getAllJobsAsync().then((jobs) => res.json(jobs)); //IS THIS THe CORRECT WAY?
});
I have a function in my model that reads data from Mongodb. I am using the Bluebird promise library to convert my model functions to return promises.
const JobModel = Promise.promisifyAll(require('../models/Job'));
My function in the model
static getAllJobs(cb) {
MongoClient.connectAsync(utils.getConnectionString()).then((db) => {
const jobs = db.collection('jobs');
jobs.find().toArray((err, jobs) => {
if(err) {
return cb(err);
}
return cb(null, jobs);
});
});
}
The promisifyAll(myModule) converts this function to return a promise.
What I am not sure is,
If this is the correct approach for returning data to the route callback function from my model?
Is this efficient?
Using promisifyAll is slow? Since it loops through all functions in the module and creates a copy of the function with Async as suffix that now returns a promise. When does it actually run? This is a more generic question related to node require statements. See next point.
When do all require statements run? When I start the nodejs server? Or when I make a call to the api?
Your basic structure is more-or-less correct, although your use of Promise.promisifyAll seems awkward to me. The basic issue for me (and it's not really a problem - your code looks like it will work) is that you're mixing and matching promise-based and callback-based asynchronous code. Which, as I said, should still work, but I would prefer to stick to one as much as possible.
If your model class is your code (and not some library written by someone else), you could easily rewrite it to use promises directly, instead of writing it for callbacks and then using Promise.promisifyAll to wrap it.
Here's how I would approach the getAllJobs method:
static getAllJobs() {
// connect to the Mongo server
return MongoClient.connectAsync(utils.getConnectionString())
// ...then do something with the collection
.then((db) => {
// get the collection of jobs
const jobs = db.collection('jobs');
// I'm not that familiar with Mongo - I'm going to assume that
// the call to `jobs.find().toArray()` is asynchronous and only
// available in the "callback flavored" form.
// returning a new Promise here (in the `then` block) allows you
// to add the results of the asynchronous call to the chain of
// `then` handlers. The promise will be resolved (or rejected)
// when the results of the `job().find().toArray()` method are
// known
return new Promise((resolve, reject) => {
jobs.find().toArray((err, jobs) => {
if(err) {
reject(err);
}
resolve(jobs);
});
});
});
}
This version of getAllJobs returns a promise which you can chain then and catch handlers to. For example:
JobModel.getAllJobs()
.then((jobs) => {
// this is the object passed into the `resolve` call in the callback
// above. Do something interesting with it, like
res.json(jobs);
})
.catch((err) => {
// this is the error passed into the call to `reject` above
});
Admittedly, this is very similar to the code you have above. The only difference is that I dispensed with the use of Promise.promisifyAll - if you're writing the code yourself & you want to use promises, then do it yourself.
One important note: it's a good idea to include a catch handler. If you don't, your error will be swallowed up and disappear, and you'll be left wondering why your code is not working. Even if you don't think you'll need it, just write a catch handler that dumps it to console.log. You'll be glad you did!
When an error occurs in NodeJS application, the server may crash and stop if an exception occurs.
How can I prevent this situation so that the server never stops on error, but returns and error code instead ?
--
EDIT
Here is a method which causes a server crash (table foo doesn't exist) :
app.get('/articles/list', function(req, res) {
connection.query('select * from foo', function(err, rows, fields) {
if (err) throw err;
res.send(JSON.stringify(rows));
});
});
-- BEST SOLUTION
Finally, I found the best solution for me :
http://rowanmanning.com/posts/node-cluster-and-express/
In summary, it consists in using a cluster of server, and to restart a server when it exits
If you are using express, you can use this
function clientErrorHandler (err, req, res, next) {
if (req.xhr) {
res.status(500).send({ error: 'Something failed!' })
} else {
next(err)
}
}
app.use(clientErrorHandler);
I suggest you to ready this article, it will clarify a lot for you.
JFK, be aware of async error catching, most common error, once you have async stuff which is wrapped in try-catch.
Also, avoid usage of
process.on("uncaughException", () => {....})
This is quite bad, because when an uncaught exception is thrown you can’t reliably continue your program at this point.Let it fail and restart with daemonizer's like pm2, forever and etc.
As general rule, you should handle all errors and promise rejects in your code to avoid uncaught Exceptions.
For errors: using try-catch statement
For promises: using .catch method
Btw Node permits you to intercept an uncaughtExceptions or unhandledRejection events, but is not a good practice to handle errors at this level and prevent program exits, because can generate unpredictable behaviours.
Use try catch... for parts of code, which throws exception
try{
//youre code here
}catch(error){
console.log(error)
}
It seems like error reporting/handling is done differently in Node.js+Express.js applications compared to other frameworks. Am I correct in understanding that it works as follows?
A) Detect errors by receiving them as parameters to your callback functions. For example:
doSomethingAndRunCallback(function(err) {
if(err) { … }
});
B) Report errors in MIDDLEWARE by calling next(err). Example:
handleRequest(req, res, next) {
// An error occurs…
next(err);
}
C) Report errors in ROUTES by throwing the error. Example:
app.get('/home', function(req, res) {
// An error occurs
throw err;
});
D) Handle errors by configuring your own error handler via app.error() or use the generic Connect error handler. Example:
app.error(function(err, req, res, next) {
console.error(err);
res.send('Fail Whale, yo.');
});
Are these four principles the basis for all error handling/reporting in Node.js+Express.js applications?
Error handling in Node.js is generally of the format A). Most callbacks return an error object as the first argument or null.
Express.js uses middleware and the middleware syntax uses B) and E) (mentioned below).
C) is bad practice if you ask me.
app.get('/home', function(req, res) {
// An error occurs
throw err;
});
You can easily rewrite the above as
app.get('/home', function(req, res, next) {
// An error occurs
next(err);
});
Middleware syntax is valid in a get request.
As for D)
(07:26:37 PM) tjholowaychuk: app.error is removed in 3.x
TJ just confirmed that app.error is deprecated in favor of E
E)
app.use(function(err, req, res, next) {
// Only handle `next(err)` calls
});
Any middleware that has a length of 4 (4 arguments) is considered error middleware. When one calls next(err) connect goes and calls error-based middleware.
People at Joyent have published a really insightful best-practices document on this. A must-read article for any Node.js developer.
Why first-parameter?
Because of the asynchronous nature of Node.js, the first-parameter-as-err pattern has become well established as a convention for userland Node.js error handling. This is because asynchronous:
try {
setTimeout(function() {
throw 'something broke' //Some random error
}, 5)
}
catch(e) {
//Will never get caught
}
So instead having the first argument of the callback is pretty much the only sensible way to pass errors asynchronously other than just throwing them.
To do so will result in an unhandled exception which, just in the way it sounds, implies that nothing was done to get the application out of its confused state.
Exceptions, why do they exist
It is worth noting however, that virtually all part of Node.js are event-emitters and the throwing of an exception is a low-level event which can be handled like all events:
//This won't immediately crash if connection fails
var socket = require("net").createConnection(5000);
socket.on("error", function(err) {
console.error("calm down...", err)
});
This can-but-shouldn't be taken to the extreme to catch all errors and make an application which will try very hard to never crash. This is a terrible idea in nearly every use-case, because it will leave the developer without any idea of what's going on in the application state and is analogous to wrapping main in try-catch.
Domains - grouping events logically
As part of dealing with this problem of exceptions making applications fall over, domains allow the developer to take, for example the Express.js application, and try and close off connections sensibly in the event of catastrophic failure.
ES6
It's probably mentioning that this will change again as ES6 allows the generator pattern to create asynchronous events which are still catchable with try/catch blocks.
Koa (written by TJ Holowaychuck, same original author of Express.js) noticeably does this. It uses the ES6 yield statement to create blocks that, while appearing nearly syncronous, are handled in the usual node asynchronous fashion:
app.use(function *(next) {
try {
yield next;
}
catch (err) {
this.status = err.status || 500;
this.body = err.message;
this.app.emit('error', err, this);
}
});
app.use(function *(next) {
throw new Error('some error');
})
This example was shamelessly stolen from here.