I am performing rest api validation using grunt-mocha-test (test written in coffeescript). The client i'm using to call the api will throw a custom defined error if the response is anything other than a 200. In some cases, such a response simply means the database isn't ready with the data that I am validating, so I want to poll the service until either it is ready or I timeout. Since an error is thrown each time it's not ready (i.e. not 200), I want to wrap my calls in a try/catch block -- retrying in the catch block. Unfortunately, mocha throws my error instead of allowing my catch block to catch it.
Below is an example of my code:
Client = require 'rest-client'
client = new Client
describe "Try catch test", ->
it 'should catch my error', (done) ->
retry = (done) ->
try
client.mightThrowAnError, (response) ->
done()
catch error
retry done
retry done
Of course, in practice I have code in the catch block that eventually errors out after a number of retries so I don't recursively call retry forever, but I've omitted that here for simplicity, since mocha throws the error instead of ever allowing my catch block to handle it:
Uncaught Error: <my custom Error>
Is this a bug/design limitation with Mocha or am I doing something wrong?
[EDIT: The error thrown is a custom Error defined in my rest client]
I'm going to suggest a possibility. client.mightThrowAnError is doing something asynchronously. (Seems obvious.) If it is the asynchronous part of what client.mightThrowAnError which raises an error, this this will happen outside the try...catch block and will not be caught.
Related
We have a fairly complex code base in NodeJS that runs a lot of Promises synchronously. Some of them come from Firebase (firebase-admin), some from other Google Cloud libraries, some are local MongoDB requests. This code works mostly fine, millions of promises being fulfilled over the course of 5-8 hours.
But sometimes we get promises rejected due to external reasons like network timeouts. For this reason, we have try-catch blocks around all of the Firebase or Google Cloud or MongoDB calls (the calls are awaited, so a rejected promise should be caught be the catch blocks). If a network timeout occurs, we just try it again after a while. This works great most of the time. Sometimes, the whole thing runs through without any real problems.
However, sometimes we still get unhandled promises being rejected, which then appear in the process.on('unhandledRejection', ...). The stack traces of these rejections look like this, for example:
Warn: Unhandled Rejection at: Promise [object Promise] reason: Error stack: Error:
at new ApiError ([repo-path]\node_modules\#google-cloud\common\build\src\util.js:59:15)
at Util.parseHttpRespBody ([repo-path]\node_modules\#google-cloud\common\build\src\util.js:194:38)
at Util.handleResp ([repo-path]\node_modules\#google-cloud\common\build\src\util.js:135:117)
at [repo-path]\node_modules\#google-cloud\common\build\src\util.js:434:22
at onResponse ([repo-path]\node_modules\retry-request\index.js:214:7)
at [repo-path]\node_modules\teeny-request\src\index.ts:325:11
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
This is a stacktrace which is completely detached from my own code, so I have absolutely no idea where I could improve my code to make it more robust against errors (error message seems to be very helpful too).
Another example:
Warn: Unhandled Rejection at: Promise [object Promise] reason: MongoError: server instance pool was destroyed stack: MongoError: server instance pool was destroyed
at basicWriteValidations ([repo-path]\node_modules\mongodb\lib\core\topologies\server.js:574:41)
at Server.insert ([repo-path]\node_modules\mongodb\lib\core\topologies\server.js:688:16)
at Server.insert ([repo-path]\node_modules\mongodb\lib\topologies\topology_base.js:301:25)
at OrderedBulkOperation.finalOptionsHandler ([repo-path]\node_modules\mongodb\lib\bulk\common.js:1210:25)
at executeCommands ([repo-path]\node_modules\mongodb\lib\bulk\common.js:527:17)
at executeLegacyOperation ([repo-path]\node_modules\mongodb\lib\utils.js:390:24)
at OrderedBulkOperation.execute ([repo-path]\node_modules\mongodb\lib\bulk\common.js:1146:12)
at BulkWriteOperation.execute ([repo-path]\node_modules\mongodb\lib\operations\bulk_write.js:67:10)
at InsertManyOperation.execute ([repo-path]\node_modules\mongodb\lib\operations\insert_many.js:41:24)
at executeOperation ([repo-path]\node_modules\mongodb\lib\operations\execute_operation.js:77:17)
At least this error message says something.
All my Google Cloud or MongoDB calls have await and try-catch blocks around them (and the MongoDB reference is recreated in the catch block), so if the promise were rejected inside those calls, the error would be caught in the catch block.
A similar problem sometimes happens in the Firebase library. Some of the rejected promises (e.g. because of network errors) get caught by our try-catch blocks, but some don't, and I have no possibility to improve my code, because there is no stack trace in that case.
Now, regardless of the specific causes of these problems: I find it very frustrating that the errors just happen on a global scale (process.on('unhandledRejection', ...), instead of at a location in my code where I can handle them with a try-catch. This makes us lose so much time, because we have to restart the whole process when we get into such a state.
How can I improve my code such that these global exceptions do not happen again? Why are these errors global unhandled rejections when I have try-catch blocks around all the promises?
It might be the case that these are the problems of the MongoDB / Firebase clients: however, more than one library is affected by this behavior, so I'm not sure.
a stacktrace which is completely detached from my own code
Yes, but does the function you call have proper error handling for what IT does?
Below I show a simple example of why your outside code with try/catch can simply not prevent promise rejections
//if a function you don't control causes an error with the language itself, yikes
//and for rejections, the same(amount of YIKES) can happen if an asynchronous function you call doesn't send up its rejection properly
//the example below is if the function is returning a custom promise that faces a problem, then does `throw err` instead of `reject(err)`)
//however, there usually is some thiAPI.on('error',callback) but try/catch doesn't solve everything
async function someFireBaseThing(){
//a promise is always returned from an async function(on error it does the equivalent of `Promise.reject(error)`)
//yet if you return a promise, THAT would be the promise returned and catch will only catch a `Promise.reject(theError)`
return await new Promise((r,j)=>{
fetch('x').then(r).catch(e=>{throw e})
//unhandled rejection occurs even though e gets thrown
//ironically, this could be simply solved with `.catch(j)`
//check inspect element console since stackoverflow console doesn't show the error
})
}
async function yourCode(){
try{console.log(await someFireBaseThing())}
catch(e){console.warn("successful handle:",e)}
}
yourCode()
Upon reading your question once more, it looks like you can just set a time limit for a task and then manually throw to your waiting catch if it takes too long(because if the error stack doesn't include your code, the promise that gets shown to unhandledRejection would probably be unseen by your code in the first place)
function handler(promise,time){ //automatically rejects if it takes too long
return new Promise(async(r,j)=>{
setTimeout(()=>j('promise did not resolve in given time'),time)
try{r(await promise)} catch(err){j(err)}
})
}
async function yourCode(){
while(true){ //will break when promise is successful(and returns)
try{return await handler(someFireBaseThing(...someArguments),1e4)}
catch(err){yourHandlingOn(err)}
}
}
Elaborating on my comment, here's what I would bet is going on: You set up some sort base instance to interact with the API, then use that instance moving forward in your calls. That base instance is likely an event emitter that itself can emit an 'error' event, which is a fatal unhandled error with no 'error' listener setup.
I'll use postgres for an example since I'm unfamiliar with firebase or mongo.
// Pool is a pool of connections to the DB
const pool = new (require('pg')).Pool(...);
// Using pool we call an async function in a try catch
try {
await pool.query('select foo from bar where id = $1', [92]);
}
catch(err) {
// A SQL error like no table named bar would be caught here.
// However a connection error would be emitted as an 'error'
// event from pool itself, which would be unhandled
}
The solution in the example would be to start with
const pool = new (require('pg')).Pool(...);
pool.on('error', (err) => { /* do whatever with error */ })
Having enabled retries for my Cloud Function, I am wondering what all the cases are that it will retry.
This section seems contradictary to me. Hence, I assume that I do not understand it.
I thought that the following implies that the function that have "Retry on failure" enabled will also retry on fatal errors:
Cloud Functions does not understand a fatal error that shouldn't be retried
However, what is the point of catching errors and using Promise.reject in that case?
I tried to use Promise.reject to cancel retries of my Cloud Function, however, that does not happen.
return Promise.reject(..) // will retry
throw Error() // will retry
Both will retry for a week if the function is not redeployed.
To me it seems like fatal errors and Promise.reject will both retry, however, the documentation says:
you should modify your code accordingly if you support fatal errors
What is the point of this section in the documentation and what should I really do now?
Cloud Functions will retry a function if it yields any sort of error at the end. This includes a rejected promise, a thrown exception, or a timeout.
If you don't want a function to retry, then catch all errors, and return a successful promise. If you know that an error will never go away for a future retry (for example, bad input values), then don't arrange for it to be retried. Otherwise, you will just be charged for all the retries that don't work.
The whole point of retries is to handle cases where there are temporary errors, such as network problems. So, you should make sure that your function only yields an error for cases where it's expected that a function would eventually succeed some time in the future.
Feel free to use the "Send feedback" button on any page of documentation if you find that documentation to be confusing.
I think the code sample in the Google docs is simply wrong. I would suggest it should be:
return doFooAsync().catch(err => {
if (isFatal(err)) {
console.error(`Fatal error ${err}`);
return true;
}
throw err;
});
If the error is fatal (you don't want to retry), log the error and return true (you do not need to Promise.resolve(true) as the framework will wrap the return value in a Promise. If the error is not fatal (you want to retry), just re-throw the error (again, Promise.reject(err) is redundant).
Having problems understanding interaction of node processes and promise chains:
doSomethingAsync()
.then()
.then()
.catch()
.finally();
The finally was introduced to close db connections opened inside doSomethingAsync().
Question: In which block does a process.exit(1) on error properly belong?
In the .catch(), since that's where errors will go, or
In the .finally() since it is the last thing that should happen? (But if there is an error and catch() is triggered, do the connections get released)?
nowhere, because node already knows the program failed?
If the goal is to have the application terminate when an error occurs then I wouldn't catch the exception at all
async function doSomething() {
try {
const result = await doSomethingAsync();
// do something with result
} finally {
// do cleanup
}
}
Using async / await syntax will allow the Promise to throw the error and the uncaught exception would terminate the application. The finally block will run regardless of whether an error was thrown or not.
I think in your case process.exit(1) belongs in finally(), because there are database connections to be closed. You would probably want to close them first and then do process.exit(1).
If there was no logic to be performed, I would exit the process in catch().
I'm using Nodejs in my windows machine. the question is Nodejs always terminate process on errors e.g. empty Mysql insert statement.
So in production time, and without manual error handling, how can prevent NodeJs exit?
example code:
app.post('/api/accounts',function(req,res){
pool.getConnection(function(error,connection){
connection.query('insert into accounts set ?',req.body,function(err,results){
if (err) {
throw err ;
} else {
console.log(results) ;
}
});
});
console.log('post received') ;
console.log(req.body);
});
Imagine i post an empty req.body.
nodejs will exit on error like this
\node_modules\mysql\lib\protocol\Parser.js:77
throw err; // Rethrow non-MySQL errors
^
Is it possible to configure something in node to just show errors but don't exit?
It's not really a good thing to be continuing execution after a unhandled exception has been thrown by the interpreter (as Ginden said in his answer) - anything could happen and it could prove to be a mistake later, any sort of hole could easily be opened by stopping the process from cleaning up after something went so unexpectedly wrong in your code.
You could sensibly add a event handler for unhandledException like the answer by Ginden points out, however, it seems you're using express and it would make much more sense actually handling the error with middleware when it happens, instead of using throw as per your code.
Replace throw err; with return next(err); and that should mean the request will fall through to the next set of middleware, which should then handle the error, do some logging, tell the user, whatever you want it to do.
app.use(function(err, req, res, next) {
// Maybe log the error for later reference?
// If this is development, maybe show the stack here in this response?
res.status(err.status || 500);
res.send({
'message': err.message
});
});
Don't try to prevent process shutdown. If error was thrown, anything could happen.
Warning: Using 'uncaughtException' correctly
Note that 'uncaughtException' is a crude mechanism for exception handling intended to be used only as a last resort. The event should not be used as an equivalent to On Error Resume Next. Unhandled exceptions inherently mean that an application is in an undefined state. Attempting to resume application code without properly recovering from the exception can cause additional unforeseen and unpredictable issues.
Exceptions thrown from within the event handler will not be caught. Instead the process will exit with a non zero exit code and the stack trace will be printed. This is to avoid infinite recursion.
Attempting to resume normally after an uncaught exception can be similar to pulling out of the power cord when upgrading a computer -- nine out of ten times nothing happens - but the 10th time, the system becomes corrupted.
Domain module: don't ignore errors.
By the very nature of how throw works in JavaScript, there is almost never any way to safely "pick up where you left off", without leaking references, or creating some other sort of undefined brittle state.
The safest way to respond to a thrown error is to shut down the process. Of course, in a normal web server, you might have many connections open, and it is not reasonable to abruptly shut those down because an error was triggered by someone else.
The better approach is to send an error response to the request that triggered the error, while letting the others finish in their normal time, and stop listening for new requests in that worker.
I'm having trouble understanding how to handle something that it seems like would be a pretty basic aspect of express. If I have some code that throws an exception in an async callback, there is no way I can catch that exception because the try/catch block is no longer in scope by the time the callback is running. In these scenarios the browser will hang until it eventually give up stating that the server is unresponsive. This is a very bad user experience. I would much rather be able to immediately return a 500 error to the client. The default express error handler apparently does not handle this situation. Here is some sample code:
var express = require("express");
var app = express();
app.use(app.router);
//express error handler (never called)
app.use(function(err, req, res, next) {
console.log(err);
res.send(500);
});
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
a.b(); //blow up
});
});
app.listen(8888);
In the above code, the line a.b() throws a "ReferenceError: a is not defined" exception. The defined error handler is never called. Notice that the err object returned by fs.readFile() is null in this case because the file was correctly read. The bug is the code inside the async handler.
I have read this post about using node's uncaughtExpception even, but the documentation says not use that method. Even if I did use it, how would I then send the 500 response back to the user? The express response object is no longer around for me to use.
So how do you handle this scenario?
OK, I'm just going to post an entirely different answer since my first answer has some valuable information but is off topic in hindsight.
The short answer: the correct thing to do is what already happens: your program should print a stack trace and exit with an error.
The underlying thinking:
So I think you need to think about errors in different categories. My first answer deals with data-related errors that a well-written program can and should handle cleanly. What you are describing is a CRASH. If you read the node.js documentation you linked to, it is correct. The only useful thing your program can do at this point is exit with a stack trace and allow a process supervisor to restart it and attain an understood state. Once your program has crashed, it is essentially unrecoverable due to the extremely wide range of errors that could be the root cause of an exception getting to the top of the stack. In your specific example, this error is just going to continue happening every time until the source code bug is fixed and the application is redeployed. If you are worried that untested and buggy code is going to get into your application, adding more untested and buggy error handling code isn't really addressing the right problem.
But in short, no, there's no way to get a reference to the HTTP request object that caused this exception so AFAIK you cannot change the way this is perceived by the end user in the browser, outside of handling this at an intermediate reverse proxy layer where you could configure a crude timeout and send a more friendly error page (which of course would be useless for any request that isn't for a full HTML document).
The Bible of Error Handling in Node
Error Handling in Node.js by Dave Pacheco is the definitive work on this topic in my opinion. It is comprehensive, extensive, and thorough. I recommend reading and re-reading periodically.
To address #asparagino's comments, if an unhandled exception is easily reproducible or happening with high frequency, it's not an edge case, it's a bug. The correct thing to do is improve your code to not generate uncaught exceptions in face of this situation. Actually handle the condition, thus converting a programmer error into an operational error, and your program can continue without a restart and without an uncaught exception.
You should use express's error handling middleware via app.use(error, req, res, next). Express maintains a separate middleware stack that it uses when the normal middleware stack throws an uncaught exception. (Side note that express just looks at the callback's arity (number of expected arguments) to categorize it as regular middleware or error handling middleware, which is a bit magical, so just keep in mind you must declare the arguments as above for express to understand this is an error handling middleware).
And based on your question and comments, just understand that exceptions aren't all that useful in node.js because every async call gets a new stack, which is why callbacks are used everywhere and the 1st argument is an error universally. Your try/catch block in the example is only going to catch an exception thrown directly by findById (like if id were undefined, as it is in your snippet), but once an actual call to the database is made, that's it, the call stack is over and no further exceptions can happen until a totally different call stack starts when node invokes the async IO callback.
Thanks for the answer, but this only works if I put the try/catch inside the async callback and have the catch do next(exp). I'd like to avoid having separate try/catch blocks in every single async callback.
Nope, that's not true. You don't have to manually call next(exp). Express will catch the error and trigger the error handling middleware for you (that's how express does it's developer-friendly exception report pages in dev mode anyway). And async libraries don't throw exceptions even under "normal" error conditions. They pass an error to the callback, so in general you don't have to bother with try/catch that much in node. Just never ignore an error parameter passed to a callback function, and you're fine.
You don't see this boilerplate in node:
someDb.query(someCriteria, function (error, result) {
try {
//some code to deal with result
} catch (exception) {
callback(exception);
}
});
You do see this though:
someDb.query(someCriteria, function (error, result) {
if (error) {
callback(error);
return;
}
//some code to deal with result
});
Node handles IO differently, which means the call stack works differently, which means exceptions work differently, which means error handling works differently. You can write a stable node/express app that handles errors without crashing without writing a single try/catch. (express has one that handles uncaught errors that bubble all the way to the top). It's not a functional limitation, it's just a consquence of async IO that means you have to write your code differently and handle errors with callbacks instead of exceptions. Thinking of it as a "limitation" as opposed to the "way it is" is putting a negative connotation on something that is really just a technical reality. There are clean and robust patterns for exception handling in both a synchronous and asynchronous paradigm.