We have a fairly complex code base in NodeJS that runs a lot of Promises synchronously. Some of them come from Firebase (firebase-admin), some from other Google Cloud libraries, some are local MongoDB requests. This code works mostly fine, millions of promises being fulfilled over the course of 5-8 hours.
But sometimes we get promises rejected due to external reasons like network timeouts. For this reason, we have try-catch blocks around all of the Firebase or Google Cloud or MongoDB calls (the calls are awaited, so a rejected promise should be caught be the catch blocks). If a network timeout occurs, we just try it again after a while. This works great most of the time. Sometimes, the whole thing runs through without any real problems.
However, sometimes we still get unhandled promises being rejected, which then appear in the process.on('unhandledRejection', ...). The stack traces of these rejections look like this, for example:
Warn: Unhandled Rejection at: Promise [object Promise] reason: Error stack: Error:
at new ApiError ([repo-path]\node_modules\#google-cloud\common\build\src\util.js:59:15)
at Util.parseHttpRespBody ([repo-path]\node_modules\#google-cloud\common\build\src\util.js:194:38)
at Util.handleResp ([repo-path]\node_modules\#google-cloud\common\build\src\util.js:135:117)
at [repo-path]\node_modules\#google-cloud\common\build\src\util.js:434:22
at onResponse ([repo-path]\node_modules\retry-request\index.js:214:7)
at [repo-path]\node_modules\teeny-request\src\index.ts:325:11
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
This is a stacktrace which is completely detached from my own code, so I have absolutely no idea where I could improve my code to make it more robust against errors (error message seems to be very helpful too).
Another example:
Warn: Unhandled Rejection at: Promise [object Promise] reason: MongoError: server instance pool was destroyed stack: MongoError: server instance pool was destroyed
at basicWriteValidations ([repo-path]\node_modules\mongodb\lib\core\topologies\server.js:574:41)
at Server.insert ([repo-path]\node_modules\mongodb\lib\core\topologies\server.js:688:16)
at Server.insert ([repo-path]\node_modules\mongodb\lib\topologies\topology_base.js:301:25)
at OrderedBulkOperation.finalOptionsHandler ([repo-path]\node_modules\mongodb\lib\bulk\common.js:1210:25)
at executeCommands ([repo-path]\node_modules\mongodb\lib\bulk\common.js:527:17)
at executeLegacyOperation ([repo-path]\node_modules\mongodb\lib\utils.js:390:24)
at OrderedBulkOperation.execute ([repo-path]\node_modules\mongodb\lib\bulk\common.js:1146:12)
at BulkWriteOperation.execute ([repo-path]\node_modules\mongodb\lib\operations\bulk_write.js:67:10)
at InsertManyOperation.execute ([repo-path]\node_modules\mongodb\lib\operations\insert_many.js:41:24)
at executeOperation ([repo-path]\node_modules\mongodb\lib\operations\execute_operation.js:77:17)
At least this error message says something.
All my Google Cloud or MongoDB calls have await and try-catch blocks around them (and the MongoDB reference is recreated in the catch block), so if the promise were rejected inside those calls, the error would be caught in the catch block.
A similar problem sometimes happens in the Firebase library. Some of the rejected promises (e.g. because of network errors) get caught by our try-catch blocks, but some don't, and I have no possibility to improve my code, because there is no stack trace in that case.
Now, regardless of the specific causes of these problems: I find it very frustrating that the errors just happen on a global scale (process.on('unhandledRejection', ...), instead of at a location in my code where I can handle them with a try-catch. This makes us lose so much time, because we have to restart the whole process when we get into such a state.
How can I improve my code such that these global exceptions do not happen again? Why are these errors global unhandled rejections when I have try-catch blocks around all the promises?
It might be the case that these are the problems of the MongoDB / Firebase clients: however, more than one library is affected by this behavior, so I'm not sure.
a stacktrace which is completely detached from my own code
Yes, but does the function you call have proper error handling for what IT does?
Below I show a simple example of why your outside code with try/catch can simply not prevent promise rejections
//if a function you don't control causes an error with the language itself, yikes
//and for rejections, the same(amount of YIKES) can happen if an asynchronous function you call doesn't send up its rejection properly
//the example below is if the function is returning a custom promise that faces a problem, then does `throw err` instead of `reject(err)`)
//however, there usually is some thiAPI.on('error',callback) but try/catch doesn't solve everything
async function someFireBaseThing(){
//a promise is always returned from an async function(on error it does the equivalent of `Promise.reject(error)`)
//yet if you return a promise, THAT would be the promise returned and catch will only catch a `Promise.reject(theError)`
return await new Promise((r,j)=>{
fetch('x').then(r).catch(e=>{throw e})
//unhandled rejection occurs even though e gets thrown
//ironically, this could be simply solved with `.catch(j)`
//check inspect element console since stackoverflow console doesn't show the error
})
}
async function yourCode(){
try{console.log(await someFireBaseThing())}
catch(e){console.warn("successful handle:",e)}
}
yourCode()
Upon reading your question once more, it looks like you can just set a time limit for a task and then manually throw to your waiting catch if it takes too long(because if the error stack doesn't include your code, the promise that gets shown to unhandledRejection would probably be unseen by your code in the first place)
function handler(promise,time){ //automatically rejects if it takes too long
return new Promise(async(r,j)=>{
setTimeout(()=>j('promise did not resolve in given time'),time)
try{r(await promise)} catch(err){j(err)}
})
}
async function yourCode(){
while(true){ //will break when promise is successful(and returns)
try{return await handler(someFireBaseThing(...someArguments),1e4)}
catch(err){yourHandlingOn(err)}
}
}
Elaborating on my comment, here's what I would bet is going on: You set up some sort base instance to interact with the API, then use that instance moving forward in your calls. That base instance is likely an event emitter that itself can emit an 'error' event, which is a fatal unhandled error with no 'error' listener setup.
I'll use postgres for an example since I'm unfamiliar with firebase or mongo.
// Pool is a pool of connections to the DB
const pool = new (require('pg')).Pool(...);
// Using pool we call an async function in a try catch
try {
await pool.query('select foo from bar where id = $1', [92]);
}
catch(err) {
// A SQL error like no table named bar would be caught here.
// However a connection error would be emitted as an 'error'
// event from pool itself, which would be unhandled
}
The solution in the example would be to start with
const pool = new (require('pg')).Pool(...);
pool.on('error', (err) => { /* do whatever with error */ })
Related
I am using a 3rd party package from npm, which in turn connects to some external API on IP address X.X.X.X and it crashes with the following error. The reason is clear, the network was down for a moment, and BOOM my entire program halts:
events.js:177
throw er; // Unhandled 'error' event
^
Error: connect ENETUNREACH X.X.X.X:80
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1054:14)
Emitted 'error' event at:
at Socket.socketErrorListener (_http_client.js:410:9)
at Socket.emit (events.js:200:13)
at emitErrorNT (internal/streams/destroy.js:91:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
at processTicksAndRejections (internal/process/task_queues.js:84:9) {
errno: 'ENETUNREACH',
code: 'ENETUNREACH',
syscall: 'connect',
address: 'X.X.X.X',
port: 80
}
I am deliberately not saying which module on npm it is that causes the problem because I can't control what 3rd party module authors are doing. If I complained to them they may or may not fix it but my coding life needs to go on. Even if any code I call has a coding error, it shouldn't completely crash my calling script. There should be a way for me to catch the condition before it cramps my style completely.
What I have tried: I use error handling meticulously, try/catch around any 3rd party async library calls etc. Example:
var ThirdPartyModule = require("thirdPartyModule");
try {
await ThirdPartyModule.doIt("some", "params");
} catch (err) {
console.log("Ok so the module call failed. Let's try something else here, but don't abort my script please!");
}
The catch does does nothing. Every time the module is called and the connection error happens, it completely tanks my entire program with the above "throw er;".
My question is, what additional "wrapper" code can I write around my calls to the library to "catch" any crashes in code that isn't mine, so my code keeps executing?
I am still kind of a noob in nodejs, so I think I am missing some greater nodejs concept that's at work here. I come from C++ or Java so whenever something crashes in these languages, I can always catch() it so I am puzzled how the module can "escape" try/catch in my main script.
Note that I am not looking for answers like "You should make sure to always be connected to the Internet, that's what ENETUNREACH is about." Instead, I am trying to understand how to be the master of the "sub modules" that I am calling, i.e. catch their errors, or if I can't be their master, I want to understand why what I want is impossible in nodejs.
Thanks!
EDIT: in a similar question linked-to below, a commenter suggests to add a process.on("uncaughtException") handler. I will try this out now.
If this is the correct solution, please post it as an answer and explain the underlying concept. Why can't I just rely on try / catch which is what a reformed Java programmer would do? What did the author of the module on npm do to not pass the Error up the chain in his async function?
EDIT 2: Perhaps my question ends up being a duplicate of Catch all uncaughtException for Node js app - I would love an explanation though why some crash deep down must be caught globally like that, and doesn't percolate up the caller chain. Is it because the 3rd party coder made an error, and failed to transform all possible problems that could occur in his code, into proper throws in his async code?
EDIT 3: Ok I managed to track down the problem. The library that I used was not the culprit. It innocently called and promisified a 4th-party library (also on npm) which connects to a server via http.request(). Just that the author of that 4th-party library forgot to install a request.on('error') callback, so when the Internet connection has problems (never happens, right!), that becomes an unhandled situation. The 3rd party library never gets the expected callback "on error" from the 4th party library, and thus never manages to promisify that error situation. Instead, the error lingers unhandled in nodejs ether, crashing my entire script. So there...
Making absolutely no assumptions about the structure of the code inside the library, you're stuck with the below:
process.on('uncaughtException', function (err) {
//do something
});
Reason being: If the library was using promises throughout, then the try/catch block would have caught it. It's very likely that inside the library the part throwing the error is inside a callback.
Best practice for async code should be to return an error to a callback, return a rejected Promise or throw inside a async block, and never throw inside functions using callbacks or raw Promises. However it seems extremely likely that is exactly what is happening. There is no way to handle the error gracefully at that point, only the catch-all uncaughtException.
Essentially in the case of a thrown error inside a Promise or a traditional async callback, the code is throwing a synchronous error and there is no synchronous try/catch block to handle it. And you can't inject one from outside the module.
So in short the error has no 'proper' mechanism to be handled. The third party module writers should be ashamed.
Try to chain it alike this:
await ThirdPartyModule.doIt("some", "params").catch(err => {
console.log(err);
});
(that's untested code, since the ThirdPartyModule is unknown).
Is there any way to catch an error that occurs in an async callback after an Express next() or res.send() has been called from middleware or a route handler? Consider the following code:
app.use('/throw-error', (req, res) => {
setTimeout(() => {
throw new Error('Async error causes thread death')
}, 500)
res.send('This thread is going to die...')
})
It will execute and send "This thread is going to die..." to the browser. It will also, a half second later, crash that Node thread it is running in. If you happen to be running an app that uses Node's cluster module, maybe it launches a new thread, but it died nonetheless. You might see something like this in your logs:
::1 [2019-07-17T18:54:55.142Z] - [71700] 4.740 ms "GET /throw-error" 200 -
/Users/moryl/Projects/crashtest/express.js:66
throw new Error('Async error causes thread death')
^
Error: Async error causes thread death
at Timeout.setTimeout [as _onTimeout] (/Users/moryl/Projects/InSight/sources/server/config/express.js:66:13)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)
That thread is now dead.
My question is, how the heck do you handle a (possibly unknown) async error that is outside the scope of a normal request, whether by design or through bad code? How do I prevent the thread from dying?
I don't want to be told that I shouldn't do this kind of stuff in async calls to begin with. I know this. I'm trying to write defensive code to catch "bad stuff" written by others.
This has been documented in express error handling doc:
You must catch errors that occur in asynchronous code invoked by route
handlers or middleware and pass them to Express for processing. For
example:
app.get('/', function (req, res, next) {
setTimeout(function () {
try {
throw new Error('BROKEN')
} catch (err) {
next(err)
}
}, 100)
})
The above example uses a try...catch block to catch errors in the
asynchronous code and pass them to Express. If the try...catch block
were omitted, Express would not catch the error since it is not part
of the synchronous handler code.
So, basically you need to try..catch the route. (the examples are basically same, mother of coincidence)
My question is, how the heck do you handle an async error that is outside the scope of a normal request, whether by design ...
You still want to handle errors in asynchronous code, even if it was fired and forgotten by design. Add a try { } catch { } or .catch to every independent task. With asynchronous code, Promises and async / awaithelp you (as they group independent callbacks into tasks, and you can handle errors per task then):
const timer = ms => new Promise(res => setTimeout(res, ms));
async function fireAndForgetThis() {
await timer(500);
throw new Error('Async error doesn't cause thread death, because its handled properly')
}
fireAndForgetThis()
.catch(console.error); // But always "handle" errors
... or through bad code?
Fix bad code.
How do I prevent the thread from dying?
That's not the thing you want to prevent. If an error occurs, and was not handled, your application gets into an unplaned state. Continuing execution might create even more problems. You don't want that. You want to prevent the unhandled rejection / unhandled error itself (by handling it properly).
For sure there are cases you can't handle, e.g. if the connection to the backing database goes down. In that case, NodeJS crashes, causes the monitoring to wake up DevOps that get the database back running. Crashing is also a form of handling the error ;)
If you read this far, and you still want to handle unhandled errors, don't. Okay well, you probably have your reasons, there you go.
Having problems understanding interaction of node processes and promise chains:
doSomethingAsync()
.then()
.then()
.catch()
.finally();
The finally was introduced to close db connections opened inside doSomethingAsync().
Question: In which block does a process.exit(1) on error properly belong?
In the .catch(), since that's where errors will go, or
In the .finally() since it is the last thing that should happen? (But if there is an error and catch() is triggered, do the connections get released)?
nowhere, because node already knows the program failed?
If the goal is to have the application terminate when an error occurs then I wouldn't catch the exception at all
async function doSomething() {
try {
const result = await doSomethingAsync();
// do something with result
} finally {
// do cleanup
}
}
Using async / await syntax will allow the Promise to throw the error and the uncaught exception would terminate the application. The finally block will run regardless of whether an error was thrown or not.
I think in your case process.exit(1) belongs in finally(), because there are database connections to be closed. You would probably want to close them first and then do process.exit(1).
If there was no logic to be performed, I would exit the process in catch().
If there are unhandled Promise rejections, Node.js currently just logs an error, with a mention that in the future such errors will cause a process to exit.
I have two questions regarding this:
How to best make Node.js today treat unhandled Promise rejections such that it exits the process?
If the failure of some promise is, in effect, the same as an uncaught exception and should cause exiting the process, how to best make that happen?
There are obviously lots of ways to do this, but I am trying to find the cleanest and simplest way that would not require me to make a new utility library just for this functionality.
Some candidates:
process.on('unhandledRejection', (err) => { throw err; }) - might be confusing as nothing shows it is from a promise, instead of just a thrown exception, and also not usable in .catch().
process.on('unhandledRejection', (err) => { console.error(err); process.exit(1); })
In most of the cases you almost never want your node server to exit when there is an exception or error. The very job of error handler (try/catch and .catch()) is to make sure your server stay up and running even when there is an exception or error.
Rather, when you get an exception or error, you should:
catch that error and throw a custom error with a specific error message (and error code if that suits you).
catch the custom error at the end of promise chain (or app.use(err,...)) and send that error message to the user.
Following is a simple psudocode where the user is giving you XML and expects parsed JSON as a response:
function parseXML(xmlStr){
return someParsingLibrary.parseXml2JSON(xmlStr)
.catch(()=>{
throw new Error("Your xml was not valid");
})
}
app.post("/parse", function (req, res, next) {
let xmlStr = req.body.xml;
parseXML(xmlStr)
.then((json)=>{
res.status(200).send({description:"Success", body:json});
})
.catch((err)=>{
res.status(422).send({description:err.message, body:err});
})
});
I know how to handle specific errors in promises but I sometimes have pieces of code that looks like this:
somePromise.then(function(response){
otherAPI(JSON.parse(response));
});
Sometimes, I get invalid JSON which causes a silent failure here when JSON.parse throws. In general I have to remember to add a .catch handler to every single promise in my code and when I don't I have no way to find out where I forgot one.
How do I find these suppressed errors in my code?
Edit
We've finally fixed this in Node.js 15, it took 5 years but native promise rejections now behave like uncaught exceptions - so it's fine to just add a process.on('uncaughtException' handler.
In Modern Node.js
Starting with io.js 1.4 and Node 4.0.0 you can use the process "unhandledRejection" event:
process.on("unhandledRejection", function(reason, p){
console.log("Unhandled", reason, p); // log all your errors, "unsuppressing" them.
throw reason; // optional, in case you want to treat these as errors
});
This puts an end to unhandled rejections problems and the difficulty of tracking them down in your code.
In Older NodeJS
These events were not yet back-ported to older versions of NodeJS and are unlikely to be. You can use a promise library that extends the native promise API such as bluebird which will fire the same events as there are in modern versions.
It's also worth mentioning that there are several userland promise libraries that offer the unhandled rejection detection facilities and much more such as bluebird (which also has warnings) and when.