Node Exception Handling - node.js

What is the best way in node to handle unhandled expections that are coming out of core node code? I have a background process that runs and crawls web content and will run for long periods of time without issue, but every so often an unexpected exception occurs and I can't seem to gracefully handle it. The usual culprit appears to be some networking issue (lost connectivity) where the http calls I'm making fail. All of the functions that I have created follow the pattern of FUNCTION_NAME(error, returned_data), but in the situations where the error occurs I'm not seeing any of the functions I created in the call stack that is printed out, instead its showing some of the core node modules. I'm not really worried about these infrequent errors and their root cause, the purpose of this posting is just trying to find a graceful way of handling these exceptions.
I've tried putting a try/catch at the top level of my code where everything runs under but it doesn't seem to capture these exceptions. Is it good practice in node to use try/catch within all the lower level functions that use any core code? Or is there some way to globally capture all unhandled exceptions?
Thanks
Chris
UPDATED TO ADD STACK
node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: connect Unknown system errno 10060
at errnoException (net.js:642:11)
at Object.afterConnect [as oncomplete] (net.js:633:18)

is there some way to globally capture all unhandled exceptions?
You can catch all exceptions using process.on('uncaughtException'). Listening to this event will avoid the default action of printing the stack and exiting. However be conscious that ignoring exceptions may lead to problems in your app execution.
Link: http://nodejs.org/docs/latest/api/process.html#process_event_uncaughtexception
Pay attention to the documentation advice:
Note that uncaughtException is a very crude mechanism for exception handling. Using try / catch in your program will give you more control over your program's flow. Especially for server programs that are designed to stay running forever, uncaughtException can be a useful safety mechanism.

To catch network errors and avoid the default behavior (printing stack and exit) you have to listen to "error" events.
For example
var net = require('net');
var client = net.connect(80, 'invalid.host', function () {
console.log("Worked");
})
client.on('error', console.log);

I wrote about this recently at http://snmaynard.com/2012/12/21/node-error-handling/. A new feature of node in version 0.8 is domains and allow you to combine all the forms of error handling into one easier manage form. You can read about them in my post and in the docs.
You can use domains to handle callback error arguments, error event emitters and exceptions all in one place. The problem in this specific case is that when you dont handle an error event emitter, node by default will print the stack trace and exit the app.

I've put together a quick error handling file which logs and emails me whenever an unhandled exception is thrown. it then (optionally) tries to restart the server.
Check it out!

Related

"No server available to handle the request" thrown by AWS Elasticsearch Service

It doesn't happen regularly, but I was able to catch it in my debugger:
And then, because of that trailing comma, things further complicate. That error is not even well formatted JSON, which makes it throw DeserializationError, which is how it reaches my code. We can ignore this one, I just needed to bitch about it a little bit.
How do we find why is the "No server available to handle the request" happening? How to mitigate this?
If this is the expected behavior of an overloaded ES cluster, how do we properly handle this error specifically?
Here are the health metrics:
Here's context around the error:

Run a function when a ClientEmitter has an error

I'm using an event handler, where each event code is in it's own files. I'm attaching the events to the client, so when that file has the event emitted, it will run that code:
// looping through all event files
client.on(file.split('.')[0], require(`events/${file}`).bind(null, client, Util);
If the event file was message.js, it would be similar to:
client.on('message', require('events/message.js').bind(null, client, Util);
So when the message event is emitted, it runs the message.js file, passing along the client and Util classes.
I also have a function that is attached to the client called report. It basically reports when there is an error. I would like it so whenever any event from the client has an error, it will run that function.
I've done this slightly with the commands: command.run(...).catch(error => client.report(error)).
Is there a similar way to do this, instead of having to put a try-catch around all code in all the event files?
Try this way
client.on('error', require('events/report.js').bind(null, client, Util);
Error handling should be context driven. This means your bot's response to the error should be dependent on what it was doing, in what channel, etc - both for debugging and for the end user's information on what happened. You'll miss out on all of the context by letting errors just travel all the way up into an uncaught exception, and without the ability to create an error message, the user will just see the bot not respond and think it's down or the command is broken.
My suggestion: Create helper methods for your most common error producing functions that wrap them with error handling. I did this mostly for sending messages, as there's a myriad of things that could cause a message send to fail out of your control and the handling consists of generating an error message and attempting to send it in the channel or DM it to the user if that fails.

How to debug mystery ENOTFOUND?

I am getting random restarts on a PM2 managed nodejs cluster. The only symptom I get on the error log is of the following pattern - an ENOTFOUND on dns.js.
Error: getaddrinfo ENOTFOUND walkinto.inhttp walkinto.inhttp:80
at errnoException (dns.js:28:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
Clearly the problem is a malformed server name - walkinto.inhttp is incorrect and it should be walkinto.in . The challenge is that this is not a host name hard coded in the code. There are many places in this fairly large code base that makes name resolution and it is of dynamic nature.
I have spent considerable time to pinpoint the root cause but so far have had no luck. I need help to print more log information from dns.js; probably a call stack 'may' would help to move forward.
Q1 : How to enable more detailed logging on nodejs core modules?
Q2 : What could cause a nodejs restart to happen for an ENOTFOUND? How to avoid a restart - This path is not desirable.
Q3: Are there any other smarter way to trouble shoot this problem?
Since there's no way for us to help you solve the issue without some code to go on, I'll answer your questions:
How to enable more detailed logging on nodejs core modules?
Run node with the inspect option and attach to the debugger with Chrome DevTools or another application. See these links:
https://nodejs.org/api/debugger.html
https://nodejs.org/en/docs/guides/debugging-getting-started/
What could cause a nodejs restart to happen for an ENOTFOUND? How to avoid a restart - This path is not desirable.
The Node runtime isn't restarting. The error you're seeing is generated from something similar to throw new Error(`getaddrinfo ${err}`), and any uncaught error from throw will crash the runtime.
The restart is happening because you run the app via PM2, and can be disabled by passing the --no-autorestart option to PM2. If you want to avoid the application from crashing, you should wrap whatever code that this could be generated from in a try/catch-block, and try to recover from the error.
Are there any other smarter way to trouble shoot this problem?
This is most likely not an issue with the dns stdlib module. If I understand correctly, you are performing name resolutions on dynamically generated data, and that is most likely your issue. Somewhere in the code you have one or more functions that are either not validating the generated data or are generating invalid data due to a bug. We can't help you solve that unfortunately, since you haven't provided any code to go on. Would be great if you could try to pinpoint what code might cause this and update the question with it.
I was getting this error in my request that was something like this:
var optionsSearch = {
host: 'https://mysite.sharepoint.com',
path: '_api/search/query?querytext="sharepoint"',
method: 'GET'
};
All did was removing the https:// leaving only mysite.sharepoint.com and it was fixed.

NodeJS process 'uncaughtException' not working in some circumstances

I have a nodeJS project works fine in my local when using
process.on('uncaughtException', function (err) {
console.log('*********some code*************');
});
This snippet of code gets called when any sync or any async Error happens.
But when I put my project to two the other Linux hosts:
Ubuntu 14.04.3 LTS,
Ubuntu 14.04 LTS.
One works fine, but the other never gets called when Error happens and project shutdown directly.
Any suggestions?
I would refer you to the Node.js documentation about using the uncaughtException event correctly.
The most important part is: "It is not safe to resume normal operation after 'uncaughtException'." The code sample you have here shows that you are logging something, but not exiting the process.
It is possible that on your other system that the process is dying due to the exception, perhaps because of a native module that is having a segmentation fault or other process-ending crash.
Consider catching your exceptions at the points where they occur, and using uncaughtException as an event of last resort for logging fatal errors, while handing off to process.exit(1) to end the process with an error code.

Random 'ECONNABORTED' error when using sendFile in Express/Node

I have set a node server with Express middleware. I get the ECONNABORTED error randomly on some files when loading an HTML file which triggers about 10 other loads (js, css, etc.). The exact error is:
{ [Error: Request aborted] code: 'ECONNABORTED' }
Generated by this simplified code (after I tried to debug the issue):
res.sendFile(res.locals.physicalUrl,function (err) {
if (err)
console.log(err);
...
}
Many posts talk about this error resulting from not specifying the full path name. That is not the situation here. I do specify the full path and indeed the error is randomly generated. There are times when the page and all its subsequent links load perfectly and there are times when they do not. I tried to flush the cache and did not find any pattern to connect it with this.
This specific error appears to be a a generic term for socket connection getting aborted and is discussed in the context of other applications like FTP.
Having realized that the node worker threads can be increased, I tried to do so using:
process.env.UV_THREADPOOL_SIZE = 20;
However, my understanding is that even absent this, at most the file transfer may have to wait for a worker thread to be free and not get aborted. I am not talking about big files here, all files are less than 1 MB.
I have a gut feeling that this has nothing to do with node directly.
Please point to any other possibilities (node or otherwise) to handle this error. Also, any other indirect solutions? Retrying a few times could be one but that would be clumsy. EDIT: No, I cannot retry. Headers are already sent with the error!
A SIDE NOTE:
Many examples on the use of sendFile skip using the callback thereby giving the impression that it is a synchronous call. It is not. Do use the callback at all times, check for success and only then move on to the "next" middleware or take appropriate steps if the send fails for whatever reason. Not doing so can make it difficult to debug the consequences in an asynchronous environment.
See https://stackoverflow.com/a/36949631/2798152
Could it be possible that in some cases you terminate the connection by calling res.end before the asynchronous call to res.sendFile ends?
If that's not the case - can you pastebin more of your application code?
Uninstalling and Re-installing MongoDB solved this for me.
I was facing the same problem. It started happening when I had to force restart my laptop because it became unresponsive. On restarting, trying to connect to mongo server using nodejs, always threw ECONNABORTED error

Resources