I have a fairly simple NodeJs server script, that handles incoming data and persists it to a database. The script runs indefinitely. Mostly this works great, but sometimes an error arrises. Is there a way to listen for an exception and if the script was terminated to rerun it again?
(I know the issues for causing the exception must be fixed, and I've done that, but I want to be sure the script always runs as I otherwise loose valuable data; the use case lies in processing IoT sensor data)
It seems like you have an uncaught exception that causes your application to crash.
First of all, you should use try...catch statements to catch exceptions where they can appear within your range of responsibilities. If you don't care about proper error handling and just want to make sure the application keeps running, it is as simple as the following. This way, errors won't bubble up becoming uncaught exceptions to crash your application:
try {
// logic prone to errors
} catch (error) {}
If your application somehow can come into an erroneous state which is unpredictable and hard to resolve during runtime, you can use solutions like process managers, PM2 for instance, to automatically restart your application in case of errors.
Related
Different users trying to access different application routes with heavy manipulation of data. At the mid time one of the request failed due to internal server error and my whole application has been crashed. Thats why other request has been failed because build has been crashed. Is there any solution to handle this situation?
If your program has thrown an uncaught error and crashed then there's nothing you can really do other than start it back up again. You could use something like pm2 to automatically restart your node process when it crashes, and then at least future requests that come in should work (although you will lose any in memory data from before the last crash).
Another thing that I think would help you would be to move your backend onto a serverless architecture where each invocation of your code is independent of the others.
And of course try to fix the code so that it handles things gracefully and doesn't actually throw errors. :)
I use forever to keep a nodejs instance up.
When there is an error, the server stops working and the log shows a javascript exception.
(for example, a get request returns nothing when the server code expected it to).
Is there a way for node to NEVER stop running? To simply "look past" the error?
You should have a look here: Node.js Best Practice Exception Handling
that's the way to handle exceptions:
process.on('uncaughtException', function(err) {
console.log(err);
})
The only safe way to prevent Node from ever stopping is to write perfect, bug-free code.
But once you've resigned yourself to the occasional uncaught exception, the safe and recommended approach is to crash and restart with an external monitor.
Unhandled exceptions inherently mean that an application is in an undefined state. Attempting to resume application code without properly recovering from the exception can cause additional unforeseen and unpredictable issues.
-- https://nodejs.org/api/process.html#process_event_uncaughtexception
I am developing a web application using MEAN stack. When an error comes in the codes, the whole application crashes and I have to restart the server (re-run the application).
This behavior may not be a problem during development. But when the application goes to production, and if some unexpected errors occurs, the whole application will crash. I have to keep an eye on the system constantly and restart it when an error occurs? What am I missing here?
This is one solution that I have seen being used in production to ensure that a node program is always running (even after server restarts).
Use Forever (https://www.npmjs.com/package/forever). You can run it through code or through command line.
$ [sudo] npm install forever -g
forever start app.js
Where app.js has the code for instantiating the web server (in the MEAN stack it's the express initialization).
If an unhandled error bubbles to the top of the stack without being caught, crashing is the intended behavior. An unhandled exception means that your app is in an undefined state.
Think of it this way. If you lose control of your car and drive off the road, the best thing to do is to slam on the brakes and stop (AKA a controlled program crash or halt) rather than continue blindly blundering through foliage, flower beds, backyards, swimming pools, toddlers, and whatever other obstacles may be in the way.
I'd recommend using a tool like forever to run your app in production, which will monitor and restart your app when it crashes. This sort of thing is standard practice. Obviously you don't want it to crash, and you should handle errors in context where you know how to recover from them. And some frameworks do a better job than others of handling errors smoothly without crashing. Restarting the process is mainly best for things that catch you completely off guard. Checkout this for more error handling tips:
https://www.joyent.com/developers/node/design/errors
The issue mentioned by you is the point you need to keep in your mind while you develop applications. Because the way of handling errors is the thing that you can't skip. There are a few useful solutions (just a small part of the whole 'Error Handling' World) you can use in order to save your applications.
But lets start from your issue. As you have already such situation, I can recommend you integrating node domain module into your application, so it will not crash in case of such exceptions. Please refer to the link below:
https://nodejs.org/api/domain.html
You may wrap your server creation and catch all the unhandled exceptions.
Example:
var domain = require('domain').create();
domain.on('error', function(err){
//track error into your log file
//do something else with error (send message to admin, send error response etc. )
});
domain.run(function(){
//run http server here
});
Here you may find good example as well:
https://engineering.gosquared.com/error-handling-using-domains-node-js
As for error handling solutions I can recommend you keeping the following rules:
use domain to catch exceptions
use node eventemitter to event and catch exceptions
always think about possible situations when you handle results of functions
develop single strategy of error handling (throw error/return error/send error to user etc.)
use try catch block to safe code blocks from unhandled exceptions
There are much more solutions you can find.
Please see the links I recommend you to check:
https://www.joyent.com/developers/node/design/errors
http://derickbailey.com/2014/09/06/proper-error-handling-in-expressjs-route-handlers/
http://expressjs.com/guide/error-handling.html
http://shapeshed.com/uncaught-exceptions-in-node/
Nodejs's default behavior is to shut down when an error makes it to the main event loop. The manual strongly recommends not overriding this behavior (for instance, via process.on('uncaughtException).
The explanation given is:
An unhandled exception means your application - and by extension
node.js itself - is in an undefined state. Blindly resuming means
anything could happen.
Think of resuming as pulling the power cord when you are upgrading
your system. Nine out of ten times nothing happens - but the 10th
time, your system is bust.
Can someone elaborate on this? Chrome, which uses the same V8 engine as node, resumes its event loop after an uncaught error by default, and AFAIK this doesn't cause any problems. So it doesn't seem like there's any intrinsic reason that V8 can't recover gracefully from an uncaught exception. Is there something in the node internals that behaves differently than Chrome?
The answer does not have anything to do with the engine's ability to restart itself.
It has to do with your own application code. If an unhandled exception occurs, then there is inherently no way of understanding your application's state. If there were, then it would not have been an unhandled exception. And, if you do not know your state, then you cannot be sure that more unhandled exceptions will not continue to occur, most likely causing worse-and-worse issues as time progresses (as unexpected states cascade into more-and-more unexpected states).
Imagine this as code that is running on the server (as it is not at all specific to node.js):
start process
open two server sockets
process incoming requests
If you were to fail to open the second server socket without handling the exception, then chances are your application will not work. Restarting the thread at the next logical step would likely fail to work properly as well. Restarting the engine could not reasonably close the one socket, and it would be unlikely to fix the cause of the second failure (most likely the port is already in use), and if it did close the successfully opened socket, then it had better restart the application so that it can be reopened (or else it made it worse).
That is perhaps an obvious case, but now imagine that you are a graphics application (e.g., a game):
start process
load models
handle state (until closing)
draw screen
If any model failed to load without exception handling, then the process cannot reasonably continue because it will simply cause more errors while drawing.
There are cases where recovering from unhandled exceptions is reasonable. In most client side GUI frameworks there is a way to register for unhandled exceptions, which allows the restarting of the event thread (GUI thread), analogous to Chrome's V8 recovery. It is dangerous because recovery is not guaranteed; whatever caused the unhandled exception could still be in memory and ready to cause the exception again on the next usage of it. However, it's also possible that a well developed application can be small enough to wipe itself clean given such exceptions. The best use of such handlers (handling of unhandled exceptions) is to log the exception so that the issue can be fixed.
To put it differently: imagine an exception occurs that you did not handle in your application anywhere. What can you do to fix it so that it does not happen on the very next pass of the code? To safely answer that implies that you know what caused it, which means that A) it should not be unhandled and B) it is isolated.
The only guaranteed safe reset is to start from the very beginning, which means to restart the application.
In the official NodeJS documentation there is code example where process tries to exit gracefully when there was exception in domain (it closes connections, waits for some time for other requests and then exits).
But why just not send the 500 error and continue to work?
In my application I want to throw some expected Errors (like FrontEndUserError) when user input is not valid, and catch these exceptions somewhere in middleware to send pretty error message to client. With domains it very easy to implement, but are there any pitfalls around this?
app.use (err, req, res, next) ->
if err instanceof FrontEndUserError
res.send {error: true, message: err.message}
else
log err.trace
res.send 500
From domain module official documentation:
By the very nature of how throw works in JavaScript, there is almost never any way to safely "pick up where you left off", without leaking references, or creating some other sort of undefined brittle state.
The safest way to respond to a thrown error is to shut down the process ...
To me that means when something thrown an error in your NodeJS application, then you're done unfortunately. If you do care about how your application works and the result is important to you, then your best bet is to kill the process and start it again. However, in that last milliseconds, you can be more nice to other clients, let them finish their work, say sorry to new clients, log couple of things if you want and then kill the process and start it again.
That's what exactly happening in the NodeJS domain module's documentation example.
Let's look at your web application/server as a state machine.
Unless your application is very small, it is very unlikely that you happen to know every state that your machine can possibly be in.
When you get an error, you have two choices:
1) Examine the error and decide what to do, or
2) ignore it.
In the first case, you gracefully change from one state to another. In the second case, you don't have any clue what state your machine is in, since you didn't bother seeing what the error was. Essentially, your machine's state is now 'undefined'.
It is for this reason, NodeJS recommends killing the process if an error propagates all the way to the event loop. Then again, this level of absolution may be overkill for pet projects and small apps, so your solution is quite fine too.
But imagine if you were writing a banking software; someday you get an error you've never seen before, you app simple ignores it and sends a 500; but each time someone is losing a 100k$. Here, I would want to make sure no error ever reaches the event loop, and if it does, kill the process with a detailed stack trace for later analysis.