Inside a nodejs connect middleware, the default way of error reporting is to call next(err), usually followed by return, if no further message should be shown to the user. The error handler may show an vanilla http 500 page then for example.
However, some errors may result in exceptions, including those throwed by used third party libs. The connect (or express?) middleware stack however catches those, and redirect them to the error handler as well.
I followed some discussions saying nodejs should be restarted on exceptions, as some state may be corrupted. However, the connect (or express) makers doesn't share this view it seems?
Having it this way, is it feasable to just throw exceptions inside middleware? Or may this baypass some connect-internal operation?
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed last month.
Improve this question
This is my first Node.js application I designed and developed.
When I run the app in staging or production it crashes when fatal errors occur.
For instance, trying to access error[0] when error is empty. This takes the entire service down and the client receives 503 errors. I am used to PHP & C# where this doesn't happen. I mean, the end-user will get an error, but the server for PHP or C# doesn't go down. With Node.js the entire Web service is no longer available.
I am working through the code to catch everything that could be a fatal error, but still, I don't have confidence in this app knowing one mistake and my clients are not able to work. To restart the services, I created a health check system that expects a 200 code.
Here is my environment:
React.js with Ant Design & Node.js (14.21.1)
This is what I run to start the services on production and staging for the web service and client respectively:
nohup node app.js > nohup.out &
nohup node node_modules/#craco/craco/scripts/start.js > nohup.out &
The web service uses Apollo GraphQL & Sequelize with a pool
{
max: 5,
min: 0,
acquire: 30000,
idle: 10000
}
Running Apache2 on AWS EC2 instances with Aurora/MySQL, but without using an RDS proxy.
PHP 8 to make health checks every 10 seconds and restarting services when anything >= 300 is returned.
Here is what I want to know:
Why does the entire web service go down when an error happens? When PHP or C# have fatal errors, it only affects the one request, not the entire site. What I am experiences is the equivalent of a PHP web service having a fatal error and the entire service crashes and goes offline.
Is this normal for Node.js services?
If this is not normal, what I am doing wrong?
How can I stop error from taking down the entire service?
When you use Node.js, your application is the server. On the other hand, when you use PHP, the server is Apache and your code is just a script being executed by Apache's mod_php module. (at least, these are the typical configurations for Node.js and PHP, though not the only ones)
So when your Node.js application has an uncaught error, it's equivalent to your HTTP server having an uncaught error. It will crash. While with PHP, Apache mod_php will catch it and handle it in a specific way.
But that doesn't mean it's acceptable for a run-of-the-mill error to cause a Node.js HTTP server to crash. If that happens, it just means your error handling needs improvement. A well-coded Node.js server will catch an error, log it, respond with an error response, and keep chugging. You have to write that code yourself though, so it's not as forgiving as PHP in that regard.
As for what to do about it, it depends on what kind of errors you're facing exactly, but the general idea is that there should be a top-level error handler that catches any error during a request, logs the error, does whatever else should be done, and returns a code 500 response. There are at least a couple of gotchas in addition:
That kind of error handler cannot catch an "uncaught promise rejection". This sort of error happens when you are not awaiting or catch()ing your promises. You should always do so, but if you want to stop your server from crashing while you diagnose where those are, you can subscribe to the unhandledRejection event. This will prevent them from crashing the process. Make sure to log them and fix them the right way.
If you are using something that inherits from EventEmitter, and it fires an error event which you have not subscribed to, this will be re-thrown as an unhandled error and crash the application. If you're using anything that fires events (not super common for an HTTP server), make sure you subscribe to its error event.
If you are using callback APIs (there's rarely a good reason to anymore) you need to be careful about throwing an error from within a callback, doing so can crash the application.
Ive built an app that pipes requests through express validator's validation chains, but I ran into a design issue related to logging and error handling.
My app is divided into distinct controllers/services/repositories. All errors thrown in repos and layers bubble up to controllers which handle errors by calling next() on them and passing them to an error handling middleware that logs and so on.
The problem is that one of my validation chains calls a repo. If my DB connection is dead, all I get on my log is what I happen to put in the validation chain's withMessage().
Id like to get a better log of the event in such case, but now a dead db connection ends in a 400 Bad request on the logs since there isnt a controller to catch the error.
Any ideas how I should structure my app to combat this?
I would not like to add specific logging on the service/data layers only because I use express validator as a middleware.
I looked more into the responsibilities of each layer and came to the conclusion that the call to the data layer should not be done with express-validator. Logic related to whether an email is already registered, for example, should happen on the service layer, as per:
In which layer should validation be performed?
I will use express validator only for easy checks and put the rest into my service layer.
I am building an applications using Node.js, Express and Mongoose. I am using Cloud9 for a development environment. I used express generator to create the base of my app and have been adding onto it.
I seemingly randomly get this error and the server stops. It doesn't happen during any specific event, it happens in between page loads. It doesn't seem to happen to any specific pages.
I get this error all the time, and I'm constantly having to start it again.
events.js:141
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at exports._errnoException (util.js:870:11)
at TCP.onread (net.js:552:26)
MongoDB does not crash when I get this error. I just have to start the node server again and it works fine. Until it randomly crashes again.
I'm not sure if this error is just a side-effect of C9, if so I'm not worried about it, I just don't want it to happen in production, especially since I can't see the logs nearly as easily.
If I really am failing to "handle the error" I'd like to know where I'm supposed to do that, I would much prefer my server to not crash when it gets a problem.
In node, whenever an 'error' event is emitted and no one listens to it, it will throw.
To make it not throw, put a listener on it and handle it yourself. That way you can log the error with more information.
You can also try using the useful node.js option --abort-on-uncaught-exception. Not only it provides much more verbose and useful error stack trace, but also saves core file on application crash allowing further debug.
To have one listener for a group of calls you can use domains and also catch other errors on runtime. Make sure each async operation related to http(Server/Client) is in different domain context comparing to the other parts of the code, the domain will automatically listen to the error events and will propagate it to it's own handler. So you only listen to that handler and get the error data. You also get more information for free here.
See also this SO answer...
You can use pm2 for starting your server. Then your server will not crash if there is any error. It will just throw the error and continue to run. It also other useful features as well.
You can read more about pm2 here https://www.npmjs.com/package/pm2
I am working on logs of the application just to give you information there are two kind of strategies used inside the application.
POST Process
PRE Process
In the first strategy if application has to make some call to third-party applications, it will first send the response to client and after execute that call, so
as my application is using express as framework, I can catch those responses in middle-wares but not in this specific case application is not sending back
any response as application has already responded to the client.
The second strategy is simple process and at the end send back response this call will be caught by express middle-ware without any issue, so that is the
model that i am using until now now what I want to do is to catch requests received, and sent by application by standing outside the application, as i
got the idea and understand the structure this is kind of a proxy server which will catch requests not just received by the application but sent by as well, and i know we can catch requests coming in but I am working on the data at runtime
so i don't want those logs i want logs that are coming into the logger application at runtime.
Now coming to the question according to my requirements is there a way to catch requests received, and sent by NodeJS server?
can you try morgan node module for that
Create a new morgan logger middleware function using the given format and options. The format argument may be a string of a predefined name (see below for the names), a string of a format string, or a function that will produce a log entry.
https://www.npmjs.com/package/morgan
I just started trying out SailsJS a few days ago.
I've realized that the Node is terminated whenever I have an uncaught exception.
I have a list of controllers and each of them calls a specific service JS file (Containing logics and DB calls) in services/.
Can I write a global error handler for all services so that any type of error that occurs from these services should be handled by it and appropriate error response has to be communicated to front-end.
I tried using process.on('uncaughtexception') or some of basic exceptions but it needs to be added to each service method.
Also can I have one common point for all service calls made from client to server through which all io.socket.post() and io..socket.get() goes through
I would appreciate any pointer/article that would show me the common best practices for handling uncaught exceptions in SailsJS and using shorter code rather than writing redundant code in all services.
Best practice is using Domains in your controller. This will handle exceptions in async code, and its relatively straight forward.
You can use something like trycatch to simplify things a little, but domain based exceptions will be most effective. It'll insure that exceptions do not crash your application. Just create a new domain in your controller and run your controller methods inside of that domain.
Sailsjs being based on express you can use connect middleware, and you can seamlessly create a new domain from middleware. There such thing as express-domain-middleware. This might be the most aesthetic option, and most convenient.
Update:
As mention by Benjamin Gruenbaum, Domains are planned to become deprecated in v1 of node. Perhaps you should read through Joyents Error Handling Best Practices. Its agnostic to the framework you are using.
Additonally you can still use Domains, while there isn't a way to globally handle errors in node.js otherwise. Once deprecated you could always remove your dependence on Domains, relatively easily. That said, it may be best not to rely solely on domains.
Strongloop also provides a library inspired by domains called Zone. This is also an option.
Its OK to let node instance error out due to a programming error, else it may continue in an inconsistent state and mess-up business logic. In production environment the server can be restarted on crash, this will reset its state and keep it available, if the error is not frequent. And in all of it its very important to Log everything. This applies to most of Node setups, including SailsJS.
The following approach can be taken:
Use a Logger : A dedicated logger should be accessible to server components. Should be connected to a service that notifies the developer (email ?) of very serious errors.
Propagate per request errors to the end: Carefully forward errors from any step in request processing. In ExperssJs/ConnectJs/middle-ware based setup's, the next(err) can be used to pass an error to the middle-ware chain. An error catching middle-ware at the end of the chain will get this error, log it verbose, and send a 500 status back. You can use Domains or Zones or Promises or async or whatever you like to process request and catch errors.
Shutdown on process.on('uncaughtexception'): Log the erorr, do necessary clean-up, and throw the same error again to shutdown process.
User PM2/Forever or Upstart/init.d on linux : Now when the process shuts down due to the bad exception, these tools will restart it and track how many time server has been crashing. If the server is crashing way too many time, its good to stop it and take immediate action.
I have not tried this, but I believe you should be able to set a catch-all exception handler in bootstrap.js using process.on('uncaughtexception').
Personally, I use promises via the bluebird library, and put a catch statement that passes all errors to a global error handling function.