I am building an applications using Node.js, Express and Mongoose. I am using Cloud9 for a development environment. I used express generator to create the base of my app and have been adding onto it.
I seemingly randomly get this error and the server stops. It doesn't happen during any specific event, it happens in between page loads. It doesn't seem to happen to any specific pages.
I get this error all the time, and I'm constantly having to start it again.
events.js:141
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at exports._errnoException (util.js:870:11)
at TCP.onread (net.js:552:26)
MongoDB does not crash when I get this error. I just have to start the node server again and it works fine. Until it randomly crashes again.
I'm not sure if this error is just a side-effect of C9, if so I'm not worried about it, I just don't want it to happen in production, especially since I can't see the logs nearly as easily.
If I really am failing to "handle the error" I'd like to know where I'm supposed to do that, I would much prefer my server to not crash when it gets a problem.
In node, whenever an 'error' event is emitted and no one listens to it, it will throw.
To make it not throw, put a listener on it and handle it yourself. That way you can log the error with more information.
You can also try using the useful node.js option --abort-on-uncaught-exception. Not only it provides much more verbose and useful error stack trace, but also saves core file on application crash allowing further debug.
To have one listener for a group of calls you can use domains and also catch other errors on runtime. Make sure each async operation related to http(Server/Client) is in different domain context comparing to the other parts of the code, the domain will automatically listen to the error events and will propagate it to it's own handler. So you only listen to that handler and get the error data. You also get more information for free here.
See also this SO answer...
You can use pm2 for starting your server. Then your server will not crash if there is any error. It will just throw the error and continue to run. It also other useful features as well.
You can read more about pm2 here https://www.npmjs.com/package/pm2
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed last month.
Improve this question
This is my first Node.js application I designed and developed.
When I run the app in staging or production it crashes when fatal errors occur.
For instance, trying to access error[0] when error is empty. This takes the entire service down and the client receives 503 errors. I am used to PHP & C# where this doesn't happen. I mean, the end-user will get an error, but the server for PHP or C# doesn't go down. With Node.js the entire Web service is no longer available.
I am working through the code to catch everything that could be a fatal error, but still, I don't have confidence in this app knowing one mistake and my clients are not able to work. To restart the services, I created a health check system that expects a 200 code.
Here is my environment:
React.js with Ant Design & Node.js (14.21.1)
This is what I run to start the services on production and staging for the web service and client respectively:
nohup node app.js > nohup.out &
nohup node node_modules/#craco/craco/scripts/start.js > nohup.out &
The web service uses Apollo GraphQL & Sequelize with a pool
{
max: 5,
min: 0,
acquire: 30000,
idle: 10000
}
Running Apache2 on AWS EC2 instances with Aurora/MySQL, but without using an RDS proxy.
PHP 8 to make health checks every 10 seconds and restarting services when anything >= 300 is returned.
Here is what I want to know:
Why does the entire web service go down when an error happens? When PHP or C# have fatal errors, it only affects the one request, not the entire site. What I am experiences is the equivalent of a PHP web service having a fatal error and the entire service crashes and goes offline.
Is this normal for Node.js services?
If this is not normal, what I am doing wrong?
How can I stop error from taking down the entire service?
When you use Node.js, your application is the server. On the other hand, when you use PHP, the server is Apache and your code is just a script being executed by Apache's mod_php module. (at least, these are the typical configurations for Node.js and PHP, though not the only ones)
So when your Node.js application has an uncaught error, it's equivalent to your HTTP server having an uncaught error. It will crash. While with PHP, Apache mod_php will catch it and handle it in a specific way.
But that doesn't mean it's acceptable for a run-of-the-mill error to cause a Node.js HTTP server to crash. If that happens, it just means your error handling needs improvement. A well-coded Node.js server will catch an error, log it, respond with an error response, and keep chugging. You have to write that code yourself though, so it's not as forgiving as PHP in that regard.
As for what to do about it, it depends on what kind of errors you're facing exactly, but the general idea is that there should be a top-level error handler that catches any error during a request, logs the error, does whatever else should be done, and returns a code 500 response. There are at least a couple of gotchas in addition:
That kind of error handler cannot catch an "uncaught promise rejection". This sort of error happens when you are not awaiting or catch()ing your promises. You should always do so, but if you want to stop your server from crashing while you diagnose where those are, you can subscribe to the unhandledRejection event. This will prevent them from crashing the process. Make sure to log them and fix them the right way.
If you are using something that inherits from EventEmitter, and it fires an error event which you have not subscribed to, this will be re-thrown as an unhandled error and crash the application. If you're using anything that fires events (not super common for an HTTP server), make sure you subscribe to its error event.
If you are using callback APIs (there's rarely a good reason to anymore) you need to be careful about throwing an error from within a callback, doing so can crash the application.
I have an API for downloading files using res.download. It works fine when single call is made.
However, when bulk call (e.g. call 10 times within 1 second), several of the request are failed and prompt below error:
Error: read ECONNRESET
Can anyone explain this behavior? Thanks in advance.
"ECONNRESET" means the other side of the TCP conversation abruptly closed its end of the connection. This is most probably due to one or more application protocol errors. You could look at the API server logs to see if it complains about something.
How do I debug error ECONNRESET in Node.js?
Your best bet is to make sure you are not throttling the API.
I had hard time debugging a problem with my NodeJS' code today.
I have problems when I open two connections to the same unix socket (reasons though); and for unknown reasons, sometimes it works fine throughout; and sometimes I don't get back any data, but connect is fired for only one of them.
I'm still trying to debug, but I deep dived into documentation and faced another question. As NodeJS Docs (12.x LTS) states: (about net.Socket)
# Event: 'connect'
Added in: v0.1.90
Emitted when a socket connection is successfully established. See net.createConnection().
# Event: 'ready'
Added in: v9.11.0
Emitted when a socket is ready to be used. Triggered immediately after 'connect'.
(https://nodejs.org/docs/latest-v12.x/api/net.html)
I wondered if that is where I should look for error:
what does immediately mean? Does it mean synchronously? If so, is there any difference between ready and connect?
is there any point for one using ready instead of connect for doing after-connection-established/opened tasks?
what is the difference between the two?
Thanks!
This event is emitted from net for consistency across different APIs. See the original commit here:
https://github.com/nodejs/node/commit/1c8149417a5dec9b2af056f306822b8a22a09706
It was created to make developers' life easier when working with fs and net code, so that they don't have to remember all the intricate details of a given stream implementation.
In practice, the Node.js socket code does this:
self.emit('connect');
self.emit('ready');
I just started trying out SailsJS a few days ago.
I've realized that the Node is terminated whenever I have an uncaught exception.
I have a list of controllers and each of them calls a specific service JS file (Containing logics and DB calls) in services/.
Can I write a global error handler for all services so that any type of error that occurs from these services should be handled by it and appropriate error response has to be communicated to front-end.
I tried using process.on('uncaughtexception') or some of basic exceptions but it needs to be added to each service method.
Also can I have one common point for all service calls made from client to server through which all io.socket.post() and io..socket.get() goes through
I would appreciate any pointer/article that would show me the common best practices for handling uncaught exceptions in SailsJS and using shorter code rather than writing redundant code in all services.
Best practice is using Domains in your controller. This will handle exceptions in async code, and its relatively straight forward.
You can use something like trycatch to simplify things a little, but domain based exceptions will be most effective. It'll insure that exceptions do not crash your application. Just create a new domain in your controller and run your controller methods inside of that domain.
Sailsjs being based on express you can use connect middleware, and you can seamlessly create a new domain from middleware. There such thing as express-domain-middleware. This might be the most aesthetic option, and most convenient.
Update:
As mention by Benjamin Gruenbaum, Domains are planned to become deprecated in v1 of node. Perhaps you should read through Joyents Error Handling Best Practices. Its agnostic to the framework you are using.
Additonally you can still use Domains, while there isn't a way to globally handle errors in node.js otherwise. Once deprecated you could always remove your dependence on Domains, relatively easily. That said, it may be best not to rely solely on domains.
Strongloop also provides a library inspired by domains called Zone. This is also an option.
Its OK to let node instance error out due to a programming error, else it may continue in an inconsistent state and mess-up business logic. In production environment the server can be restarted on crash, this will reset its state and keep it available, if the error is not frequent. And in all of it its very important to Log everything. This applies to most of Node setups, including SailsJS.
The following approach can be taken:
Use a Logger : A dedicated logger should be accessible to server components. Should be connected to a service that notifies the developer (email ?) of very serious errors.
Propagate per request errors to the end: Carefully forward errors from any step in request processing. In ExperssJs/ConnectJs/middle-ware based setup's, the next(err) can be used to pass an error to the middle-ware chain. An error catching middle-ware at the end of the chain will get this error, log it verbose, and send a 500 status back. You can use Domains or Zones or Promises or async or whatever you like to process request and catch errors.
Shutdown on process.on('uncaughtexception'): Log the erorr, do necessary clean-up, and throw the same error again to shutdown process.
User PM2/Forever or Upstart/init.d on linux : Now when the process shuts down due to the bad exception, these tools will restart it and track how many time server has been crashing. If the server is crashing way too many time, its good to stop it and take immediate action.
I have not tried this, but I believe you should be able to set a catch-all exception handler in bootstrap.js using process.on('uncaughtexception').
Personally, I use promises via the bluebird library, and put a catch statement that passes all errors to a global error handling function.
Inside a nodejs connect middleware, the default way of error reporting is to call next(err), usually followed by return, if no further message should be shown to the user. The error handler may show an vanilla http 500 page then for example.
However, some errors may result in exceptions, including those throwed by used third party libs. The connect (or express?) middleware stack however catches those, and redirect them to the error handler as well.
I followed some discussions saying nodejs should be restarted on exceptions, as some state may be corrupted. However, the connect (or express) makers doesn't share this view it seems?
Having it this way, is it feasable to just throw exceptions inside middleware? Or may this baypass some connect-internal operation?