We're running a Node.js/Express application which runs for a few hours and will then start to throw 504 errors for no good reason. Since we're currently unable to track these errors down we need to restart the application every hour or so to ensure it's still running during the weekend.
Our Ubuntu server runs Dokku, which then has a container setup for our application. Every time the application spits a 504 we have to run docker restart appid as root.
So what's the best way of automatically restarting the node process every hour?
throw 504 errors for no good reason
It's throwing these because you application is crashing
currently unable to track these errors down
You have to track them down. They are very likely unhandled exceptions which you can catch and log via:
process.on('uncaughtException', function(error) {
//look Ma, I died
});
So what's the best way of automatically restarting the node process every hour?
Since I'd feel bad not at least attempting to address you actual question, even though you are most certainly fixing the symptom instead of the problem in a seriously bad way...
Use cron. Put a script in /etc/cron.hourly/restart_express to do it. Make sure the script file has execute permissions and conforms to the run-parts naming constraints (no dots, etc).
Related
I have a nodejs application running on Linux, as we all know, whenever I restart the nodejs app it will get a new PID, suppose while the nodejs app is running, a client connects to it and running some process and the process status is processing, during that point of time, if the nodejs app restarts(on the server-side), how can we make sure the client connects back to the previous processing state.
What is happening now is, whenever the server restarts, the process stucks in processing forever.
Just direct me to a sample of how this scenario is handled in real life.
Thank You.
If I'm understanding you correctly, then the answer is you can't...
The reason for this is that, when you restart the process the event loop is restarted, meaning any processes that were running or were waiting in the event loop are gone. You are essentially clearing out the event loop when you restart.
I would say though, if you know the process is 'crashing' node then you probably want to look into that process and see why is crashing, place it in a try catch to it wont kill the server.
now with that said ( and without knowing what, processing state really means ) you could set a flag in your DB server for say 'job1' and have a status column of say 'running' when it was kicked off. When the node server restarts it can read Job status for 'running' jobs, if the 'job' is in a 'running' state you can fire off the job again and once complete update the table to 'completed'
This probably not the most efficient way as it's much better to figure out why the process if crashing, but as a fall-back this could work although in a clustered environment this could cause issues because server 1 may fail while server 2 is processing because server 1 does not know what server two is doing. With more details as to the use case, environment etc would probably allow for a better answer
I have a web application running as a service on an Ubuntu EC2 Instance. As of the past 24 hours, the application has been crashing randomly 2-4 hours after running with the message attached in the image below. The error is:
[nodemon] app crashed - waiting for file changes before starting...
I have run into this error before but usually, it is a syntax error and it will not allow me to actually start the application. In this case, the app functions normally for several hours before crashing. I have no idea where to even start as there's nothing above it that looks like it could be causing the crash. The only thing is it looks like the website receives 3 Get / Requests before the server can respond then it crashes. Most of the posts I've found online about this also block the application from running and don't mention the fact that the app runs normally then crashes.
Any help would be greatly appreciated.
Thanks!
Error Log from Journalctl
It looks like a silent error. I would try to log every input (e.g. http request and timeouts) with timestamp and also log the crash with time. When a crash occurs I would compare the time to events happening right before.
Also check your /var/log/ if the programm was terminated by the system or another programm.
Different users trying to access different application routes with heavy manipulation of data. At the mid time one of the request failed due to internal server error and my whole application has been crashed. Thats why other request has been failed because build has been crashed. Is there any solution to handle this situation?
If your program has thrown an uncaught error and crashed then there's nothing you can really do other than start it back up again. You could use something like pm2 to automatically restart your node process when it crashes, and then at least future requests that come in should work (although you will lose any in memory data from before the last crash).
Another thing that I think would help you would be to move your backend onto a serverless architecture where each invocation of your code is independent of the others.
And of course try to fix the code so that it handles things gracefully and doesn't actually throw errors. :)
I am developing a web application using MEAN stack. When an error comes in the codes, the whole application crashes and I have to restart the server (re-run the application).
This behavior may not be a problem during development. But when the application goes to production, and if some unexpected errors occurs, the whole application will crash. I have to keep an eye on the system constantly and restart it when an error occurs? What am I missing here?
This is one solution that I have seen being used in production to ensure that a node program is always running (even after server restarts).
Use Forever (https://www.npmjs.com/package/forever). You can run it through code or through command line.
$ [sudo] npm install forever -g
forever start app.js
Where app.js has the code for instantiating the web server (in the MEAN stack it's the express initialization).
If an unhandled error bubbles to the top of the stack without being caught, crashing is the intended behavior. An unhandled exception means that your app is in an undefined state.
Think of it this way. If you lose control of your car and drive off the road, the best thing to do is to slam on the brakes and stop (AKA a controlled program crash or halt) rather than continue blindly blundering through foliage, flower beds, backyards, swimming pools, toddlers, and whatever other obstacles may be in the way.
I'd recommend using a tool like forever to run your app in production, which will monitor and restart your app when it crashes. This sort of thing is standard practice. Obviously you don't want it to crash, and you should handle errors in context where you know how to recover from them. And some frameworks do a better job than others of handling errors smoothly without crashing. Restarting the process is mainly best for things that catch you completely off guard. Checkout this for more error handling tips:
https://www.joyent.com/developers/node/design/errors
The issue mentioned by you is the point you need to keep in your mind while you develop applications. Because the way of handling errors is the thing that you can't skip. There are a few useful solutions (just a small part of the whole 'Error Handling' World) you can use in order to save your applications.
But lets start from your issue. As you have already such situation, I can recommend you integrating node domain module into your application, so it will not crash in case of such exceptions. Please refer to the link below:
https://nodejs.org/api/domain.html
You may wrap your server creation and catch all the unhandled exceptions.
Example:
var domain = require('domain').create();
domain.on('error', function(err){
//track error into your log file
//do something else with error (send message to admin, send error response etc. )
});
domain.run(function(){
//run http server here
});
Here you may find good example as well:
https://engineering.gosquared.com/error-handling-using-domains-node-js
As for error handling solutions I can recommend you keeping the following rules:
use domain to catch exceptions
use node eventemitter to event and catch exceptions
always think about possible situations when you handle results of functions
develop single strategy of error handling (throw error/return error/send error to user etc.)
use try catch block to safe code blocks from unhandled exceptions
There are much more solutions you can find.
Please see the links I recommend you to check:
https://www.joyent.com/developers/node/design/errors
http://derickbailey.com/2014/09/06/proper-error-handling-in-expressjs-route-handlers/
http://expressjs.com/guide/error-handling.html
http://shapeshed.com/uncaught-exceptions-in-node/
So, I need to run my node.js app on heroku, it works very well, but when my app crashes, i need something to restart it, so i added forever to package.json, and created a file named forever.js with this:
var forever = require('forever');
var child = new (forever.Monitor)('web.js', {
max: 3,
silent: false,
options: []
});
//child.on('exit', this.callback);
child.start();
forever.startServer(child);
on my Procfile (that heroku uses to know what to start) i put:
web: node forever.js
alright! Now everytime my app crashes it auto restarts, but, from time to time (almost every 1 hour), heroku starts throwing H99 - Platform error, and about this error, they say:
Unlike all of the other errors which will require action from you to correct, this one does not require action from you. Try again in a minute, or check the status site.
But I just manually restart my app and the error goes away, if I don't do that, it may take hours to go away by itself.
Can anyone help me here? Maybe this is a forever problem? A heroku issue?
This is an issue with free Heroku accounts: Heroku automatically kills unpaid apps after 1 hour of inactivity, and then spins them back up the next time a request comes in. (As mentioned below, this does not apply to paid accounts. If you scale up to two servers and pay for the second one, you get two always-on servers.) - https://devcenter.heroku.com/articles/dynos#dyno-sleeping
This behavior is probably not playing nicely with forever. To confirm this, run heroku logs and look for the lines "Idling" and " Stopping process with SIGTERM" and then see what comes next.
Instead of using forever, you might want to try the using the Cluster API and automatically create a new child each time one dies. http://nodejs.org/api/cluster.html#cluster_cluster is a good example, you'd just put your code into the else block.
The upshot is that your app is now much more stable, plus it gets to use all of the available CPU cores (4 in my experience).
The downside is that you cannot store any state in memory. If you need to store sessions or something along those lines, try out the free Redis To Go addon (heroku addons:add redistogo).
Here's an example that's currently running on heroku using cluster and Redis To Go: https://github.com/nfriedly/node-unblocker
UPDATE: Heroku has recently made some major changes to how free apps work, and the big one is they can only be online for a maximum of 18 hours per day, making it effectively unusable as a "real" web server. Details at https://blog.heroku.com/archives/2015/5/7/heroku-free-dynos
UPDATE 2: They changed it again. Now, if you verify your ID, you can run 1 free dyno constantly: https://blog.heroku.com/announcing_heroku_free_ssl_beta_and_flexible_dyno_hours#flexible-free-dyno-hours