I'm using pm2 to manage process in my nodejs express application (running in cluster mode).
We had 2 kind of error handler
FIRST: 'uncaughtException' will be handled with
process.on('uncaughtException', function(err){});
Actually, I do not declare an handler like this cause of letting pm2 detect died worker in this case so restart the died worker automatically.
SECOND: express error handler, I mean the error will be forwarded to express error handler, not uncaughtException handler, the error handler like below
app.use(function(err, req, res, next) {})
I also do not declare this error handler for same purpose as uncaughtException.
But pm2 does not restart node in this case.
Any idea about this problem?
Many thanks
When catching errors with express error handler or even "uncaughtException" event, the process is still running, so pm2 won`t restart it.
If you want pm2 to restart after each exception, I would suggest something like this:
process.on('uncaughtException', function(e) {
console.log('An error has occured. error is: %s and stack trace is: %s', e, e.stack);
console.log("Process will restart now.");
process.exit(1);
})
Same goes for the express error handler. When we execute process.exit method, the process will terminate and pm2 will restart it.
Related
I am deploying a server on aws ec2. I use pm2 and nginx. However, when the app is crash, the web app return 502 bad gate way. The server can not restart automatically.
I reviewed other articles which suggest below code but it does not help.
process.on('uncaughtException', function(e) {
console.log('An error has occured. error is: %s and stack trace is: %s', e, e.stack);
console.log("Process will restart now.");
process.exit(1);
});
I'm trying to use pm2 to run my node app as a service.
Right now, starting and stopping the app works. However, I want to do a graceful shutdown.
My app already listens for SIGINT, shutdowns the server and then exits the process. However, trying to put pm2 to send the SIGINT, just causes the app to restart, like if pm2 was killing and starting it again.
This is how I create the process:
pm2 start server.js --name ProcessName --silent --kill-timeout 3000
Here's my app's code for listening the SIGINT:
process.on("SIGINT", function () {
//graceful shutdown
server.end().then(() => {
process.exit();
}).catch((err) => {
console.error(err);
});
});
Then to shutdown the app using pm2, I'm running:
pm2 sendSignal SIGINT ProcessName
Which, again, restarts the app.
Reading over pm2 docs, I found that pm2 will also send a shutdown event to the app, so I added:
process.on('message', function(msg) {
if (msg == 'shutdown') {
server.end().then(() => {
process.exit();
}).catch((err) => {
console.error(err);
});
}
});
Which isn't working either.
Any idea how to solve this?
Thanks!
If you haven't solved it yet...
Based on the information you provided, I assume you are running it on Windows.
Your app cannot catch SIGINT sent by PM2 on Windows.
shutdown message works on windows too, but it is sent only by gracefulReload command.
(update)
These are not complete solutions, but might be helpful (hopefully...)
sendSignal command calls process.kill() eventually, and some of these signals might work (haven't tried).
I also found the below method. This can gracefully shutdown a process without restarting only if autorestart option is turned off.
And then your clusters will not reload in case of an accident, so it might not be what you want though...
pm2 lets you send a custom message (reference).
Put the code below in a new file:
var pm2 = require('pm2');
var id = process.argv[2];
pm2.connect(() => {
pm2.sendDataToProcessId({
type: 'shutdown',
data:{some: 'data'},
id: id,
topic: 'some topic'
}, (err, res) => {
console.log('message sent');
pm2.disconnect();
if(err) throw err;
})
});
Modify the part that listens the shutdown message like below:
process.on('message', function(msg){
if(msg == 'shutdown' || msg.type == 'shutdown'){
// code to clean up
}
});
And run the first file with node with id of the cluster you want to shutdown as an argument.
The reason for extra msg.type == 'shutdown' in the condition is that pm2.sendDataToProcessId() requires the argument to be an object with those keys, and does not accept simple shutdown string.
In general pm2 stop is the right way to stop your application. However if you run appliation inside of the Docker you need to use pm2-runtime instead of pm2 which is a part of pm2 npm package and passes system SIGINT to all child processes. See http://pm2.keymetrics.io/docs/usage/docker-pm2-nodejs
Catching the sigint and exiting gracefully should work in your first example.
To actually stop the server, use pm2 stop instead of pm2 sendSignal SIGINT ProcessName.
See http://pm2.keymetrics.io/docs/usage/signals-clean-restart/
I've been browsing around but to no success. I've found some npm packages like nodemon and forever but documentation doesn't explain how to call a restart inside the script properly.
I've found this code snippet on another question but I'm not using Express or other frameworks since the script is using a pulling service not a response one.
This is code I've made so far using internal Node.js dependencies but no luck.
'use strict'
process.on('uncaughtException', (error) => {
console.error('Global uncaughtException error caught')
console.error(error.message)
console.log('Killing server with restart...')
process.exit(0)
})
process.on('exit', () => {
console.log('on exit detected')
const exec = require('child_process').exec
var command = 'node app.js'
exec(command, (error, stdout, stderr) => {
console.log(`error: ${error.message}`)
console.log(`stdout: ${stdout}`)
console.log(`stderr: ${stderr}`)
})
})
setTimeout(() => {
errorTriggerTimeBomb() // Dummy error for testing triggering uncaughtException
}, 3000)
Just to note I'm running the server on Termux, a Linux terminal app for android. I know it's better to run from desktop but I'm always at a WiFi or mobile data area and that I don't like leaving my PC on overnight.
A typical restart using something like nodemon or forever would be triggered by calling process.exit() in your script.
This would exit the current script and then the monitoring agent would see that it exited and would restart it for you. It's the same principal as if it crashed on its own, but if you're trying to orchestrate it shutting down, you just exit the process and then the monitoring agent will restart it.
I have a home automation server that is being monitored using forever. If it crashes forever will automatically restart it. I also have it set so that at 4am every morning, it will call process.exit() and then forever will automatically restart it. I do this to prevent any memory leak accumulation over a long period of time and 30 seconds of down time in the middle of the night for my application is no big deal.
I'm somewhat new to nodejs.
Using express / npm.
Once i hit '/' from my browser, I get this in output:
[TypeError: Object #<Object> has no method 'push']
No line numbers, no traceback, nothing.
I need to be able to fix this error, and to do that I need to know what file / line # this occurred.
How do I get nodejs to output a traceback?
First, add an express error handler, which is middleware expecting 4 arguments:
app.use(function (error, req, res, next) {
console.error(error.stack);
});
Make sure to put the above code at the very end of your middleware stack after all your normal middleware and route handlers are defined.
If that doesn't get you your stack trace, catch process-level uncaught exceptions:
process.on("uncaughtException", function (error) {
console.error(error.stack);
});
And one final tip would be run your app under the debugger with node --debug=3001 app.js, connect to it with node-inspector, open the debugger in chrome, enable "stop on exceptions", then trigger the error. The debugger will take you to the line where the exception occurs.
Running the following server.js:
cluster(app)
.use(cluster.logger(path_to_logs))
.use(cluster.stats())
.use(cluster.pidfiles(path_to_pids))
.use(cluster.cli())
.use(cluster.repl(8888))
.listen(3000);
it works as expected. However, let's throw in an unhandled exception like so:
setTimeout(function () {
throw new Error('User generated fault.');
},5000);
Running the server with `$ node server.js, it starts and the exception is thrown after five seconds. Consequently the server is quit in what is seemingly the same as pressing ctrl+c.
However not quite. Because now trying to restart the server using $ node server.js I receive the following error:
Express server listening on port 3000
node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: EADDRINUSE, Address already in use
...
And running $ ps aux | grep node I can see that I still have two node processes running. Killing them allows me to start the server again. But since it was a manual kill, if I start the server again the same procedure starts over. 5 seconds pass, throw error, unable to restart.
This is a problem because with forever, it causes an infinite death cycle upon the first unhandled exception.
So my questions are:
Do you have any further ideas on why this might occur?
How can I listen for all exceptions and react by kill the process(es)
Is the above a bad approach?
Sorry for posting this on ServerFault aswell, but I realized this IS actually a code question.
About handling unhanded exceptions, you can use: http://nodejs.org/docs/v0.4.12/api/process.html#event_uncaughtException_ .
About the process not quitting when an error is thrown: I do not know enough about cluster, but I believe to "scale" it creates child processes, and manages them, and does not die when a child dies. Taking a basic look at the source code it seems to be throwing a series of events, try seeing what events it is throwing and gather more information.