pm2 is killing detached child processes on watch restart (ie which have been spawned with detached:true, stdio:'ignore', and child.unref().
Is there a way of telling pm2 not to kill the tree of child processes on restart?
The answer was to put the following in the ecosystem file (main section for app, not under the watch settings):
"treekill": false
I ended up using
pm2 restart myapp --no-treekill
and
pm2 stop myapp --no-treekill
I restart my nodejs app from within using the pm2 API. Here is the function I use so that detached child processes are not killed (notice the treekill: false option).
function restartPm2() {
pm2.connect(function(err) {
if (err) {
console.error("PM2 FAILED TO CONNECT:", err);
} else {
pm2.restart('myapp.js', { treekill: false }, function() {});
}
});
pm2.disconnect();
}
Related
Say that I am running a process from the command line (NOT a nodejs app):
myProcess doSomething --withParam1
I am also running a nodejs app that connects to this process (rpc).
node myApp
myProcess will randomly silently fail to work properly without crashing and myApp will detect it. I need myApp to be able to restart myProcess (kill it and start it again). After I restarted myProcess, myApp will also restart itself using pm2 (I am already handling the pm2 restart part for the nodejs app - my issue is that I cannot use pm2 to restart myProcess since it is not a nodejs app). Also, I cannot change the code of myProcess since it is a 3rd party software.
How can I restart the external process from my nodejs app?
I ended up using process.kill to kill the process and nodejs child process to restart it.
To find the pid of the process before killing it, I used this:
const childProcess = require('child_process');
function printPid() {
childProcess.exec('pidof -s myProcess', function(error, stdout, stderr) {
if (error) {
console.log(error);
} else {
console.log(stdout);
}
});
}
I built a home-grown Netlify. It answers webhooks from Github and executes a bash script to deploy a cute React SPA.
It works great but when run under a PM2 process manager environment, it restarts once the child process closes/exits. I've tested many things - I've even built a test endpoint that feeds it fake data as repeatedly committing fake deploys got tiresome.
Here's some basic code:
const childProcess = require('child_process');
async function processDeploy(deploy) {
console.log(`Processing commit: ${deploy.commit.id}...`);
childProcess.execFile('./deploy.sh', function (error, stdout, stderr) {
if (!error) {
// a function to report the result of the commit
postDeploy(stdout, stderr, deploy.commit, { state: 'deployed' });
} else {
postDeploy('{}', error, deploy.commit, { state: 'error' });
}
});
}
There's a bunch of supporting functions that keep track of the queue and so on to serve a basic status frontend.
I've spent a long time picking through Node and PM2 logs only to find... Nothing. And I think it's partly due to my suspicion that there isn't actually an error - PM2 executes the script, the script exits once completed, and PM2 decides that it's time to kill the process.
The process then restarts as it's set to restart on kill/die.
So, how does PM2 differ to npm start when running my Express-based app? More importantly, how do I fix it...
In case it's relevant, this is my ecosystem.config.js for PM2:
module.exports = {
apps: [
{
name: 'App',
script: 'bin/www',
exec_mode: 'fork',
instances: 1,
autorestart: true,
watch: false,
ignore_watch: ['node_modules', '.log'],
max_memory_restart: '4G',
},
],
};
Update:
I've tried Node versions 10, 12, and 14.
I've switched away from child_process.execFile to child_process.spawn
I've investigated memory issues and scoured PM2's Github issues.
I am now confident that PM2 is catching the child process's event and confusing it with the parent process's.
Terminal output:
Deploy child process exited with code 0 and signal null // child process event listener for exit
Deploy child process closed with code 0 // child process event listener for close
Processing commit completed.
Queue empty.
Something is trying to kill me. // parent process event listener for SIGINT
I want to programmatically restart a process using PM2 API in NodeJS. From their docs, I learned that you can do this by calling pm2.restart(process, errback). Thing is that it is not said anywhere what type the process argument must be.
Does it have to be the name of the process? Or its PM2 process ID? Or a PM2 process object?
Did a quick test, name of the process should work, for example:
var pm2 = require('pm2');
pm2.connect(function(err) {
if (err) {
console.error(err);
process.exit(2);
}
pm2.restart('timer', function(err) {
pm2.disconnect(); // Disconnects from PM2
if (err) throw err
});
});
Where timer is the name of the process started with pm2.
I'm trying to use pm2 to run my node app as a service.
Right now, starting and stopping the app works. However, I want to do a graceful shutdown.
My app already listens for SIGINT, shutdowns the server and then exits the process. However, trying to put pm2 to send the SIGINT, just causes the app to restart, like if pm2 was killing and starting it again.
This is how I create the process:
pm2 start server.js --name ProcessName --silent --kill-timeout 3000
Here's my app's code for listening the SIGINT:
process.on("SIGINT", function () {
//graceful shutdown
server.end().then(() => {
process.exit();
}).catch((err) => {
console.error(err);
});
});
Then to shutdown the app using pm2, I'm running:
pm2 sendSignal SIGINT ProcessName
Which, again, restarts the app.
Reading over pm2 docs, I found that pm2 will also send a shutdown event to the app, so I added:
process.on('message', function(msg) {
if (msg == 'shutdown') {
server.end().then(() => {
process.exit();
}).catch((err) => {
console.error(err);
});
}
});
Which isn't working either.
Any idea how to solve this?
Thanks!
If you haven't solved it yet...
Based on the information you provided, I assume you are running it on Windows.
Your app cannot catch SIGINT sent by PM2 on Windows.
shutdown message works on windows too, but it is sent only by gracefulReload command.
(update)
These are not complete solutions, but might be helpful (hopefully...)
sendSignal command calls process.kill() eventually, and some of these signals might work (haven't tried).
I also found the below method. This can gracefully shutdown a process without restarting only if autorestart option is turned off.
And then your clusters will not reload in case of an accident, so it might not be what you want though...
pm2 lets you send a custom message (reference).
Put the code below in a new file:
var pm2 = require('pm2');
var id = process.argv[2];
pm2.connect(() => {
pm2.sendDataToProcessId({
type: 'shutdown',
data:{some: 'data'},
id: id,
topic: 'some topic'
}, (err, res) => {
console.log('message sent');
pm2.disconnect();
if(err) throw err;
})
});
Modify the part that listens the shutdown message like below:
process.on('message', function(msg){
if(msg == 'shutdown' || msg.type == 'shutdown'){
// code to clean up
}
});
And run the first file with node with id of the cluster you want to shutdown as an argument.
The reason for extra msg.type == 'shutdown' in the condition is that pm2.sendDataToProcessId() requires the argument to be an object with those keys, and does not accept simple shutdown string.
In general pm2 stop is the right way to stop your application. However if you run appliation inside of the Docker you need to use pm2-runtime instead of pm2 which is a part of pm2 npm package and passes system SIGINT to all child processes. See http://pm2.keymetrics.io/docs/usage/docker-pm2-nodejs
Catching the sigint and exiting gracefully should work in your first example.
To actually stop the server, use pm2 stop instead of pm2 sendSignal SIGINT ProcessName.
See http://pm2.keymetrics.io/docs/usage/signals-clean-restart/
I'm trying to bind some shutdown function to my nodejs application (version 0.8.12). Since I'm spawning A LOT of child processes and working in a distributed evironment, I'm currently killing the application through
var n = spawn('killall', ['node']);
The problem is that with this apparently the on('exit', ...) logic is no longer working, indeed I have something like this:
process.on('exit', function() {
if(exit_cb)
exit_cb()
else
console.log('About to exit.');
});
And it's not firing whenever I kill the application.
Is there a way to add a shutdown hook with a killall command or should I find another way to kill my child processes in order to have the hook working?
Thanks
You need to listen for SIGTERM signal that killall sends by dafault. But also you need to stop your process manually after all jobs was finished.
process.on('SIGTERM', function() {
if(exit_cb) {
exit_cb();
process.exit();
} else {
console.log('About to exit.');
process.exit();
}
});