I'm working on implementing graceful shutdown for a node API. Typically, during development, this process is started using the common start script, but I have noticed that this causes some annoying behavior that I'm surprised I've never really noticed before in my 3 or so years as a node developer.
To begin shutdown during development, we simply hit ctrl+C in the bash terminal, which of course causes a SIGINT to be sent to the npm process and all of its child processes. I can capture this with a process.on handler to initiate a graceful shutdown from there-- closing off new requests, waiting for existing requests to finish, then killing the database connection, all that good stuff.
However, if the developer hits ctrl+C a second time, npm behaves differently. It seems to be sending a SIGTERM to the sh process it used to invoke my start script in the first place. This causes that sh process to print out Terminated and exit, returning control of the terminal to the user without waiting for the node process to exit.
This is really annoying because it gives the impression that the node process has been stopped, but of course it hasn't. It will continue until shutdown is complete, or it is forcibly killed with something like SIGKILL or SIGQUIT. If it happens to print anything out to the console, it will do so directly in the middle of whatever else the developer might now be running in that terminal.
For a trivial example, try this:
package.json:
{
"private": true,
"scripts": {
"start": "node index.js"
}
}
index.js:
async function waitForever() {
while(true) {
console.log('waiting...');
await new Promise((resolve) => {
setTimeout(resolve, 5000);
});
}
}
process.on('SIGINT', () => {
console.log('SIGINT recieved');
});
process.on('SIGTERM', () => {
console.log('SIGTERM recieved');
})
waitForever();
Run npm start in your terminal, then hit ctrl+c once. You'll see the signal make it through to node, but of course it won't exit. Now, do it a second time. You'll see the signal make it through again, but then you immediately see "Terminated" followed by your shell prompt. Until you find the process ID of the node process and kill it with kill -9 you'll keep seeing that waiting... message every five seconds.
I did some more fiddling around with this example and it very much seems like npm is completely responsible for this. If you send kill -2 directly to the npm process twice, the termination of the shell process occurs, without SIGINT ever being received by the node process.
So I have two main questions:
What the heck is going on here? Am I missing something about how my shell works, or is this some kind of feature built in to npm run-script? If so, where I can find information about this? npm help run-script shows nothing about it.
I know that start scripts are pretty common in projects like this, so it seems like someone else should be encountering this problem. How do people typically deal with it? I've Googled around a bunch and it's been hard to find a clear answer.
This isn't a huge deal, of course. The start script is just a convenience to make sure a TS compilation runs before starting up. I can have developers run the built app directly in their shell after building, or write a script that performs the build and start outside of an npm script. But it would be nice to not have to do this.
Really I'm just puzzled and would appreciate some assistance. Thanks!
To answer your 1 - looking at npm code this is expected behavior of handling SIGINT. First occurrence of signal is passed down to child process and also attaches one-time listener for subsequent SIGINT that will kill the npm parent process immediately. You can see code here.
I assume this is because npm start is meant as development stage shorthand only and there it makes sense to have a "handbrake" to kill process immediately in cases where you e.g. get signal handling wrong (unfortunately even that doesn't work in all cases as you found out).
I don't have answer for 2 but sometime ago there was a lengthy debate about this going in various npm issues regarding signal handling and npm start. Official NPM statement was mostly that npm start is not replacement for proper process manager (e.g. supervisor or systemd) and shouldn't be used in production environment like that.
EDIT: An answer for 2 from sripberger:
What I ended up doing follows. shutdown is a function that performs the shutdown, returning a promise. The first SIGINT will begin the shutdown, but a second will forcibly kill the process whether or not shutdown is finished. This does not prevent npm from terminating the shell process, but it does make sure the node process dies with it when this happens:
process.once('SIGINT', () => {
console.log('\nShutting down, please wait...');
// Begin the graceful shutdown.
shutdown().catch((err) => {
console.error('Could not shut down gracefully:', err);
});
// Attach a subsequent handler to force kill.
process.on('SIGINT', () => {
console.log('\nProcess killed.');
process.exit(0);
});
});
Of course, as noted by blami, it is not recommended to use npm scripts to control services in production. This is simply a convenience for development environments.
Related
I'm writing a very demanding program in rust that had variable Threads for processing very important data, and want to know if there is a way that i can send a signal to stop it with systemctl in a way that i can be sure that it is finishing it's dutties before stop, as its very demanding, uses http_request, and threads are variables I can not make an estimation of how much time i have to wait since signal sended until the process is dead.
In esscense, it is a daemon that is in a loop until a variable sets false, like this
loop {
// Process goes here
if !is_alive {
break;
}
}
What i'm doing right now is that the program is asking a "config.json" file if it is "alive", but i think it's not the best way because i don't know when the program stops, just can see that is stoping, but not how much is going to last, and if i do this way, systemctl is going to show the service alive, even if i shuted it down manually.
If you want to experiment with Systemd service behavior, I would take a look at the Systemd documentation. In this case, I would direct you to the section about TimeoutStopSec.
According to the documentation, you can disable any timeout on systemd stop commands with TimeoutStopSec=infinity. This combined with actually handling the SIGTERM signal that systemd uses by default should do the trick.
Furthermore, there is the KillSignal option by which you can specify the signal that is sent to your program to stop it or ExecStop to specify a program to run in order to stop your service.
With these you should be able to figure it out, I hope.
I am building Node.js scripts that will be ran as CRON jobs. (full terminal scripts). These scripts fetch data from all around using APIs and MongoDB (native driver) is used. I don't didn't use db.close() statement and because of that script will never end by itself (or at least it looks like that way from the terminal), to finish it, it is necessary to press CTRL+C to cancel.
Back then when I was writing these scripts, someone from Stack overflow told me that it is not required to close connection anyway. So I let it be.
Now I wonder, do these scripts actually are still running? And as these would be ran as CRON jobs, with small intervals, does that mean that these scripts will eventually kill RAM from the server? Does that mean, there will be thousands scripts running and waiting for db.close() statement?
Example code:
MongoClient.connect(mongoUrl, (err, db) => {
if (err) {
console.log(err);
return;
}
var usersCollection = db.collection('users').find();
usersCollection.on('data', (doc) => {
console.log(doc);
});
Node scripts exit by themselves only when nothing listens for events any more.
With your scripts you probably know when nothing needs to be done after the main purpose of your script is achieved. For example, when the purpose of your script is to execute a certain code, print a summary etc. then you can add process.exit(0); after that command to make sure that the script finishes when it should.
In the case like this one when you don't have a one line of summary after which you can exit, you can listen for a certain event and exit when when it arrives. For example:
usersCollection.on('end', process.exit);
In addition to a process.exit in the right spot, you can also set a timeout to terminate the script after a certain time. For example:
setTimeout(process.exit, 20*1000);
will terminate the script after 20 seconds. This can be added just in case that somethings goes wrong with the database or the connection and you never get the 'end' event, or it takes too long to wait for it.
Sometimes adding process.exit(0); for scripts like this may be more convenient than closing all the database connections that may be open at that time or other things that may prevent your script from terminating.
Is there a way to execute a piece of code in Node.js Express just before the node.js process exits, regardless whether it was due to an error being thrown, or pressing Ctrl+C, or any other reason?
You're looking for the exit event. From the documentation:
Emitted when the process is about to exit. This is a good hook to
perform constant time checks of the module's state (like for unit
tests). The main event loop will no longer be run after the 'exit'
callback finishes, so timers may not be scheduled.
And it would be implemented as
process.on('exit', function() {
console.log('About to close');
});
It's worth mentioning, that it's generally not a good idea to try to manipulate data if you don't know the reason for the exit or exception. If you don't know what's wrong, it's usually a better idea to start fresh, than try to accomplish something with something that may very well be FUBAR.
Node js module during the operation requests some resources on remote service, that better be released when it exits. We know that there is very nice:
process.on('exit', function() {
// ...
});
But then it is said that it won't wait for any async operations to complete. So the question is if there's any workaround (there should be some, since it's quite widespread usage case)? Maybe one could start separate process or something?..
Only workaround I've seen is adding a wait loop and not finishing/returning from the .on('exit', function until a property has been updated globally.
Totally a bodge-job design-wise, very bad practice, but I've seen it work for short calls (I think there is some timeout but I never bothered to look into the details).
I think you could/should do clean-up before on('exit') by listening for ctrl-c signal like in this post.
Is it possible to intercept the default kill signal and use it as a command for a graceful shutdown? This is for Solaris SMF. The easiest way to have a stoppable service that I have found is to set :kill as the shutdown script and then to add a shutdown hook in Java. In this case, I want to do it for Node.JS. How should I do it?
Edit: The purpose is to
Stop receiving new requests.
Give existing callbacks a few seconds to finish.
Write some information to stderr.
#alienhard's first suggestion was to use process.on('exit'... but it seems that I would not be able to accomplish number 2 with this method.
There is an exit event: http://nodejs.org/docs/v0.3.1/api/process.html#event_exit_
process.on('exit', function() {
console.log('About to exit.');
});
Edit: An alternative that could work for you, is instead of killing the process, sending a signal like SIGUSR1 (kill -s SIGUSR1), and then listening for this signal (see link posted by #masylum in another answer) and after you are done or some time has elapsed explicitly terminate with process.exit().
The only thing that comes to my mind is using signal events.
http://nodejs.org/docs/v0.3.1/api/process.html#signal_Events