Related
I discovered an odd behaviour in node's promisify() function and I cannot work out why it's doing what it's doing.
Consider the following script:
#!/usr/bin/env node
/**
* Module dependencies.
*/
var http = require('http')
var promisify = require('util').promisify
;(async () => {
try {
// UNCOMMENT THIS, AND NODE WILL QUIT
// var f = function () { return 'Straight value' }
// var fP = promisify(f)
// await fP()
/**
* Create HTTP server.
*/
var server = http.createServer()
/**
* Listen on provided port, on all network interfaces.
*/
server.listen(3000)
server.on('error', (e) => { console.log('Error:', e); process.exit() })
server.on('listening', () => { console.log('Listening') })
} catch (e) {
console.log('ERROR:', e)
}
})()
console.log('OUT OF THE ASYNC FUNCTION')
It's a straightforward self-invoking function that starts a node server.
And that's fine.
NOW... if you uncomment the lines under "UNCOMMENT THIS", node will quit without running the server.
I KNOW that I am using promisify() on a function that does not call the callback, but returns a value instead. So, I KNOW that that is in itself a problem.
However... why is node just quitting...?
This was really difficult to debug -- especially when you have something more complex that a tiny script.
If you change the function definition to something that actually calls a callback:
var f = function (cb) { setTimeout( () => { return cb( null, 'Straight value') }, 2000) }
Everything works as expected...
UPDATE
Huge simplification:
function f () {
return new Promise(resolve => {
console.log('AH')
})
}
f().then(() => {
console.log('Will this happen...?')
})
Will only print "AH"!
Call promisify() on a non-callback function: “interesting” results in node. Why?
Because you allow node.js to go to the event loop with nothing to do. Since there are no live asynchronous operations in play and no more code to run, node.js realizes that there is nothing else to do and no way for anything else to run so it exits.
When you hit the await and node.js goes back to the event loop, there is nothing keeping node.js running so it exits. There are no timers or open sockets or any of those types of things that keep node.js running so the node.js auto-exit-detection logic says that there's nothing else to do so it exits.
Because node.js is an event driven system, if your code returns back to the event loop and there are no asynchronous operations of any kind in flight (open sockets, listening servers, timers, file I/O operations, other hardware listeners, etc...), then there is nothing running that could ever insert any events in the event queue and the queue is currently empty. As such, node.js realizes that there can never be any way to run any more code in this app so it exits. This is an automatic behavior built into node.js.
A real async operation inside of fp() would have some sort of socket or timer or something open that keeps the process running. But because yours is fake, there's nothing there and nothing to keep node.js running.
If you put a setTimeout() for 1 second inside of f(), you will see that the process exit happens 1 second later. So, the process exit has nothing to do with the promise. It has to do with the fact that you've gone back to the event loop, but you haven't started anything yet that would keep node.js running.
Or, if you put a setInterval() at the top of your async function, you will similarly find that the process does not exit.
So, this would similarly happen if you did this:
var f = function () { return 'Straight value' }
var fP = promisify(f);
fP().then(() => {
// start your server here
});
Or this:
function f() {
return new Promise(resolve => {
// do nothing here
});
}
f().then(() => {
// start your server here
});
The issue isn't with the promisify() operation. It's because you are waiting on a non-existent async operation and thus node.js has nothing to do and it notices there's nothing to do so it auto-exits. Having an open promise with a .then() handler is not something that keeps node.js running. Rather there needs to be some active asynchronous operation (timer, network socket, listening server, file I/O operation underway, etc...) to keep node.js running.
In this particular case, node.js is essentially correct. Your promise will never resolve, nothing else is queued to ever run and thus your server will never get started and no other code in your app will ever run, thus it is not actually useful to keep running. There is nothing to do and no way for your code to actually do anything else.
If you change the function definition to something that actually calls a callback:
That's because you used a timer so node.js has something to actually do while waiting for the promise to resolve. A running timer that has not had .unref() called on it will prevent auto-exit.
Worth reading: How does a node.js process know when to stop?
FYI, you can "turn off" or "bypass" the node.js auto-exit logic by just adding this anywhere in your startup code:
// timer that fires once per day
let foreverInterval = setInterval(() => {
// do nothing
}, 1000 * 60 * 60 * 24);
That always gives node.js something to do so it will never auto-exit. Then when you do want your process to exit, you could either call clearInterval(foreverInterval) or just force things with process.exit(0).
I am have multiple micro services written in Nodejs Koa running in Docker Swarm.
Since container orchestration tools like Kubernetes or Swarm can scale up and down services instantly, I have a question on gracefully shutting down Nodejs service to prevent unfinished running process.
Below is the flow I can think of:
sending a SIGNINT signal to each worker process, Does docker swarm
send SIGNINT to worker when scaling down service?
the worker are responsible to catch the signal, cleanup or free any
used resource and finish the its process, How can I stop new api
request, wait for any running process to finish before shutting
down?
Some code below from reference:
process.on('SIGINT', () => {
const cleanUp = () => {
// How can I clean resources like DB connections using Sequelizer
}
server.close(() => {
cleanUp()
process.exit()
})
// Force close server after 5secs, `Should I do this?`
setTimeout((e) => {
cleanUp()
process.exit(1)
}, 5000)
})
I created a library (https://github.com/sebhildebrandt/http-graceful-shutdown) that can handle graceful shutdowns as you described. Works well with Express and Koa.
This package also allows you to create function (should return a promise) to additionally clean up things like DB stuff, ... here some example code how to use it:
const gracefulShutdown = require('http-graceful-shutdown');
...
server = app.listen(...);
...
// your personal cleanup function - this one takes one second to complete
function cleanup() {
return new Promise((resolve) => {
console.log('... in cleanup')
setTimeout(function() {
console.log('... cleanup finished');
resolve();
}, 1000)
});
}
// this enables the graceful shutdown with advanced options
gracefulShutdown(server,
{
signals: 'SIGINT SIGTERM',
timeout: 30000,
development: false,
onShutdown: cleanup,
finally: function() {
console.log('Server gracefulls shutted down.....')
}
}
);
I personally would increase the final timeout from 5 secs to higher value (10-30 secs). Hope that helps.
I want the Express.js-based server of the app I am working on to properly close all connections before exiting the node process when a shutdown signal (SIGTERM or SIGINT) is sent.
I have slightly modified the express-graceful-shutdown middleware (which is really simple), so the relevant part of my code looks approximately as follows:
server.js
import gracefulShutdown from './tools/graceful-shutdown';
const server = app.listen(config.port, () => {
console.log(`Server is running`);
});
app.use(gracefulShutdown(server));
graceful-shutdown.js:
function gracefulShutdownMiddleware(server) {
let shuttingDown = false;
const forceTimeout = 10 * 1000; // giving the app 10 seconds to shutdown gracefully
process.on('SIGTERM', gracefulExit); // listen for TERM signal (e.g. kill)
process.on ('SIGINT', gracefulExit); // listen for INT signal (e.g. Ctrl-C)
function gracefulExit() {
if (shuttingDown) return;
shuttingDown = true;
console.log('Received kill signal (SIGTERM), shutting down');
setTimeout(function () {
console.log('Could not close connections in time, forcefully shutting down');
process.exit(1);
}, forceTimeout);
server.close(function () {
console.log('Closed out remaining connections.');
process.exit();
});
}
function middleware(req, res, next) {
if (!shuttingDown) return next()
res.set('Connection', 'close')
res.status(503).send('Server is in the process of restarting.')
}
return middleware
}
export default gracefulShutdownMiddleware;
The problem I am observing is this. If I shut down the node server immediately after it starts, it shuts down as expected (function gracefulExit in the middleware works correctly, server.close executes fine, and the callback to server.close prints out the console message). If, however, I open the browser and actually visit my app on the localhost, so that the server has some work to do, and then try to shut the server down, the server.close function does not seem to finish (so it never calls the callback with process.exit until finally the setTimeout calls its own callback and forcefully shuts down the app.
The server.close function is this function from node’s lib.
Could you help me figure out why node server’s close function may fail to complete?
.close() doesn't actually close any active connections, it will only stop new connections from being made, and it'll wait for the active connections to get closed (by the client).
If clients are connected that keep a connection open (say using a keep-alive connection), the callback may never get called.
There's a package called http-shutdown that will help with gracefully shutting down an HTTP server.
I'm writing some tests and would like to be able to start/stop my HTTP server programmatically. Once I stop the HTTP server, I would like the process that started it to exit.
My server is like:
// file: `lib/my_server.js`
var LISTEN_PORT = 3000
function MyServer() {
http.Server.call(this, this.handle)
}
util.inherits(MyServer, http.Server)
MyServer.prototype.handle = function(req, res) {
// code
}
MyServer.prototype.start = function() {
this.listen(LISTEN_PORT, function() {
console.log('Listening for HTTP requests on port %d.', LISTEN_PORT)
})
}
MyServer.prototype.stop = function() {
this.close(function() {
console.log('Stopped listening.')
})
}
The test code is like:
// file: `test.js`
var MyServer = require('./lib/my_server')
var my_server = new MyServer();
my_server.on('listening', function() {
my_server.stop()
})
my_server.start()
Now, when I run node test.js, I get the stdout output that I expect,
$ node test.js
Listening for HTTP requests on port 3000.
Stopped listening.
but I have no idea how to get the process spawned by node test.js to exit and return back to the shell.
Now, I understand (abstractly) that Node keeps running as long as there are bound event handlers for events that it's listening for. In order for node test.js to exit to the shell upon my_server.stop(), do I need to unbind some event? If so, which event and from what object? I have tried modifying MyServer.prototype.stop() by removing all event listeners from it but have had no luck.
I've been looking for an answer to this question for months and I've never yet seen a good answer that doesn't use process.exit. It's quite strange to me that it is such a straightforward request but no one seems to have a good answer for it or seems to understand the use case for stopping a server without exiting the process.
I believe I might have stumbled across a solution. My disclaimer is that I discovered this by chance; it doesn't reflect a deep understanding of what's actually going on. So this solution may be incomplete or maybe not the only way of doing it, but at least it works reliably for me. In order to stop the server, you need to do two things:
Call .end() on the client side of every opened connection
Call .close() on the server
Here's an example, as part of a "tape" test suite:
test('mytest', function (t) {
t.plan(1);
var server = net.createServer(function(c) {
console.log("Got connection");
// Do some server stuff
}).listen(function() {
// Once the server is listening, connect a client to it
var port = server.address().port;
var sock = net.connect(port);
// Do some client stuff for a while, then finish the test
setTimeout(function() {
t.pass();
sock.end();
server.close();
}, 2000);
});
});
After the two seconds, the process will exit and the test will end successfully. I've also tested this with multiple client sockets open; as long as you end all client-side connections and then call .close() on the server, you are good.
http.Server#close
https://nodejs.org/api/http.html#http_server_close_callback
module.exports = {
server: http.createServer(app) // Express App maybe ?
.on('error', (e) => {
console.log('Oops! Something happened', e));
this.stopServer(); // Optionally stop the server gracefully
process.exit(1); // Or violently
}),
// Start the server
startServer: function() {
Configs.reload();
this.server
.listen(Configs.PORT)
.once('listening', () => console.log('Server is listening on', Configs.PORT));
},
// Stop the server
stopServer: function() {
this.server
.close() // Won't accept new connection
.once('close', () => console.log('Server stopped'));
}
}
Notes:
"close" callback only triggers when all leftover connections have finished processing
Trigger process.exit in "close" callback if you want to stop the process too
To cause the node.js process to exit, use process.exit(status) as described in http://nodejs.org/api/process.html#process_process_exit_code
Update
I must have misunderstood.
You wrote: "...but I have no idea how to get the process spawned by node test.js to exit and return back to the shell."
process.exit() does this.
Unless you're using the child_processes module, node.js runs in a single process. It does not "spawn" any further processes.
The fact that node.js continues to run even though there appears to be nothing for it to do is a feature of its "event loop" which continually loops, waiting for events to occur.
To halt the event loop, use process.exit().
UPDATE
After a few small modifications, such as the proper use of module.exports, addition of semicolons, etc., running your example on a Linux server (Fedora 11 - Leonidas) runs as expected and dutifully returns to the command shell.
lib/my_server.js
// file: `lib/my_server.js`
var util=require('util'),
http=require('http');
var LISTEN_PORT=3000;
function MyServer(){
http.Server.call(this, this.handle);
}
util.inherits(MyServer, http.Server);
MyServer.prototype.handle=function(req, res){
// code
};
MyServer.prototype.start=function(){
this.listen(LISTEN_PORT, function(){
console.log('Listening for HTTP requests on port %d.', LISTEN_PORT)
});
};
MyServer.prototype.stop=function(){
this.close(function(){
console.log('Stopped listening.');
});
};
module.exports=MyServer;
test.js
// file: `test.js`
var MyServer = require('./lib/my_server');
var my_server = new MyServer();
my_server.on('listening', function() {
my_server.stop();
});
my_server.start();
Output
> node test.js
Listening for HTTP requests on port 3000.
Stopped listening.
>
Final thoughts:
I've found that the conscientious use of statement-ending semicolons has saved me from a wide variety of pernicious, difficult to locate bugs.
While most (if not all) JavaScript interpreters provide something called "automatic semicolon insertion" (or ASI) based upon a well-defined set of rules (See http://dailyjs.com/2012/04/19/semicolons/ for an excellent description), there are several instances where this feature can inadvertently work against the intent of the programmer.
Unless you are very well versed in the minutia of JavaScript syntax, I would strongly recommend the use of explicit semicolons rather than relying upon ASI's implicit ones.
I have a Node.js application that contains an http(s) server.
In a specific case, I need to shutdown this server programmatically. What I am currently doing is calling its close() function, but this does not help, as it waits for any kept alive connections to finish first.
So, basically, this shutdowns the server, but only after a minimum wait time of 120 seconds. But I want the server to shutdown immediately - even if this means breaking up with currently handled requests.
What I can not do is a simple
process.exit();
as the server is only part of the application, and the rest of the application should remain running. What I am looking for is conceptually something such as server.destroy(); or something like that.
How could I achieve this?
PS: The keep-alive timeout for connections is usually required, hence it is not a viable option to decrease this time.
The trick is that you need to subscribe to the server's connection event which gives you the socket of the new connection. You need to remember this socket and later on, directly after having called server.close(), destroy that socket using socket.destroy().
Additionally, you need to listen to the socket's close event to remove it from the array if it leaves naturally because its keep-alive timeout does run out.
I have written a small sample application you can use to demonstrate this behavior:
// Create a new server on port 4000
var http = require('http');
var server = http.createServer(function (req, res) {
res.end('Hello world!');
}).listen(4000);
// Maintain a hash of all connected sockets
var sockets = {}, nextSocketId = 0;
server.on('connection', function (socket) {
// Add a newly connected socket
var socketId = nextSocketId++;
sockets[socketId] = socket;
console.log('socket', socketId, 'opened');
// Remove the socket when it closes
socket.on('close', function () {
console.log('socket', socketId, 'closed');
delete sockets[socketId];
});
// Extend socket lifetime for demo purposes
socket.setTimeout(4000);
});
// Count down from 10 seconds
(function countDown (counter) {
console.log(counter);
if (counter > 0)
return setTimeout(countDown, 1000, counter - 1);
// Close the server
server.close(function () { console.log('Server closed!'); });
// Destroy all open sockets
for (var socketId in sockets) {
console.log('socket', socketId, 'destroyed');
sockets[socketId].destroy();
}
})(10);
Basically, what it does is to start a new HTTP server, count from 10 to 0, and close the server after 10 seconds. If no connection has been established, the server shuts down immediately.
If a connection has been established and it is still open, it is destroyed.
If it had already died naturally, only a message is printed out at that point in time.
I found a way to do this without having to keep track of the connections or having to force them closed. I'm not sure how reliable it is across Node versions or if there are any negative consequences to this but it seems to work perfectly fine for what I'm doing. The trick is to emit the "close" event using setImmediate right after calling the close method. This works like so:
server.close(callback);
setImmediate(function(){server.emit('close')});
At least for me, this ends up freeing the port so that I can start a new HTTP(S) service by the time the callback is called (which is pretty much instantly). Existing connections stay open. I'm using this to automatically restart the HTTPS service after renewing a Let's Encrypt certificate.
If you need to keep the process alive after closing the server, then Golo Roden's solution is probably the best.
But if you're closing the server as part of a graceful shutdown of the process, you just need this:
var server = require('http').createServer(myFancyServerLogic);
server.on('connection', function (socket) {socket.unref();});
server.listen(80);
function myFancyServerLogic(req, res) {
req.connection.ref();
res.end('Hello World!', function () {
req.connection.unref();
});
}
Basically, the sockets that your server uses will only keep the process alive while they're actually serving a request. While they're just sitting there idly (because of a Keep-Alive connection), a call to server.close() will close the process, as long as there's nothing else keeping the process alive. If you need to do other things after the server closes, as part of your graceful shutdown, you can hook into process.on('beforeExit', callback) to finish your graceful shutdown procedures.
The https://github.com/isaacs/server-destroy library provides an easy way to destroy() a server with the behavior desired in the question (by tracking opened connections and destroying each of them on server destroy, as described in other answers).
As others have said, the solution is to keep track of all open sockets and close them manually. My node package killable can do this for you. An example (using express, but you can call use killable on any http.server instance):
var killable = require('killable');
var app = require('express')();
var server;
app.route('/', function (req, res, next) {
res.send('Server is going down NOW!');
server.kill(function () {
//the server is down when this is called. That won't take long.
});
});
var server = app.listen(8080);
killable(server);
Yet another nodejs package to perform a shutdown killing connections: http-shutdown, which seems reasonably maintained at the time of writing (Sept. 2016) and worked for me on NodeJS 6.x
From the documentation
Usage
There are currently two ways to use this library. The first is explicit wrapping of the Server object:
// Create the http server
var server = require('http').createServer(function(req, res) {
res.end('Good job!');
});
// Wrap the server object with additional functionality.
// This should be done immediately after server construction, or before you start listening.
// Additional functionailiy needs to be added for http server events to properly shutdown.
server = require('http-shutdown')(server);
// Listen on a port and start taking requests.
server.listen(3000);
// Sometime later... shutdown the server.
server.shutdown(function() {
console.log('Everything is cleanly shutdown.');
});
The second is implicitly adding prototype functionality to the Server object:
// .extend adds a .withShutdown prototype method to the Server object
require('http-shutdown').extend();
var server = require('http').createServer(function(req, res) {
res.end('God job!');
}).withShutdown(); // <-- Easy to chain. Returns the Server object
// Sometime later, shutdown the server.
server.shutdown(function() {
console.log('Everything is cleanly shutdown.');
});
My best guess would be to kill the connections manually (i.e. to forcibly close it's sockets).
Ideally, this should be done by digging into the server's internals and closing it's sockets by hand. Alternatively, one could run a shell-command that does the same (provided the server has proper privileges &c.)
I have answered a variation of "how to terminate a HTTP server" many times on different node.js support channels. Unfortunately, I couldn't recommend any of the existing libraries because they are lacking in one or another way. I have since put together a package that (I believe) is handling all the cases expected of graceful HTTP server termination.
https://github.com/gajus/http-terminator
The main benefit of http-terminator is that:
it does not monkey-patch Node.js API
it immediately destroys all sockets without an attached HTTP request
it allows graceful timeout to sockets with ongoing HTTP requests
it properly handles HTTPS connections
it informs connections using keep-alive that server is shutting down by setting a connection: close header
it does not terminate the Node.js process
Usage:
import http from 'http';
import {
createHttpTerminator,
} from 'http-terminator';
const server = http.createServer();
const httpTerminator = createHttpTerminator({
server,
});
await httpTerminator.terminate();
const Koa = require('koa')
const app = new Koa()
let keepAlive = true
app.use(async (ctx) => {
let url = ctx.request.url
// destroy socket
if (keepAlive === false) {
ctx.response.set('Connection', 'close')
}
switch (url) {
case '/restart':
ctx.body = 'success'
process.send('restart')
break;
default:
ctx.body = 'world-----' + Date.now()
}
})
const server = app.listen(9011)
process.on('message', (data, sendHandle) => {
if (data == 'stop') {
keepAlive = false
server.close();
}
})
process.exit(code); // code 0 for success and 1 for fail