In the Node docs, http servers appear to have a close event:
Event: 'close'
function () { }
Emitted when the server closes.
But I can't figure out how to trigger it:
// server.js
var http = require('http');
var server = http.createServer();
server.on('close', function() {
console.log(' Stopping ...');
});
server.listen(8000);
I expected the message "Stopping ..."
$ node server.js
^C Stopping ...
$
But instead there's nothing
$ node server.js
^C$
Does close mean something else here?
Killing the server with Ctrl+C doesn't allow the server to close itself; you have to do this yourself. Something like this should do the trick (on UNIX-like systems):
var http = require('http');
var server = http.createServer();
server.on('close', function() {
console.log(' Stopping ...');
});
process.on('SIGINT', function() {
server.close();
});
server.listen(8000);
Once the HTTP server closes and Node realizes there are no async operations that could be pending, it will automatically stop the process. If you have other stuff on the event loop (timers, etc.) and you want to force the event loop to stop after the server closes, close takes a callback:
process.on('SIGINT', function() {
server.close(function() {
process.exit(0);
});
});
You added an event handler for close. Use server.close(); to trigger the server event.
Below is the snapshot , notice that i am closing the server after 2 seconds at last, so the event 'close' listener will perform callback function soon after.
Incase you want code snippet:
const http = require('http');
const server = http.createServer();
// When Server starts listening
server.on('listening',()=>{
console.log('Captured listening event');
});
// When User tries to connect to server
server.on('connection', ()=>{
console.log('Connection Established');
});
// When the server is closed
server.on('close', ()=>{
console.log('Connection terminated');
});
// 'listening' Event Fired
server.listen(3000);
// Close Server to get 'close' event fired
setTimeout(function(){
server.close();
},2000);
Hope this helped :)
The server doesn't seem to fire any kind of close/finish/terminate event despite what is claimed in the doc.
The workaround I found is to listen to the termination of the node process.
let wasCleanedUp = false;
const runBeforeExiting = (fun: Function) => {
const exitSignals = ['exit', 'SIGINT', 'SIGUSR1', 'SIGUSR2', 'uncaughtException'];
for (const signal of exitSignals) {
process.on(signal as any, async () => { // eslint-disable-line #typescript-eslint/no-explicit-any
if (!wasCleanedUp) {
await fun();
wasCleanedUp = true;
}
process.exit();
});
}
};
// And then before starting your server...
runBeforeExiting(() => {
console.log('clean my application, close the DB connection, etc');
});
Of course it only works if serving requests is your process' only activity, but I assume it is the case in most servers.
Related
When running my Express application in production, I want to shut down the server gracefully when its process is killed (i.e. a SIGTERM or SIGINT is sent).
Here is a simplified version of my code:
const express = require('express');
const app = express();
app.get('/', (req, res) => res.json({ ping: true }));
const server = app.listen(3000, () => console.log('Running…'));
setInterval(() => server.getConnections(
(err, connections) => console.log(`${connections} connections currently open`)
), 1000);
process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);
function shutDown() {
console.log('Received kill signal, shutting down gracefully');
server.close(() => {
console.log('Closed out remaining connections');
process.exit(0);
});
setTimeout(() => {
console.error('Could not close connections in time, forcefully shutting down');
process.exit(1);
}, 10000);
}
When I run it and call the URL http://localhost:3000/ in a browser, the log statement in the setInterval function will keep printing “1 connection currently open” until I actually close the browser window. Even closing the tab will keep the connection open, apparently.
So when I kill my server by hitting Ctrl+C, it will run into the timeout and print “Could not close connections” after 10 seconds, all the while continuing to print “1 connection open”.
Only if I close the browser window before killing the process I get the “closed out remaining connections” message.
What am I missing here? What is the proper way to shut down an Express server gracefully?
I added a listener for connections opening on the server, storing references to those connections in an array. When the connections are closed, they are removed from the array.
When the server is killed, each of the connection is closed by calling its end methods. For some browsers (e.g. Chrome), this is not enough, so after a timeout, I call destroy on each connection.
const express = require('express');
const app = express();
app.get('/', (req, res) => res.json({ ping: true }));
const server = app.listen(3000, () => console.log('Running…'));
setInterval(() => server.getConnections(
(err, connections) => console.log(`${connections} connections currently open`)
), 1000);
process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);
let connections = [];
server.on('connection', connection => {
connections.push(connection);
connection.on('close', () => connections = connections.filter(curr => curr !== connection));
});
function shutDown() {
console.log('Received kill signal, shutting down gracefully');
server.close(() => {
console.log('Closed out remaining connections');
process.exit(0);
});
setTimeout(() => {
console.error('Could not close connections in time, forcefully shutting down');
process.exit(1);
}, 10000);
connections.forEach(curr => curr.end());
setTimeout(() => connections.forEach(curr => curr.destroy()), 5000);
}
The problem you are experiencing is that all modern browsers reuse single connection for multiple requests. This is called keep-alive connections.
The proper way to handle this is to monitor all new connections and requests and to track status of each connection (is it idle or active right now). Then you can forcefully close all idle connections and make sure to close active connections after current request is being processed.
I've implemented the #moebius/http-graceful-shutdown module specifically designed to gracefully shutdown Express applications and Node servers overall. Sadly nor Express, nor Node itself doesn't have this functionality built-in.
Here's how it can be used with any Express application:
const express = require('express');
const GracefulShutdownManager = require('#moebius/http-graceful-shutdown').GracefulShutdownManager;
const app = express();
const server = app.listen(8080);
const shutdownManager = new GracefulShutdownManager(server);
process.on('SIGTERM', () => {
shutdownManager.terminate(() => {
console.log('Server is gracefully terminated');
});
});
Feel free to check-out the module, the GitHub page has more details.
There is open source project https://github.com/godaddy/terminus recommended by the creators of Express (https://expressjs.com/en/advanced/healthcheck-graceful-shutdown.html).
The basic example of terminus usage:
const http = require('http');
const express = require('express');
const terminus = require('#godaddy/terminus');
const app = express();
app.get('/', (req, res) => {
res.send('ok');
});
const server = http.createServer(app);
function onSignal() {
console.log('server is starting cleanup');
// start cleanup of resource, like databases or file descriptors
}
async function onHealthCheck() {
// checks if the system is healthy, like the db connection is live
// resolves, if health, rejects if not
}
terminus(server, {
signal: 'SIGINT',
healthChecks: {
'/healthcheck': onHealthCheck,
},
onSignal
});
server.listen(3000);
terminus has a lot of options in case you need server lifecycle callbacks (ie. to deregister instance from service registry, etc.):
const options = {
// healtcheck options
healthChecks: {
'/healthcheck': healthCheck // a promise returning function indicating service health
},
// cleanup options
timeout: 1000, // [optional = 1000] number of milliseconds before forcefull exiting
signal, // [optional = 'SIGTERM'] what signal to listen for relative to shutdown
signals, // [optional = []] array of signals to listen for relative to shutdown
beforeShutdown, // [optional] called before the HTTP server starts its shutdown
onSignal, // [optional] cleanup function, returning a promise (used to be onSigterm)
onShutdown, // [optional] called right before exiting
// both
logger // [optional] logger function to be called with errors
};
If you allow me, there is even a better solution that involves less work by using server-destroy package. Internally this package will terminate gracefully each connection and then allow the server to be "destroyed". In this case we ensure to definitively end the express application (and potentially start it again if we use a call function). This works for me using electron, and can potentially be ported to a standard server:
const express = require('express')
const { ipcMain } = require('electron')
const enableDestroy = require('server-destroy')
const port = process.env.PORT || 3000
export const wsServer = () => {
try {
let app = null
let server = null
const startServer = () => {
if (app) {
app = null
}
app = express()
app.use(express.static('public'))
app.use('/', (req, res) => {
res.send('hello!')
})
server = app.listen(3000, () => {
console.log('websocket server is ready.')
console.log(`Running webserver on http://localhost:${port}`)
})
enableDestroy(server)
}
const stopServer = () => {
if (server !== null) {
server.destroy()
app = null
server = null
}
}
const restartServer = () => {
stopServer()
startServer()
}
ipcMain.on('start-socket-service', (event) => {
startServer()
console.log('Start Server...')
event.returnValue = 'Service Started'
})
ipcMain.on('stop-socket-service', (event) => {
stopServer()
console.log('Stop Server...')
event.returnValue = 'Service Stopped'
})
ipcMain.on('restart-socket-service', () => {
restartServer()
})
} catch (e) {
console.log(e)
}
}
Check out https://github.com/ladjs/graceful#express
const express = require('express');
const Graceful = require('#ladjs/graceful');
const app = express();
const server = app.listen();
const graceful = new Graceful({ servers: [server] });
graceful.listen();
Http terminator seems to be the 2022 solution that handles keep alive conections properly and force shutdown after some time
https://www.npmjs.com/package/http-terminator
The main benefit of http-terminator is that:
it does not monkey-patch Node.js API
it immediately destroys all sockets without an attached HTTP request
it allows graceful timeout to sockets with ongoing HTTP requests
it properly handles HTTPS connections
it informs connections using keep-alive that server is shutting down by setting a connection: close header
it does not terminate the Node.js process
Try the NPM express-graceful-shutdown module, Graceful shutdown will allow any connections including to your DB to finish, not allow any fresh/new ones to be established. Since you are working with express that may be the module you are looking for, however a quick NPM search will reveal a whole list of modules suited to Http servers etc.
I'm missing something really basic here. I have two very standard express processes with socket io. I'm trying to get a server broadcast on one process to broadcast the same message to another servers clients. I have following setup:
var server = require('http').createServer(app);
var socketio = require('socket.io')(server, {
serveClient: (config.env !== 'production'),
path: '/socket.io-client'
});
require('./config/express')(app);
require('./routes')(app);
server.listen(config.port, config.ip, function () {
logger.info('Express server listening on %d, in %s mode', config.port, app.get('env'));
});
var redis = require('socket.io-redis');
var adapter = redis({ host: config.redis.uri, port: config.redis.port });
adapter.pubClient.on('error', function(){
logger.error('Error on pub client');
});
adapter.subClient.on('error', function(){
logger.error('Error on sub client');
});
socketio.adapter(adapter);
require('./config/socketio')(socketio);
Then my socket io file is:
module.exports = function (socketio) {
//listen to other server broadcasts
socketio.on('socket.io#/#', function(data) {
logger.debug('received message from other process');
});
//client connection code
socketio.on('connection', function (socket) {
socket.address = socket.handshake.address !== null ?
socket.handshake.address + ':' + socket.handshake.address.port :
process.env.DOMAIN;
socket.connectedAt = new Date();
// Call onDisconnect.
socket.on('disconnect', function () {
onDisconnect(socket);
logger.debug('[%s] DISCONNECTED', socket.address);
});
// Call onConnect.
onConnect(socketio, socket);
logger.debug('[%s] CONNECTED', socket.address);
});
};
From a different server I'm just calling a standard emit
socketio.emit('message', object);
In terms of debugging, I load up the redis cli and check the channels that are created:
1) "socket.io#/#"
2) "socket.io-sync#request#/#"
I even run SUBSCRIBE "socket.io#/#" in the cli and can see the message I'm trying to get on the other server displayed via the cli.
This seems so basic, how do I properly subscribe to events on the redis channel 'socket.io#/#'?
EDIT: So I figured out that I receive messages on my other process if I do
socketio.sockets.adapter.subClient.on('message', function (channel, message) {
logger.warn('GOT STUFF ');
});
However here I'm just going straight to the redis sub client. The channel and message are just buffers of non usable data. Isn't this suppose to provide a higher level of abstraction?
EDIT 2: I've got it "mostly" working now. Doing this:
socketio.sockets.adapter.subClient.on('message', function (channel, message) {
logger.warn('GOT STUFF ');
socketio.sockets.in(channel).emit(message);
});
Turns on the socket interface will take the channel and message buffers directly. Now to my next issue, instead of doing socketio.emit(message, doc); I want to do this socketio.sockets.in(doc._id).emit('message;, doc);. Basically i only want it to end in clients that are a part of that room. Doesn't quite seem to be working as is...
I try to setup two node.js servers communication with each other over socket.io. The node servers use SSL, but I don't get it running. I do not get any feedback, its close to this:
Node.js socket.io-client connect_failed / connect_error event
This will not work. No response.
var clientio = require('socket.io-client');
console.log('Trying stuff ...');
// the channel does not exist
var socket = clientio.connect( 'http://localhost:4000/news' );
// I expect this event to be triggered
socket.on('connect_failed', function(){
console.log('Connection Failed');
});
socket.on('connect', function(){
console.log('Connected');
});
socket.on('disconnect', function () {
console.log('Disconnected');
});
but if I try:
// Bind to the news namespace, also get the underlying socket
var ns_news = clientio.connect( 'https://localhost:9000' );
var socket = ns_news.socket
// Global events are bound against socket
socket.on('connect_failed', function(){
console.log('Connection Failed');
});
socket.on('connect', function(){
console.log('Connected');
});
socket.on('disconnect', function () {
console.log('Disconnected');
});
// Your events are bound against your namespace(s)
ns_news.on('myevent', function() {
// Custom event code here
});
I can see that ns_news has no element socket, so I get:
TypeError: Cannot call method 'on' of undefined
So how do I connect these two servers with feedback if the connection is successful or not?
And my following question would be:
How can these two servers authenticate to each other?
Means: Server A says to server B:
- hey, gimme that secret string
And Server B checks the certificate of server A and if it's ok
- here's the string
How do I do it with node?
I have an http server created using:
var server = http.createServer()
I want to shut down the server. Presumably I'd do this by calling:
server.close()
However, this only prevents the server from receiving any new http connections. It does not close any that are still open. http.close() takes a callback, and that callback does not get executed until all open connections have actually disconnected. Is there a way to force close everything?
The root of the problem for me is that I have Mocha tests that start up an http server in their setup (beforeEach()) and then shut it down in their teardown (afterEach()). But since just calling server.close() won't fully shut things down, the subsequent http.createServer() often results in an EADDRINUSE error. Waiting for close() to finish also isn't an option, since open connections might take a really long time to time out.
I need some way to force-close connections. I'm able to do this client-side, but forcing all of my test connections to close, but I'd rather do it server-side, i.e. to just tell the http server to hard-close all sockets.
You need to
subscribe to the connection event of the server and add opened sockets to an array
keep track of the open sockets by subscribing to their close event and removing the closed ones from your array
call destroy on all of the remaining open sockets when you need to terminate the server
You also have the chance to run the server in a child process and exit that process when you need.
For reference for others who stumble accross this question, the https://github.com/isaacs/server-destroy library provides an easy way to destroy() a server (using the approach described by Ege).
I usually use something similar to this:
var express = require('express');
var server = express();
/* a dummy route */
server.get('/', function (req, res) {
res.send('Hello World!');
});
/* handle SIGTERM and SIGINT (ctrl-c) nicely */
process.once('SIGTERM', end);
process.once('SIGINT', end);
var listener = server.listen(8000, function(err) {
if (err) throw err;
var host = listener.address().address;
var port = listener.address().port;
console.log('Server listening at http://%s:%s', host, port);
});
var lastSocketKey = 0;
var socketMap = {};
listener.on('connection', function(socket) {
/* generate a new, unique socket-key */
var socketKey = ++lastSocketKey;
/* add socket when it is connected */
socketMap[socketKey] = socket;
socket.on('close', function() {
/* remove socket when it is closed */
delete socketMap[socketKey];
});
});
function end() {
/* loop through all sockets and destroy them */
Object.keys(socketMap).forEach(function(socketKey){
socketMap[socketKey].destroy();
});
/* after all the sockets are destroyed, we may close the server! */
listener.close(function(err){
if(err) throw err();
console.log('Server stopped');
/* exit gracefully */
process.exit(0);
});
}
it's like Ege Özcan says, simply collect the sockets on the connection event, and when closing the server, destroy them.
I've rewriten original answers using modern JS:
const server1 = http.createServer(/*....*/);
const server1Sockets = new Set();
server1.on("connection", socket => {
server1Sockets.add(socket);
socket.on("close", () => {
server1Sockets.delete(socket);
});
});
function destroySockets(sockets) {
for (const socket of sockets.values()) {
socket.destroy();
}
}
destroySockets(server1Sockets);
My approach comes from this one and it basically does what #Ege Özcan said.
The only addition is that I set a route to switch off my server because node wasn't getting the signals from my terminal ('SIGTERM' and 'SIGINT').
Well, node was getting the signals from my terminal when doing node whatever.js but when delegating that task to a script (like the 'start' script in package.json --> npm start) it failed to be switched off by Ctrl+C, so this approach worked for me.
Please note I am under Cygwin and for me killing a server before this meant to close the terminal and reopen it again.
Also note that I am using express for the routing stuff.
var http=require('http');
var express= require('express');
var app= express();
app.get('/', function (req, res) {
res.send('I am alive but if you want to kill me just go to /exit');
});
app.get('/exit', killserver);
var server =http.createServer(app).listen(3000, function(){
console.log('Express server listening on port 3000');
/*console.log(process);*/
});
// Maintain a hash of all connected sockets
var sockets = {}, nextSocketId = 0;
server.on('connection', function (socket) {
// Add a newly connected socket
var socketId = nextSocketId++;
sockets[socketId] = socket;
console.log('socket', socketId, 'opened');
// Remove the socket when it closes
socket.on('close', function () {
console.log('socket', socketId, 'closed');
delete sockets[socketId];
});
// Extend socket lifetime for demo purposes
socket.setTimeout(4000);
});
// close the server and destroy all the open sockets
function killserver() {
console.log("U killed me but I'll take my revenge soon!!");
// Close the server
server.close(function () { console.log('Server closed!'); });
// Destroy all open sockets
for (var socketId in sockets) {
console.log('socket', socketId, 'destroyed');
sockets[socketId].destroy();
}
};
There is now a closeAllConnections() method in v18.2.0
In Express 3, how do you handle closing database connections when the process exists?
The .on('close', ... event is not emitted unless you explicitly use a HTTP server's .close() call.
So far, this is the closest I've come, but it uses process.on instead of server.on:
process.on('SIGTERM', function () {
// Close connections.
process.exit(0);
});
Based on the info from:
https://github.com/visionmedia/express/issues/1366
http://blog.argteam.com/coding/hardening-node-js-for-production-part-3-zero-downtime-deployments-with-nginx/
--
var express = require('express');
var app = express();
var server = app.listen(1337);
var shutting_down = false;
app.use(function (req, resp, next) {
if(!shutting_down)
return next();
resp.setHeader('Connection', "close");
resp.send(503, "Server is in the process of restarting");
// Change the response to something your client is expecting:
// html, text, json, etc.
});
function cleanup () {
shutting_down = true;
server.close( function () {
console.log( "Closed out remaining connections.");
// Close db connections, other chores, etc.
process.exit();
});
setTimeout( function () {
console.error("Could not close connections in time, forcing shut down");
process.exit(1);
}, 30*1000);
}
process.on('SIGINT', cleanup);
process.on('SIGTERM', cleanup);
The Connection: close header is used to tell any keep-alive connection to close the next time they send a HTTP request. More info here: http://www.jmarshall.com/easy/http/#http1.1s4
I have no idea if there are other ways of closing the keep-alive connection. The node process will hang unless the keep-alive connections are closed. The default idle timeout is 2 mins. More info. on node.js keep-alive timeouts (including on changing the timeout): How to set the HTTP Keep-Alive timeout in a nodejs server