What the title says. I ran into a bug where the issue was an express endpoint not ending the request connection which caused it to seemingly hang. I am still confused how the request and response flow looks like.
The Express http server object has a configurable timeout and after that timeout with no response on the http connection, the server will close the socket.
Similarly, most http clients at the other end (such as browsers) have some sort of timeout that will likely close the TCP socket if they've been waiting for a response for too long.
The http server timeout is built into the underlying http server object that Express uses and you can see how to configure its timeout here: Express.js Response Timeout. By default, the nodejs http server timeout is set to 0 which means "no timeout" is enforced.
So, if you have no server timeout and no client timeout, then the connection will just sit there indefinitely.
You can configure your own Express timeout with:
// set server timeout to 2 minutes
server.timeout = 1000 * 60 * 2;
See doc here.
Where server is the http server object created by either http.createServer() or app.listen().
Related
Given a NodeJS http server with a timeout of 10s:
const httpServer = require('http').createServer(app);
httpServer.timeout = 10 * 1000;
On timeout, Postman shows this without any response code:
Error: socket hang up
Warning: This request did not get sent completely and might not have all the required system headers
If the NodeJS server is behind an nginx reverse proxy, nginx returns a 502 response (upstream prematurely closed connection while reading response header from upstream). But here it is just NodeJS/express running on localhost. Still one would expect a proper http response.
According to this answer, this is expected behavior, the socket is simply destroyed.
In an architecture with an nginx reverse proxy, is it usual that the server just destroys the socket without sending a timeout response to the proxy?
You're setting the socket timeout when you're setting the http server timeout. The socket timeout prevents abuse from clients that might want to hang on to your connection to DOS you. It has other benefits like ensuring a certain level of service (though these are often more important when you're a client).
The reason it uses a socket timeout instead of sending a 408 status code (Request Timeout) is because the status code might have already been sent for a successful message.
If you want to implement a response timeout on your backend and handle it gracefully, you can timeout the response yourself. Note, you should likely respond with a 408 instead. 502 is for gateways like http proxies (nginx) to indicate that a downstream connection failed.
Here's a simple strawman implementation of handling that.
const httpServer = require('http').createServer((req, res) => {
setTimeout(()=>{
res.statusCode = 200;
res.statusMessage = "Ok";
res.end("Done"); // I'm never called because the timeout will be called instead;
}, 10000)
});
httpServer.on('request', (req, res) => {
setTimeout(()=>{
res.statusCode = 408;
res.statusMessage = 'Request Timeout';
res.end();
}, 1000)
});
httpServer.listen(8080);
I get an issue with ExpressJS. I get an error message when uploading large files...
Error message :
Bad Gateway
The proxy server received an invalid response from an upstream server.
Apache/2.4.7 (Ubuntu) Server at xxxxxxxxx Port 80
Everything works when the connexion of the user is above 3G...
Is there any configuration to do on my Apache or about ExpressJS options ?
Edit #1 :
Sounds like the reason is a timeout happening on express server. Try increasing the timeout for the express connections.
const server = app.listen(8080);
server.timeout = 300000; // 5min = 5*60*1000
Problem
Node's default configuration timeouts requests after 2 minutes. I would like to change the request timeouts to:
1 minute for 'normal' requests
5 minutes for requests that serve static files (big assets in this case)
8 hours for uploads (couple of thousand pictures per request)
Research
Reading through Node's documentation, I've discovered that there are numerous ways of defining timeouts.
server.setTimeout
socket.setTimeout
request.setTimeout
response.setTimeout
I'm using Express which also provides middleware to define timeout's for (specific) routes. I've tried that, without success.
Question
I'm confused about how to properly configure the timeout limit globally and per route. Should I configure all of the above timeouts? How is setting the server's timeout different to setting the socket's or request's timeout?
As I saw on your other question concerning the usage of the timeout middleware, you are using it somehow differently.
See documentation of timeout-connect middleware.
Add your errorHandler-function as an EventListener to the request, as it is an EventEmitter and the middleware causes it to emit the timeout-event:
req.on("timeout", function (evt) {
if (req.timedout) {
if (!res.headersSent) {
res
.status(408)
.send({
success: true,
message: 'Timeout error'
});
}
}
});
This is called outside of the middleware stack, causing the function call to next(err) to be invalid. Also, you have to keep in mind, that if the timeout happens while the request is hanging server-side, you have to prevent your server code from further processing this request (because headers are already sent and its underlying connection will no longer be available).
Summary
nodejs timeout API are all inactivity timeout
expressjs/timeout package is response hard timeout
nodejs timeout API
server.timeout
inactivity/idle timeout
equal to socket timeout
default 2min
server.setTimeout
inactivity/idle timeout
equal to socket timeout
default 2min
have callback
socket.setTimeout
inactivity/idle timeout
callback responsible to end(), destroy() socket
default no timeout
response.setTimeout
socket.setTimeout front end
request.setTimeout
socket.setTimeout front end
expressjs/timeout package
response hard-timeout (vs inactivity)
have callback
Conclusion
max. time allowed for an action(request+response), express/timeout package is needed.
This is properly what you need, but the callback need to end the request/response. As the timeout only trigger the callback, it does not change the state or interfere with the connection. It is the callback job.
idle timeout, set nodejs api request/response timeout
I don't recommend touching these, as it is not necessary in most cases. Unless you want to allow a connection to idle(no traffic) over 2min.
There is already a Connect Middleware for Timeout support. You can try this middleware.
var timeout = express.timeout // express v3 and below
var timeout = require('connect-timeout'); //express v4
app.use(timeout(120000)); // should be changed with your desire time
app.use(haltOnTimedout);
function haltOnTimedout(req, res, next){
if (!req.timedout) next();
}
In Node.js there is a default timeout for a server (for an incoming HTTP request) at 120000ms (2 minutes) (see HTTP's server.timeout documentation).
But if I want to do an HTTP request in Node.js (using http.request), looking at the documentation, I only find a function request.setTimeout() to set the timeout manually.
Anyone know if there is a default timeout for HTTP requests in Node.js? Or does Node.js try to send the HTTP request with no end?
You want to set the server.timeout property (it defaults to 120,000, as you've found).
Update: Node.js 13 has removed the default timeout:
https://nodejs.org/api/http.html#servertimeout
https://github.com/nodejs/node/pull/27558
I was also interested in this. By reading the code, Node.js uses Socket under the hood of http request (naturally). (The source link below is referring v8.8.0 at the point when I'm writing this)
https://github.com/nodejs/node/blob/master/lib/_http_outgoing.js
And Socket does not have the timeout by default by this document
https://nodejs.org/dist/latest-v6.x/docs/api/net.html#net_socket_settimeout_timeout_callback
And the source tells the same.
https://github.com/nodejs/node/blob/master/lib/net.js
No. There is no default timeout.
Node.js uses Socket under the hood of http request and socket does not have the timeout by default.
Use the {timeout: XX} parameter of http.request to configure a proper request timeout.
I'm actually doing some load testing against an ExpressJS server, and I noticed that the response send by the server includes a "Connection: Keep-Alive" header. As far as I understand it, the connection will remain opened until the server or the client sends a "Connection: Close" header.
In some implementations, the "Connection: Keep-Alive" header comes up with a "Keep-Alive" header setting the connection timeout and the maximum number of consecutive requests send via this connection.
For example : "Keep-Alive: timeout=15, max=100"
Is there a way (and is it relevant) to set these parameters on an Express server ?
If not, do you know how ExpressJS handles this ?
Edit:
After some investigations, I found out that the default timeout is set in the node standard http library:
socket.setTimeout(2 * 60 * 1000); // 2 minute timeout
In order to change this:
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end("Hello World");
}).on('connection', function(socket) {
socket.setTimeout(10000);
}).listen(3000);
Anyway it still looks a little bit weird to me that the server doesn't send any hint to the client concerning its timeout.
Edit2:
Thanks to josh3736 for his comment.
setSocketKeepAlive is not related to HTTP keep-alive. It is a TCP-level option that allows you to detect that the other end of the connection has disappeared.
For Express 3:
var express = require('express');
var app = express();
var server = app.listen(5001);
server.on('connection', function(socket) {
console.log("A new connection was made by a client.");
socket.setTimeout(30 * 1000);
// 30 second timeout. Change this as you see fit.
});
To set keepAliveTimeout on the express server do:
var express = require('express');
var app = express();
var server = app.listen(5001);
server.keepAliveTimeout = 30000;
For Node.js 10.15.2 and newer with express, only server.keepAliveTimeout was not enough. We also need to configure server.headersTimeout longer than server.keepAliveTimeout.
server.keepAliveTimeout = 30000;
// Ensure all inactive connections are terminated by the ALB, by setting this a few seconds higher than the ALB idle timeout
server.headersTimeout = 31000;
// Ensure the headersTimeout is set higher than the keepAliveTimeout due to this nodejs regression bug: https://github.com/nodejs/node/issues/27363
Update
Since this issue Regression issue with keep alive connections has been closed. We could just set keepAliveTimeout on the latest node.js version.
One more thing, If your node.js server is deployed under AWS ELB and encounters 502 error code occasionally.
Clients -> AWS ELB -> Node Server
AWS ELB has 60 seconds of connection idle timeout by default, and per doc
We also recommend that you configure the idle timeout of your application to be larger than the idle timeout configured for the load balancer
Config the value of keepAliveTimeout to greater than 60 seconds could be one option to eliminate this issue.