I have following code. It sends simple GET request to the target server.
However when there's no webserver listening on given IP address, node keeps hanging for around 2 minute+ - totally disrespecting timeout value - which should be few seconds.
I've launched tcpdump and i noticed that node keeps sending SYN packets constantly to target server during these 2 minutes.
Why is this happening on http library? I tested alternative library (node-request) and timeout is working correctly there. However for some reasons, i can't use it and need to stick to lower level libs.
http = require('http');
const options = {
hostname: '52.178.167.109',
port: 80,
path: '/upload',
method: 'GET',
timeout: 5000, //5s timeout
};
const req = http.request(options, (res) => {
console.log(`STATUS: ${res.statusCode}`);
req.end();
});
req.on('error', (e) => {
console.error(`problem with request: ${e.message}`);
});
// Write data to request body
req.end();
root#abc:~# time node test.js
problem with request: connect ETIMEDOUT 52.178.167.109:80
real **2m15.474s** <-------
user 0m0.616s
sys 0m0.084s
The timeout property is only applicable to socket inactivity. You want to use request.setTimeout to set a read timeout. The github ticket https://github.com/nodejs/node/issues/12005 clarifies this.
Related
I have a system that is running on Node 8.11.1 on AWS. There is a function that writes logs to another server. This function takes a request object that it logs.
My problem arises during the actual POST attempt, giving me the following error:
Error: write EPROTO 139746875082624:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:827:
I cannot see anything wrong with my code.
Why is the error occurring, and what can I do to fix it?
Here is the code inside of the function:
const https = require('https');
try
{
const postData = "New log finished " + JSON.stringify(request, null, 2);
const options =
{
hostname: LOG_DOMAIN,
port: LOG_PORT,
path: '/',
method: 'POST'
};
const req = https.request(options);
req.on('error', (e) =>
{
console.error("ERROR writing logs: " + e);
});
req.write(postData);
req.end();
}
catch (e)
{
console.log(e);
}
LOG_DOMAIN and LOG_PORT are variables passed to the function.
After further research, it appears that Aravind Voggu (see comments) was in the right vein: the error comes from attempts to use HTTPS for a server that only allows HTTP.
The OpenSSL dies when attempting to secure a connection to something that is unsecured.
The only change required to make my code work correctly was to remove the "s" from "https" in the locations used.
const http = require("http");
This issue can be encountered if server side ssl certificate gets expired
I have developed a nodejs server and react client application. The client requests an information from server using axios. This information has to be retrieved from a database and takes approx 5 mins. But the client retries the request exactly after 2 mins. How do I ask the client to wait for 5 mins.
client code
axios.get(apiServer + 'teradata/'+table).then( (resp) => {
console.log(resp.data.data)
)
In server, I tried below. But nothing works.
server.on('connection', function(socket) {
//Teradata takes too long to respond. Set timeoute to 3 mins
socket.setTimeout(0);
})
server.on('connection', function(socket) {
//Teradata takes too long to respond. Set timeoute to 3 mins
socket.setTimeout(60 * 6 * 1000);
// 30 second timeout. Change this as you see fit.
})
Update:
On further debugging, I see that
1. the retry originates from browser. because, retry happens in firefox, but not in chrome.
2. the connection disconnects exactly at 2 mins. This leads to firefox to retry.
I used the following code in express js and saw that it indeed disconnects. But I don't know who disconnects it. Is it axios or express js?
req.on("close", function() {
console.log('request closed unexpectedly')
});
req.on("end", function() {
console.log('equest ended normally')
});
According to documentation you can create an instance of axios and configure timeout too.
const instance = axios.create({
baseURL: 'https://some-domain.com/api/',
timeout: 5000,
headers: {'X-Custom-Header': 'foobar'}
});
I'm using axios to make HTTP requests.
axios offers the option to specify new http.Agent() in the request config when used in Node.
The http Agent options are:
const agentOptions = {
keepAlive: true, // Keep sockets around even when there are no outstanding requests, so they can be used for future requests without having to reestablish a TCP connection. Defaults to false
keepAliveMsecs: 1000, // When using the keepAlive option, specifies the initial delay for TCP Keep-Alive packets. Ignored when the keepAlive option is false or undefined. Defaults to 1000.
maxSockets: Infinity, // Maximum number of sockets to allow per host. Defaults to Infinity.
maxFreeSockets: 256 // Maximum number of sockets to leave open in a free state. Only relevant if keepAlive is set to true. Defaults to 256.
}
There is also the HTTP header with timeout and max options.
The options don't seem to line up (keepAliveMsecs, maxSockets and maxFreeSockets vs timeout and max), which suggests they are different.
What's the difference between Node's new Agent({ keepAlive: true }) and the HTTP header Keep-Alive: timeout=5, max=1000..?
This is based on what I've gathered myself.
The HTTP header Keep-Alive: timeout=5, max=1000 is just a header sent with HTTP requests. See it as a way to communicate between two hosts (client and server). The host says 'hey keep the connection alive please'. This is automatic for modern browsers and servers might implement it or not. The keepAlive: true of the agent is as the documentation says
Not to be confused with the keep-alive value of the Connection header.
What that means is that keepAlive: false != Connection: close. It doesn't really have anything to do with the header. The agent will take care of things at the TCP level with sockets and such on the HTTP client.
keepAlive boolean Keep sockets around even when there are no outstanding requests, so they can be used for future requests without having to reestablish a TCP connection
As soon as you use an agent for your HTTP client, the Connection: Keep-Alive will be used. Unless keepAlive is set to false and maxSockets to Infinity.
const options = {
port: 3000,
agent: new http.Agent({
keepAlive: false ,
maxSockets: Infinity,
})
};//----> Connection: close
What exactly is an agent?
An Agent is responsible for managing connection persistence and reuse for HTTP clients. It maintains a queue of pending requests for a given host and port, reusing a single socket connection for each until the queue is empty, at which time the socket is either destroyed or put into a pool where it is kept to be used again for requests to the same host and port. Whether it is destroyed or pooled depends on the keepAlive option.
Pooled connections have TCP Keep-Alive enabled for them, but servers may still close idle connections, in which case they will be removed from the pool and a new connection will be made when a new HTTP request is made for that host and port. Servers may also refuse to allow multiple requests over the same connection, in which case the connection will have to be remade for every request and cannot be pooled. The Agent will still make the requests to that server, but each one will occur over a new connection.
Regarding timeout and max, as far as I know, these are set (automatically?) when adding config for Apache
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5
which gives
Connection:Keep-Alive
Keep-Alive:timeout=5, max=100
But these are irrelevant for NodeJS? I'll let more experimented people answer this. Anyway, the agent won't set these and won't modify Connection: Keep-Alive unless setting keepAlive to false and maxSockets to Infinity as said above.
However, for the agent config to have any meaning, Connection must be set to Keep-Alive.
Okay, now for a little experiment to see the agent at work!
I've set up a client for testing (since axios use http.agent for the agent anyway, I just use http).
const http = require('http');
const options = {
port: 3000,
agent: new http.Agent({
keepAlive: true,
maxSockets: 2,
}),
// headers: {
// 'Connection': 'close'
// }
};
var i = 0;
function request() {
console.log(`${++i} - making a request`);
const req = http.request(options, (res) => {
console.log(`STATUS: ${res.statusCode}`);
console.log(`HEADERS: ${JSON.stringify(res.headers)}`);
res.setEncoding('utf8');
res.on('data', (chunk) => {
console.log(`BODY: ${chunk}`);
});
res.on('end', () => {
console.log('No more data in response.');
});
});
req.on('error', (e) => {
console.error(`problem with request: ${e.message}`);
});
req.end();
}
setInterval(function(){ request(); }, 3000); // send a request every 3 seconds
And the server is an express application (I'll skip the details)
server.on('connection', function(socket) {
socket.id = shortid.generate();
//socket.setTimeout(500)
console.log("A new connection was made by a client." + ` SOCKET ${ socket.id }`);
socket.on('end', function() {
console.log(`SOCKET ${ socket.id } END: other end of the socket sends a FIN packet`);
});
socket.on('timeout', function() {
console.log(`SOCKET ${ socket.id } TIMEOUT`);
});
socket.on('error', function(error) {
console.log(`SOCKET ${ socket.id } ERROR: ` + JSON.stringify(error));
});
socket.on('close', function(had_error) {
console.log(`SOCKET ${ socket.id } CLOSED. IT WAS ERROR: ` + had_error);
});
});
To make you see that keepAlive: false != Connection: close, let set keepAlive to false and see what happens server-side.
agent: new http.Agent({
keepAlive: false,
maxSockets: 20
})
Server
Client
As you can see, I've not set maxSockets to Infinity so even though keepAlive in the agent was set to false, the Connection header was set to Keep-Alive. However, each time a request was sent to the server, the socket on the server was immediately closed after each request. Let's see what happens when we set keepAlive to true.
Server
Client
This time around, only one socket have been used. There was a persistent connection between the client and the server that persisted beyond a single request.
One thing I've learned, thanks to this great article is that on Firefox, you can have as many as 6 concurrent persistent connections at a time. And you can reproduce this with the agent by setting maxSockets to 6. For testing purposes, I'll set this to 2. And also, I won't return anything from the server so the connection will be left hanging.
agent: new http.Agent({
keepAlive: true,
maxSockets: 2,
}),
//res.send('response from the server');
Server
Client
The client keeps sending requests but only two have been received by the server. Yet after two minutes, see http_server_timeout
The number of milliseconds of inactivity before a socket is presumed to have timed out.
two new requests are accepted. Actually, the client has queued the subsequent requests and once the server freed the sockets, the client was able to send two new requests from the queue.
So, I hope this helps.
I'm piping to a file an HTTPS request, it works ok 99.9% of calls, but occasionally (maybe when server or network are not available) hangs indefinitely...
This obviously cause my application to stop working and requiring a manual restart...
I have other https connections that used to occasionally hang that always complete now using the following error code on the request object, as suggested on node documentation:
request.on('socket', function(socket) {
socket.setTimeout(10000);
socket.on('timeout', function() { request.abort(); });
});
request.on('error', function(e) {
// Handle the error...
console.error("FAILED!");
});
... but it seems that timeouts on the request are ignored if the destination is piped to a file stream, maybe I should handle an error with a timeout on the filesystem object, but the documentation is not clear if there is an event I have to wait for except for 'finish'...
Here is the sample code, I hope someone can help me:
var https = require('https'),
fs = require('fs');
var opts = {
host: 'www.google.com',
path: '/',
method: 'GET',
port: 443
};
var file = fs.createWriteStream('test.html');
var request = https.request(opts, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(function(){
console.log("OK!");
});
});
});
request.on('socket', function(socket) {
socket.setTimeout(10000);
socket.on('timeout', function() { request.abort(); });
});
request.on('error', function(e) {
console.error("FAILED!");
});
request.end();
If you wanna try the hang, change host and path with a huge file and disconnect the network cable during the transfer, it should time out after 10 seconds, but it doesn't...
I set up a demo node.js http server that sends a very slow answer and a client similar to your sample code.
When I start the client and then stop the server while sending the response then I also don't get a timeout event on the socket but I get a end event on the response within the client:
var request = https.request(opts, function(response) {
response.pipe(file);
file.on('finish', function() {
file.close(function(){
console.log("OK!");
});
});
response.on('end', function() {
// this is printed when I stop the server
console.log("response ended");
});
});
```
Maybe you could listen to that event?
I have a problem with the following code that calls a remote service with a POST request and receives JSON data:
[..]
var options = {
host: REMOTE_HOST,
port: REMOTE_HOST_POST,
path: REMOTE_PATH,
method: 'POST'
};
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (data) {
// handles OK
});
});
req.on('error', function(e) {
// handles KO
});
req.write('user=myname&password=mypassword');
req.end();
[..]
It works ok on my OSX machine, but when I try it on a co-worker Windows computer, the data sent to the server is wrong and I don't know what could be causing the problem. The remote application doesn't receive user=myname&password=mypassword as expected but:
35\r\nuser=myname&password=mypassword
What could be the cause of this? thanks
EDIT
Seems like the problem is not Node.js Further investigation evidences that also my OSX box gives the same error when calling the dev server, while it works ok in production. Seems like a Rails problem. I'