I have developed a nodejs server and react client application. The client requests an information from server using axios. This information has to be retrieved from a database and takes approx 5 mins. But the client retries the request exactly after 2 mins. How do I ask the client to wait for 5 mins.
client code
axios.get(apiServer + 'teradata/'+table).then( (resp) => {
console.log(resp.data.data)
)
In server, I tried below. But nothing works.
server.on('connection', function(socket) {
//Teradata takes too long to respond. Set timeoute to 3 mins
socket.setTimeout(0);
})
server.on('connection', function(socket) {
//Teradata takes too long to respond. Set timeoute to 3 mins
socket.setTimeout(60 * 6 * 1000);
// 30 second timeout. Change this as you see fit.
})
Update:
On further debugging, I see that
1. the retry originates from browser. because, retry happens in firefox, but not in chrome.
2. the connection disconnects exactly at 2 mins. This leads to firefox to retry.
I used the following code in express js and saw that it indeed disconnects. But I don't know who disconnects it. Is it axios or express js?
req.on("close", function() {
console.log('request closed unexpectedly')
});
req.on("end", function() {
console.log('equest ended normally')
});
According to documentation you can create an instance of axios and configure timeout too.
const instance = axios.create({
baseURL: 'https://some-domain.com/api/',
timeout: 5000,
headers: {'X-Custom-Header': 'foobar'}
});
Related
I have a use case where I need to make more than 10,000 external HTTP requests(to one API) in an infinite loop every 10 seconds. This API takes anywhere from 200-800ms to respond.
Goals:
Call an external API more than 10,000 times in 10 seconds
Have a failproof polling system that can run for months at a time without failing
Attempts:
I have attempted to use the Async library and limit requests to 300 concurrent calls(higher numbers fail quickly) but after about 300,000 requests I run into errors where I receive a connection refused(I am calling a local Node server on another port). I am running the local server to mimic the real API because our application has not scaled to more than 10,000 users yet and the external API requires 1 unique user per request. The local response server is a very simple Node server that has one route that waits between 200-800ms to respond.
Am I getting this error because I am calling a server running locally on my machine or is it because Node is having an issue handling this many requests? Any insight into the best way to perform this type of polling would be appreciated.
Edit: My question is about how to create a client that can send more than 10,000 requests in a 10 second interval.
Edit2:
Client:
//arr contains 10,000 tokens
const makeRequests = arr => {
setInterval(() => {
async.eachLimit(arr, 300, (token, cb) => {
axios.get(`http://localhost:3001/tokens/${token}`)
.then(res => {
//do something
cb();
})
.catch(err => {
//handle error
cb();
})
})
}, 10000);
}
Dummy Server:
const getRandomArbitrary = (min, max) => {
return Math.random() * (max - min) + min;
}
app.get('/tokens:token', (req, res) => {
setTimeout(() => {
res.send('OK');
}, getRandomArbitrary(200, 800))
});
I have following code. It sends simple GET request to the target server.
However when there's no webserver listening on given IP address, node keeps hanging for around 2 minute+ - totally disrespecting timeout value - which should be few seconds.
I've launched tcpdump and i noticed that node keeps sending SYN packets constantly to target server during these 2 minutes.
Why is this happening on http library? I tested alternative library (node-request) and timeout is working correctly there. However for some reasons, i can't use it and need to stick to lower level libs.
http = require('http');
const options = {
hostname: '52.178.167.109',
port: 80,
path: '/upload',
method: 'GET',
timeout: 5000, //5s timeout
};
const req = http.request(options, (res) => {
console.log(`STATUS: ${res.statusCode}`);
req.end();
});
req.on('error', (e) => {
console.error(`problem with request: ${e.message}`);
});
// Write data to request body
req.end();
root#abc:~# time node test.js
problem with request: connect ETIMEDOUT 52.178.167.109:80
real **2m15.474s** <-------
user 0m0.616s
sys 0m0.084s
The timeout property is only applicable to socket inactivity. You want to use request.setTimeout to set a read timeout. The github ticket https://github.com/nodejs/node/issues/12005 clarifies this.
i am working upon mongoose to list all the data from a collection in mongodb but i want the response to come after 5 minutes but it is not respondin when timeout value is 5 minute but it responds when timeout is 1 minute
router.get(routeIdentifier+'/list/:id', function(req, res, next) {
model.find({}, function (err, objects) {
setTimeout(function(){
if (err) return res.send(err);
objects.push({id:req.params.id})
return res.json(objects);
},300000)
});
})
;
As I can sense, it is mainly due to the HTTP server timeout. It times out before sending the response from the database model .
The default timeout is 2 minutes as per the documentation here: https://nodejs.org/dist/latest-v9.x/docs/api/http.html#http_server_settimeout_msecs_callback
Please note that ExpressJS sits on the top of built-in NodeJS HTTP server.
Try the following:
let server = http.createServer( expressApp );
server.setTimeout( 300000 );
This will initialize the HTTP-Server timeout to be 5 minutes.
I am implementing a real time commenting tool using Redis PubSub and NodeJS. On the server, I watch for req close or end event to ensure I close/end the corresponding subscribers. On my OSX dev machine, I am able to detect the close event but on AWS elasticbeanstalk setup, the close event just doesn't get fired. I have tried listening to the other events as well(both for req and res) but those don't seem to be firing either!.
Both my dev and aws setups use Node v0.12.2, Express v4.5.1 nginx 1.6.2 on aws. The aws setup runs behind a load balancer.
Has anyone else ever faced anything similar to this issue?
var publisherClient = redis.createClient();
publisherClient.on("error",function(err){
console.log(err);
})
setInterval(function(){
publisherClient.publish("Ping", '{"Pong":'+"}");
},20000)
exports.amastream=function (req,res){
req.socket.setTimeout(40*1000);
var messageCount = 0;
var subscriberClient = redis.createClient();
subscriberClient.subscribe("Ping");
subscriberClient.on("error", function(err) {
console.log("Redis Error: " + err);
});
subscriberClient.on("message", function(channel, message) {
messageCount++;
res.write('id: ' + messageCount + '\n');
res.write("data: " +channel + '\n');
res.write("data: " +message + '\n\n');
res.flush();
});
req.on("end", function() {
console.log("closing connection due to end of request!");
subscriberClient.punsubscribe();
subscriberClient.quit();
});
req.on("close", function() {
console.log("closing connection!");
subscriberClient.punsubscribe();
subscriberClient.quit();
});
req.on("error", function() {
console.log("closing connection due to abrupt ending!");
subscriberClient.punsubscribe();
subscriberClient.quit();
});
req.on("timeout",function() {
console.log("closing connection due to timeout!");
subscriberClient.punsubscribe();
subscriberClient.quit();
})
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache',
'You-Are-Like-Us':'Contact us at tarkeshwar#fundamine.com'
});
res.write('id: ' + 0 + '\n');
res.write("data: " +"first message" + '\n\n');
res.flush();
}
I have the same issue. This behaviour is caused by the Elastic Load Balancing.
We should take a look to the Elastic Load Balancing Connection Timeout Management for better unsderstanind on how ELB works.
So, it has the client and the backend connection and idle timeout value.
For some reason ELB can not properly detect when your client connection is closed and close the associated backend connection. Acctually your backend connection stay opened. Thats why you don`t receive the close event.
Idle timeout at the backend side will not work since you are sending ping to client. Howhever, I wonder, where does the ELB send this data if the client connection does not exist?
There is a post on the AWS support forum about this issue. The topic starter tells that it takes approximately 55 mintues to close the connection to backend. From my investigation it could take much longer.
We also use the Server-Sent Events in our application and we discovered that it leads to the enourmous number of idle connections, which is holded by the ELB after the user closed or reloaded the page.
You could monitor the number of connections using the following code (I didn`t find a metric at AWS wich shows the number of active 'backend' connection to a particular server from ELB):
var server = require('http').createServer(app);
setInterval(function (){
server.getConnections(function (err, count){
console.log("Number of connections : " + count);
});
}, 60000);
Unfortunately we cant move to the web sockets because EBS does not support it without additional workarounds or using the HAProxy server.
As a workaround it is possible setup a timer wich will perform the req.end each N minutes. In this case client code should detect that server connection is closed and establish a new one.
Using node.js http.createServer to listen POST requests. If request completes fast, all works good. If request complete time > 5 seconds i get no response returned to client.
Added event listeners on created sockets:
server.on('connection', function(socket) {
log.info('SOCKET OPENED' + JSON.stringify(socket.address()));
socket.on('end', function() {
log.info('SOCKET END: other end of the socket sends a FIN packet');
});
socket.on('timeout', function() {
log.info('SOCKET TIMEOUT');
});
socket.on('error', function(error) {
log.info('SOCKET ERROR: ' + JSON.stringify(error));
});
socket.on('close', function(had_error) {
log.info('SOCKET CLOSED. IT WAS ERROR: ' + had_error);
});
});
Got those messages on middle of request (after about 10 sec after start):
info: SOCKET TIMEOUT
info: SOCKET CLOSED. IT WAS ERROR: false
But on client socket do not get closed, so client wait for response. End of request completed with success, response sent (on closed socket!), but client still wait.
No idea how to block those timeouts. Removed all timeouts from code. Tried to add KeepAlive, no result.
socket.setKeepAlive(true);
How to prevent socket bultin timeout?
From node's setTimeout documentation the connection will not be severed when a timeout occurs. That's why the browser's still waiting. You're still required to end() or destroy() the socket. You can increase the timeout by calling setTimeout on the socket.
socket.setTimeout(1000 * 60 * 300); // 5 hours
You can do a couple of things:
socket.setTimeout(/* number of milliseconds */);
If you do this, then the server can get a timeout event:
server.on('timeout', function(timedOutSocket) {
timedOutSocket.write('socket timed out!');
timedOutSocket.end();
});