Koa - cannot increase default request timeout - node.js

I need help increasing the 2 min default request timeout in a Koa server.
I have a long task operation that is taking a few minutes. When it is finished I'm sending a response back. The problem is that the connection is automatically closed after a timeout of 2 min which is node js default.
I tried everything in google.
tried using all kind of third party npm modules.
tried ctx.request.socket.setTimeout(0)
I am out of ideas and need help.
I am executing my requests to the server using postmen with infinite timeout.
Update - This is a code snipped of something im trying to do:
const Koa = require('koa')
const app = new Koa()
const PORT = 7555
const server = app.listen(PORT)
app.use(async (ctx, next) => {
ctx.req.setTimeout(0);
await next();
});
app.use(async (ctx, next) => {
const wait = async () => {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve()
}, 1000 * 60 * 5)
})
}
await wait()
console.log("settimeout finished", new Date())
ctx.response.body = { success: "true", date: new Date() }
})

I tested this one ... worked fine:
const Koa = require('koa')
const app = new Koa()
const PORT = 7555
const server = app.listen(PORT);
server.setTimeout(0); // <-- this should do the trick ...
app.use(async (ctx, next) => {
console.log("request started", new Date())
const wait = async () => {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve()
}, 1000 * 60 * 5)
})
}
await wait()
console.log("settimeout finished", new Date())
ctx.response.body = { success: "true", date: new Date() }
})
Be sure that - if apache or nginx is involved in your production system to also modify their configurations (here I increased it to 500 seconds).
For NGINX (proxy)
vim /etc/nginx/nginx.conf
Add following in http{..} section
(see http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout)
http {
#...
proxy_read_timeout 500s;
proxy_send_timeout 500s;
#...
}
For Apache
vim /etc/apache2/apache2.conf:
Search for TimeOut
# ...
TimeOut 500
# ...

You could add a middleware early on your app like this:
app.use(async (ctx, next) => {
ctx.req.setTimeout(0);
await next();
});
It seems koa keeps the 120s timeout default from http_server.js
So, you could try to hack into it, but this middleware should do the trick.
You could also try this:
const server = app.listen(PORT); // This returns the https_server instance
server.setTimeout(600000)
You can see relevant info here

Related

setTimeout gets overwritten by second setTimeout in Node.js

const express = require("express");
const REFRESH_INTERVAL = 1000;
const app = express();
const recurringSetTimeout = (id) => {
setTimeout(
handle = async () => {
console.log(`Starting setTimeout${id}`);
setTimeout(handle, REFRESH_INTERVAL);
},
REFRESH_INTERVAL,
)
}
const main = () => {
app.get("/t1", async (req, res) => {
const msg = "Starting timeout 1...";
console.log(msg)
recurringSetTimeout(1);
res.send(msg);
});
app.get("/t2", async (req, res) => {
const msg = "Starting timeout 2...";
console.log(msg)
recurringSetTimeout(2);
res.send(msg);
});
app.listen(3000, () => {
console.log("Server is running...");
});
}
main();
I have this code that should run two different setTimeouts on two route calls, t1 runs first and t2 runs second. After calling t1 I am getting correct results in logs, "Starting setTimeout1" and after each second again. But when I call t2, I am expecting to get "Starting setTimeout2" but as well as "Starting setTimeout1" from the previous route call. But it seems that setTimeout2 somehow overrides setTimeout1, since I am getting only "Starting setTimeout2" but after each second two logs, instead of one. So it seems that setTimeout1 is running but gets overwritten by setTimeout2, since I am getting only timeout2 logs (2 per second).
If I run setInterval, instead of setTimeout, it works fine, but I want to understand this behaviour of setTimeout, can someone please explain. Thanks!

Is it bad practice to perform http requests in main module?

I want to sync data from an external resource (file, server etc) into my DB every time the server starts.
app.js
const app = express();
app.use('/add', require('./addLayer'));
...
(async() => {
const external_resource = await axios.get('...'); // get data
await axios.post('http://localhost:3000/add', external_resource.data); // add it to db
});
addLayer.js
const DBObject = require('../../models/DBObject')
const addNewLayer = async (req, res) => {
try {
const data = validateData(req.body)
await DBObject.create(data)
} catch (err) {
res.status(400).send(err)
}
}
I do not want to re-write the code in the /add route (which includes data validations), but I find performing POST requests in here not good for some reason. Is there a better way to do so?

How to retry api calls using node-fetch

I am trying to POST a request to external API using node-fetch in my node js code.
I want to retry the request 3 times if there are any timeouts or network failures while doing the POST request.
Could you let me know how to retry the request using node-fetch? I see that there is a npm module https://www.npmjs.com/package/node-fetch-retry but it doesnt seem to work as expected and it also doesn't accept retry interval between retries. Any code snippets will be very helpful.
EDIT:
Thanks i tried using promise-fn-retry but it doesnt seem to do any retry. Below is the code snippet i tried by switching off my WIFI and then doing the FETCH call to see if it retried 3 times. But it does the fetch just once and returns the error.
const promiseFn = () => fetch(url,{method:"POST",headers:header,body:JSON.stringify(payload)});
const options = {
times: 3,
initialDelay: 100,
onRetry: (error) => {
console.log(error);
}
};
console.log('PromiseFn result ****'+retry(promiseFn, options).then(res => res.json()).then((res)=>{console.log('Promise Fn result is'+JSON.stringify(res))}).catch((err)=>{console.log('Error in Promise Fn'+err)}));
this snippet helped me in the past! Hope it is what you are looking for
const fetch = require('node-fetch');
const delay = (ms) => new Promise((resolve) => setTimeout(() => resolve(), ms));
const retryFetch = (
url,
fetchOptions = {},
retries = 3,
retryDelay = 1000,
timeout
) => {
return new Promise((resolve, reject) => {
// check for timeout
if (timeout) setTimeout(() => reject('error: timeout'), timeout);
const wrapper = (n) => {
fetch(url, fetchOptions)
.then((res) => resolve(res))
.catch(async (err) => {
if (n > 0) {
await delay(retryDelay);
wrapper(--n);
} else {
reject(err);
}
});
};
wrapper(retries);
});
};
retryFetch('http://localhost:8080/test', {}, 20)
.then((res) => res.text())
.then(console.log)
.catch(console.error);
Try the promise-fn-retry package, it has properties to specify retry times, initial delay time and callback methods to handle failed fetch/promises.
Hope this helps!

[Koa]404 while passing throught the routeur

I'm having some trouble with the Koa framework. I'm trying to build a pretty basic server by I'm having a problem with my router. The ctx always return 404 despite passing in my functions.
Some code :
//www.js
const Koa = require('koa');
const app = new Koa();
const version = require('./routes/version');
app.listen(config.port, () => {
console.log('Server is listenning on port ' + config.port);
});
app.use(version.routes());
app.use(ctx => {
console.log ('test')
});
//version.js
const Router = require('koa-router');
const router = new Router();
router.prefix('/version');
router.use((ctx, next) => {
ctx.vFactory = new VersionFactory(ctx.app.db);
next();
});
router.get('/', getAllVersions);
async function getAllVersions(ctx, next) {
const ret = await ctx.vFactory.getAllVersions();
ctx.body = JSON.stringify(ret.recordset);
console.log(ctx.body)
await next();
}
I've checked a few threads. Most of the time, the problem seems to come from a non Promise based function in the await part of the router function. Here it is a simple DAO using mssql which is pretty promise based.
class DaoVersion {
constructor(db) {
this.pool = db;
}
async getAllVersions() {
const me = this;
return new Promise((resolve) => {
const ret= me.pool
.query(getVersion);
resolve(ret);
});
}
}
The console output seems good. I have my ctx.body set with my db data but if I try to check the whole context, I still have a 404. More interesting, if I try to ctx.res.write (using default node response) I got the "already end" message. So it seems Koa have sent the message before passing threw my function.
Any idea why and how I could correct that ?
Koa default response.status code is 404, unlike node's res.statusCode which defaults to 200.
Koa changes the default status code to 200 - when your route set's a non empty value to ctx.body or in some cases you can manually change (like if you need to set it to 202) it by using ctx.status = xxx.
You can use this documentation for reference: https://github.com/koajs/koa/blob/master/docs/api/response.md
Also, your route should be an async function:
router.get('/', async(ctx, next) => {
ctx.body = await getAllVersions
await next()
}

Add intentional latency in express

Im using express with node.js, and testing certain routes. I'm doing this tute at http://coenraets.org/blog/2012/10/creating-a-rest-api-using-node-js-express-and-mongodb/
Im calling the http://localhost:3000/wines via ajax (the content doesn't matter). But I want to test latency. Can I do something like make express respond after 2 seconds? (I want to simulate the ajax loader and I'm running on localhost so my latency is pretty much nil)
Use as middleware, for all your requests
app.use(function(req,res,next){setTimeout(next,1000)});
Just call res.send inside of a setTimeout:
setTimeout((() => {
res.send(items)
}), 2000)
To apply it globaly on all requests you can use the following code:
app.use( ( req, res, next ) => {
setTimeout(next, Math.floor( ( Math.random() * 2000 ) + 100 ) );
});
Time values are:
Max = 2000 (sort of.... min value is added so in reality its 2100)
Min = 100
Try connect-pause module. It adds delay to all or only some routes in your app.
app.get('/fakeDelay', function(req,res){
let ms = req.query.t;
ms = (ms>5000 || isNaN(ms)) ? 1000 : parseInt(ms);
setTimeout((()=> res.status(200).send({delay:ms})), ms);
})
Then request the URL as: http://localhost/fakeDelay/?t=2000
(max 5000ms and default of 1000ms on this example)
Update:
Using a Promise. The function 'sleep' can be used for delaying any Express response or other async function.
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
app.get('/fakeDelay', async (req, res) => {
await sleep(500);
res.send([])
})
just add a comment on top of the solution of #maggocnx : put this middleware early (before your route handler)
app.use(function(req,res,next){setTimeout(next,1000)});
You could also just write your own generic delay handler using a Promise or callback (using a q promise in this case):
pause.js:
var q = require('q');
function pause(time) {
var deferred = q.defer();
// if the supplied time value is not a number,
// set it to 0,
// else use supplied value
time = isNaN(time) ? 0 : time;
// Logging that this function has been called,
// just in case you forgot about a pause() you added somewhere,
// and you think your code is just running super-slow :)
console.log('pause()-ing for ' + time + ' milliseconds');
setTimeout(function () {
deferred.resolve();
}, time);
return deferred.promise;
}
module.exports = pause;
then use it however you'd like:
server.js:
var pause = require('./pause');
router.get('/items', function (req, res) {
var items = [];
pause(2000)
.then(function () {
res.send(items)
});
});
You can use the express-delay package.
The code below will delay all incoming requests by one second.
const app = require('express')();
const delay = require('express-delay');
app.use(delay(1000));
The package also offers the possibility to introduce a random delay within specified boundaries, e.g. by calling delay(1000, 2000) for a delay between one and two seconds.
In my case I wanted a way to have the same processing time for all of my endpoints.
The solution I found is to override one of the Response methods :
/* MINIMUM DELAY */
const minDelay = 200; /* ms */
app.use((req, res, next) => {
const tmp = res.json;
const start = new Date().getTime();
(res.json as any) = async (body: any) => {
await new Promise((re) => setTimeout(re, minDelay - new Date().getTime() + start));
tmp.apply(res, [body]);
};
next();
});
This way an attacker would not be able to differentiate failed login requests from OK requests just by looking at the response time :)

Resources