NodeJS to NodeJS http requests hang - node.js

Question

I have 2 services: 1. worker 2.fetch documents
Both are Nodejs express services.
The worker uses Promise.all to send 48 http POST (dont ask why post and not GET) requests to the fetch service, the fetch service then fetches all 48 documents (each in a separate request) and sends them back to the worker.
Logs are showing the fetch service finished everything successfuly and the worker sending all 48 requests, but showing partial responses (sometimes 0, sometimes 15/48 sometimes 31/48 but never fully succeeds)
The worker keeps waiting for a response it seems, untill the 15 minute timeout for the job end and its moved to failed.
code examples:
worker.js (service 1 - NodeJS with express)
await Promise.all(docIDs.map(async (docID) => {
try {
logger.info(`Worker Get Document: Fetching ${docID.docID} transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
var document = await axios.post(getVariableValue("GET_DOCUMENT_SERVICE_URL"), docID, {
httpsAgent: new https.Agent({
rejectUnauthorized: false,
keepAlive: false
}),
auth: {
username: getVariableValue("WEBAPI_OPTIDOCS_SERVICE_USERNAME"),
password: getVariableValue("WEBAPI_OPTIDOCS_SERVICE_PASSWORD")
},
headers: {
"x-global-transaction-id": transactionId,
"timeStamp": timeStamp
}
});
logger.info(`Worker Get Document: Fetched ${docID.docID} Status: ${document.data.status}. transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
documents.push(document.data.content);
}
catch (err) {
const responseData = err.response ? err.response.data : err.message
logger.error(`Worker Get Document: Failed DocID ${docID.docID}, Error received from Get Document: ${responseData} - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
throw Error(responseData)
}
}));
fetch.js (service 2 - NodeJS with express)
module.exports = router.post('/getDocument', async (req, res, next) => {
try {
var transactionId = req.headers["x-global-transaction-id"]
var timeStamp = req.headers["timestamp"]
logger.info(`GetDocumentService: API started. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
var document = await getDocuemntService(req.body, transactionId, timeStamp);
var cloneDocument = clone(document)
logger.info(`GetDocumentService: Document size: ${JSON.stringify(cloneDocument).length / 1024 / 1024}. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
logger.info(`GetDocumentService: API Finished. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
res.status(200);
res.set("Connection", "close");
res.json({
statusDesc: "Success",
status: true,
content: cloneDocument
});
logger.info(`GetDocumentService: Response status: ${res.finished} . - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
}
catch (err) {
logger.error(`GetDocumentService: Error:${err.message} / Stack: ${err.stack}. - transaction ID: ${transactionId} / timeStamp: ${timeStamp}`, {})
res.status(500)
res.send("stack: " + err.stack + "err: " + err.message + " fromgetdocument")
}
});
So the logs from the get document are full and show success.
The logs from the worker show like this:
"fetching" X48
"fetched" X0, X15, X31 (three different attempts at fetching 48 docs)
I have tried changing keep-alive to true.
Anything else I might be missing? anyone knows why it hangs forever (atleast 15 minutes untill job gets "timed out")
Thanks :)

Answer 1

I only have guesses here but I feel they are strong guesses.
Axios has a default timeout of none but I'm gonna guess somewhere there is a timeout of 900 seconds. Edit: This timeout is very likely in the server side web server.
You are hammering the server with 48 request in a fraction of a second. I'm not surprised this is failing.
Their server is likely rate limiting you or just getting overloaded.
For testing limit your request to one at a time. When one finishes send the next.
Simple work queue example - Not tested
The addWork or doWork caller will need to await
let queue = []
const sleep = (ms)=> new Promise((resolve)=>setTimeout(resolve,ms))
async addWork(work, cb, sleepMS) {
if (Array.isArray(work) queue = [...queue, ...work]
else queue.push(work)
if (cb) return doWork(cb, sleepMS)
}
async doWork(cb, sleepMS) {
let work = queue.pop()
var document = await getDocuemntService(work)
cb(document)
if (queue.length > 0) {
if (sleepMS) await sleep(sleepMS)
return doWork(cb, sleepMS)
}
}
Once you confirm that one at a time works then you can look at a more complex work queue. I'd check NPM for something I'm sure there is something out there to handle this.