Acknowledge Response sent too late; NodeJS Express Asynchrone Response - node.js

Question

I am using NodeJS with express.
I receive from an information publisher a request. I must send back an acknowledge response as soon as I receive the request. Then I should process the request and push back a response.
According to their logs, I sent back the acknowledge response too late and also after the push of the real response. This is wrong and not what I want. I want to follow the model in this picture:
So I have a router.js file with the following code:
app.post('/import/message.io', Importer.handleMessage);
In the importer controller the handleMessage function handles the request:
function handleMessage(req, res){
let otaRequestBuffer = [];
req.on('readable', () => {
let data;
while (data = req.read()) {
otaRequestBuffer.push(data.toString());
}
});
req.on('end', () => {
let otaRequest = otaRequestBuffer.join('');
try {
let parser = new DOMParser();
let xmlDoc = parser.parseFromString(otaRequest, "text/xml");
let soapHeader = parseSoapHeader(xmlDoc);
const soapAction = req.headers['soapaction'].toString();
// Acknowledge the request; The soapHeader is used in here too. That is why it is after the code above
res.writeHead(200);
res.write(
`<?xml version='1.0' encoding='utf-8'?> ...
`
);
res.end();
// Kick off a new job by adding it to the work queue; Response is send in the consumer.js of this job
let job = jobs.create('worker-job', args);
job.on('complete', function(result){
console.log('Job completed');
// Other job.on code would be here
})
} catch (error) {
console.log(error);
} finally {
console.log('DevInfo: Finally called here');
}
});
Other insights:
So the acknowledge response is in the right format.
When I test locally I receive the acknowledge immediately
The logs of the information publisher are here
According to the logs it seems like they actual response (Line5) is pushed to them before the acknowledge response (Line1) .
Is this possible with the code example I pasted above?
Thanks for the help.

Answer 1

It looks to be entirely possible that the real response (push) is sent before the acknowledgement is received by the client. Since response.end() is asynchronous (e.g. the call will complete before the response is sent), we're effectively racing
res.end();
with
let job = jobs.create('worker-job', args);
This is the kind of annoying thing one tends to see only once one goes to production!
Perhaps you could try queuing your job once the end() callback is called. e.g.
res.end(() => {
console.log("End callback, response stream is complete, starting job...");
// Kick off a new job by adding it to the work queue; Response is send in the consumer.js of this job
let job = jobs.create('worker-job', args);
job.on('complete', function(result){
console.log('Job completed');
// Other job.on code would be here
})
});
Since (from the Node.js docs)...
If callback is specified, it will be called when the response stream is finished.
https://nodejs.org/api/http.html#http_response_end_data_encoding_callback
(This is the method that Express wraps with response.end)
http://expressjs.com/en/5x/api.html#res.end
I'd also suggest logging detailed timestamps down to the millisecond on your server, so you can see exactly when calls are happening.
The setTimeout approach is certainly worth a try, it might get the results you wish:
// Set delay as appropriate
const delayMs = 2000;
setTimeout(() => {
console.log("Timeout complete, starting job...");
// Kick off a new job by adding it to the work queue; Response is send in the consumer.js of this job
let job = jobs.create('worker-job', args);
job.on('complete', function(result){
console.log('Job completed');
// Other job.on code would be here
})
}, delayMs);