(PERL-> NODEJS) 500: Server closed connection without sending any data back - node.js

Question

My Perl script talks to a Node.js server and sends the commands that the Node.js server needs to execute. While some commands take less time, others take a lot longer. While the command is executing on the server, there is silence on the connection. After a while, I receive the error: 500: Server closed connection without sending any data back
During this error, the command is still executing on the server and the desired results are obtained (if you check the server logs). My problem is that I don't want the connection to reset as there are other follow on commands that need to run after these long running commands. Some commands might take 20 mins
Perl Side Code:
my $ua = LWP::UserAgent->new;
$ua->timeout(12000);
my $uri = URI->new('http://server');
my $json = JSON->new;
my $data_to_json = {DATA};
my $json_content = $json->encode($data_to_json);
# set custom HTTP request header fields
my $req = HTTP::Request->new(POST => $uri);
my $resp = $ua->request($req);
my $message = $resp->decoded_content;
NodeJS Code
var express = require('express');
var http = require('http');
var app = express();
app.use(express.json());
var port = process.env.PORT || 8080;
app.get('<API URL>', function (req, res) {
<get all the passed arguments>
<send output to the console>
});
app.post('<API URL>', function(req, res) {
.
.
.
req.connection.setTimeout(0);
var exec = require('child_process').exec;
var child = exec(command);
}
// start the server
const server = app.listen(port);
server.timeout = 12000;
console.log('Server started! Listening on port' + port);
I have tried to add the timeout for the server using server.timeout and req.connection.setTimeout(0);.
How do I make sure that the connection is not broken?

Answer 1

It's generally a bad idea to make a web worker carry out long running tasks. It ties up the worker and you end up having problems like this.
Instead, have the web worker add a job to some sort of job queue (Perl's Minion is really nice). The queue operates independently of the web server. Clients can poll the server to check on the status of a job and get the output or artifacts when it's complete.
Another advantage of a proper job queue is that you can restart jobs if they fail. The queue knows the job was there. As you've seen, a broken web connection means that it fails and you've probably lost track of those inputs.