I have an expressjs server and it uses a fork inside a route to make sure that the main loop isn't blocked (it's somewhat computing intensive). Every time this route is called, the server restarts. I've debugged this problem and found that the forking is what causes this behaviour but I don't understand why. The route is defined as follows:
module.exports = async function someComputingIntensiveFunction(req, res) {
try {
// some stuff
const childProcess = fork('../path/to/file.js');
childProcess.on('message', (data) => {
res.status(201).json(data).end();
});
catch (error) {
res.status(500).end()
}
}
Inside this file is
process.on('message', (data) => {
// do some stuff with data
// based on whatever the result is
process.send(result);
process.exit(result.status);
});
Am I forgetting a crucial part of forking which causes the expressjs server to restart? Thanks in advance for any help.
Related
I've been having trouble with handling data from MongoDB using MongoJS where I have a promise that refuses to resolve. I am honestly confused why this does not work. It works completely fine on my local machine but once I move it to my server it breaks. Except it only breaks as a module. Is it because I'm using deasync? If it is, how do I fix it?
Here's what I'm doing compressed down to what doesn't work for me
In the index:
const db = require('./db')
//Obviously this works I'm just showing what I mean by "module"
data = db.get('somedata') //Passthrough for getData
console.log(data)
//On local machine, console says: thedata
//On deb server, console says: Promise { 'thedata' }
//On deb server, console ALSO says: MongoDB request timed out: Promise { 'thedata' }
// ^ (this is from the timeout part I added because it froze)
In the DB module:
const db = mongojs(`connectionURIstring`, ['bot'])
const bdata = db.bot
function syncP(promise) {
var answer;
promise.then(value => {
//Runs every time on local machine but not on server as module
answer = value
})
//I also have .catch but it's not relevant
setTimeout(function () {
//Never runs on local machine but runs every time on server as module
if (answer) return;
answer = promise
console.error('MongoDB request timed out:', promise)
}, 2000)
require('deasync').loopWhile(function () { return !answer })
return answer
}
function getData(datapath) {
//Datapath processing here
var GDP = new Promise((resolve, reject) => {
bdata.findOne({
_id: primary
}, function (err, dataJ) {
//Stuff where I format data, comes out as variable data
resolve(data)
})
})
return syncP(GDP)
}
I genuinely don't know why this doesn't work. There's no reason it shouldn't work as a module if it works independently. It works on another computer. The only thing I can assume is that it's something to do with how deasync works, but even then, there's no reason why that would interfere with the promise, as it's separate from the promise itself.
This is driving me insane.
How to stop a process in Express when the request timeouts?
install 'connect-timeout'
import timeout from 'connect-timeout';
// halt_on_timeout.js
module.exports = function haltOnTimedout(req, res, next) {
if (!req.timedout) next();
};
expensive process takes longer than 30 seconds
the route timesouts, but the expensive_long_operation never stops running..
route.post(
'upload_alot_of_content/'
timeout('30s'),
haltOnTimedout,
async (req, res) => {
const result = await expensive_long_operation();
if (req.timedout) {
next('error!')
}
res({....})
})
There is no way to arbitrarily stop an asynchronous process in nodejs, however there are a couple options that you can implement to achieve what you want.
You could run the the expensive_long_operation in a forked nodejs process and then kill that process if there is timeout.
You could write the expensive_long_operation as an asynchronous C++ module and build in a mechanism that would allow you to break out of the operation.
I'm using react, electron, nodejs, asyncjs redux and thunk.
I wrote the following code which is supposed to download a list of files and write it to disk. In my code when the user presses a button i call this actionCreator:
export function downloadList(pack) {
return (dispatch, getState) => {
const { downloadManager } = getState();
async.each(downloadManager.downloadQueue[pack].libs, async (url, callback) => {
const filename = url.split('/').pop().split('#')[0].split('?')[0];
await downloadFile(url, `dl/${filename}`);
callback();
}, (err) => {
if (err) {
console.log('A file failed to process');
} else {
dispatch({
type: DOWNLOAD_COMPLETED,
packName: pack
});
}
});
};
}
async function downloadFile(url, path) {
const file = fs.createWriteStream(path);
const request = https.get(url, (response) => {
response.pipe(file);
file.on('finish', () => {
file.close();
});
}).on('error', (err) => { // Handle errors
fs.unlink(path); // Delete the file async. (But we don't check the result)
});
}
It does what it's supposed to do but while it does that, it blocks the entire UI. I really can't understand why it's happening since if i use an
setTimeout
with 3000ms delay inside the async.each it doesn't block the UI.
Another strange behaviour is that if i use the eachLimit function of asyncJS it just downloads me the limit of files, so if i want to download 100 files but i set eachLimit to 10 parallel, it just downloads the first 10 files and then stops. Can you enlight me about this?
I wanted to use axios to download files since it doesn't need to know if the urls are http or https but i can't find any resource on using axios with stream responsetype
I can answer the first part. Pretty much every existent implementation of JavaScript runs on one thread. This means that the runtime is concurrent, but not parallel, i.e. the runtime does one and exactly one thing at a time. This means that if there is a function call that takes a while, it will block everything else. Therefore, something in the downloadList function is blocking the event loop. However, if you use setTimeout, then the downloadList function will be pushed onto the message queue, which will unblock the event and allow the UI to be rendered. For more information on the event loop check out this video
I have this function and it's called during my node server starts up in index.js file. In the init function, it sets setInterval to call refreshKeyValues periodically which takes about 10 sec.
I ran some tests and sent a health check (simple get call to health end point returning 200 ok) and it seemed that the health check blocked when refreshKeyValues was running.
Is setInterval blocking? How do I make this refreshing logic running on background so that it won't block the incoming requests?
export default function initMyClient(server) {
server.clients.myclient = MyClient.createNewClient();
server.clients.myclient.init();
server.keyValues = getKeyValues();
function refreshKeyValues() {
server.keyValues = getKeyValues();
}
const refreshInterval = setInterval(
refreshKeyValues,
1000
);
server.on('close', function onClose() {
clearInterval(refreshInterval);
});
}
The JavaScript's setInterval function is asynchronous, so it not blocks the main flow when it in "waiting" state. But your function refreshKeyValues seems to be not async, so it may block the main JavaScript flow. When it called, the refreshKeyValues can block the flow, because of JavaScript is single threaded.
You can take better understanding while reading through this: JavaScript event loop
Ivan Matveev is completely correct. Node.js is single threaded, even though it might look like it is not due to its async nature. However you can achieve what you need with the cluster module. Here is how to do it:
var cluster = require('cluster');
if (cluster.isMaster) {
var httpWorker = cluster.fork(); // worker thread for server
var intervalWorker = cluster.fork(); // worker thread for setInterval
// send messages to your wokers specifying the worker type
httpWorker.send({ server: true });
intervalWorker.send({ server: false });
} else {
process.on('message', function (data) {
if (data.server) {
// start server
var server = require('http').createServer(function (req, res) {
res.end('OK');
});
server.listen(8080, function () {
console.log('server running');
});
} else {
// start your interval
setInterval(function () {
console.log('1s passed');
}, 1000);
}
});
}
EDIT:
Also why are you running your refreshKeyValues function once every second if it takes 10s to run? I would recommend you call it every 10s...
I have this piece of code:
var app = require('http').createServer(function(req, res){
console.log(req);
req.addListener('end', function () {
fileServer.serve(req, res);
});
});
var statics = require('node-static');
var fileServer = new statics.Server('./');
app.listen(1344, '127.0.0.1');
app.on('error', function(err){
console.log(err);
})
It was working just fine, till I made a couple of changes, node gives an error, and when I went back, that error wasn't there anymore, but instead of work like it was working before the end event is not being fired. So, anything inside req.addListener('end', function (){});is not called.
And even if I run another node.js that uses the same event, is not being fired either. So is like if the end event of the request is broken. But how can that be possible?
Is not the first time it happens. Last time I ended up re-installing node (after try lots of different things). I would prefer to find a solution, so I can understand the problem!
NOTE: The original code include socket.io and other kind of connections, but I've just pasted the piece of code were the app is stuck on.
It could also be useful to know how to debug the problem!
#InspiredJW should get credit for pointing this out, since I had forgotten about it, but undoubtedly your problem is because of the changes in the readable streams. In order for the end event to get called you either have to attach a listener to the data event, or you have to call stream.resume().
require('http').createServer(function(req, res){
req.addListener('end', function () {
// won't ever get called in node v0.10.3
});
});
require('http').createServer(function(req, res){
req.addListener('end', function () {
// will get called in node v0.10.3 because we called req.resume()
});
req.resume();
});
require('http').createServer(function(req, res){
req.on('data', function (chunk) { });
req.addListener('end', function () {
// also will get called because we attached a data event listener
});
});
http://nodejs.org/api/stream.html#stream_compatibility