I have a POST call to my nodeJS server that searches for some data on the mongo database and returns a CSV file with the requested data. The problem is that the data search and processing exceeds the 2 minute nodeJS default timeout.
On a diferent scenario y used:
res.writeHeader(200,'application/json');
res.write('starting fetch .... ');
to keep alive the request and prevent the timeout from the client by sending some res.write(''); from time to time.
Now Im using res.download() to download the resulting csv file, so not just sending JSON as response.
Tried to use this solution like this:
res.writeHeader(200,'application/json');
res.write('starting fetch .... ');
res.download()
But i get the "headers already sent" error.
Any idea on how to prevent the timeout until the data is processed and the file download is done ?
Thanks in advance
A few small things first of all, i think it's supposed to be res.writeHead() instead of res.writeHeader(). Secondly once you send the headers you can not send them again. You are getting the error because res.download() uses res.sendFile() which re-sends the headers. You can't use res.download() or res.sendFile() or res.send() because all of these three methods implicitly set the headers and send as part of the response. You'll need to use a combination of res.writeHead() ,res.write() and res.end(). I am just GET so i can test it from the browser, you can use post obviously.
router.get('/test', function (req,res,next) {
res.writeHead(200, {'Content-Type': 'application/json',
'Content-Disposition': 'attachment',
'filename':'package.json'
});
fs.readFile("package.json",function (err,data) {
if (err) {
throw err;
}else{
setTimeout(function () {
res.write(data);
res.end()
},200000);
}
});
function keepBusy(){
if(!res.finished){
res.writeProcessing();
setTimeout(keepBusy,1000)
}
}
setTimeout(keepBusy,1000);
});
This works fine for me. res.writeProcessing() sends the 102 status code. It is recommended for requests that last greater than 20 second but this blog post says that
102 has been left out more recent revisions of HTTP specifications
but it works fine for me in most browsers. If you have some control over the client as well there can be a much better and reliable solution.
Related
i am trying to stream a response. But i want to be able to read the response (and work with the data) while it is still being sent. I basically want to send multiple messages in one response.
It works internally in node.js, but when i tried to do the same thing in typescript it doesnt work anymore.
My attempt was to do the request via fetch in typescript and the response is coming from a node.js server by writing parts of the response on the response stream.
fetch('...', {
...
}).then((response => {
const reader = response.body.getReader();
reader.read().then(({done, value}) => {
if (done) {
return response;
}
console.log(String.fromCharCode.apply(null, value)); //just for testing purposes
})
}).then(...)...
On the Node.js side it basically looks like this:
// doing stuff with the request
response.write(first_message)
// do some more stuff
response.write(second_message)
// do even more stuff
response.end(last_message)
In Node.js, like i said, i can just read every message once its sent via res.on('data', ...), but the reader.read in typescript only triggers(?) once and that is when the whole response was sent.
Is there a way to make it work like i want, or do i have to look for another way?
I hope it is kinda understandable what i want to do, i noticed while writing this how much i struggled explaining this :D
I found the problem, and as usual it was sitting in front of the pc.
I forgot to write a header first, before writing the response.
I am using express (v:4.17.1) and here is my code snippet:
//upload.array is for multer npm file uplaoding....
//LIMIT is how many files you can upload via multer ....
app.post('/processFiles', upload.array('myFile', LIMIT), (req, res, next) => {
if (!validEnv) {
res.setHeader('X-Correlation-ID', corrId);
res.status(400).json({'error':'invalid env'});
return next(); //A
}
//checking how many file hvae been uplaoded
if (totalFiles > LIMIT) {
res.status(400).json({'error':'Too many files uploaded'});
return next(); //B
}
//Some xml parsing code and later it inserts into db
}
Few questions on the way I have added 'return next();' in the routes. Please note that I have not added any explicit middleware, just relying on what express is providing:
If I keep [A] or [B] as it is, then it gives a proper error message to the browser if the conditions get TRUE and returns the response.
But if I comment the line //A or //B, it gives the error response back to browser but also prints few error logs as mentioned below:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
I don't know why the above line is appearing when I comment the //A or //B (but wont complain if I keep as it is without commenting them) as the response was already out to the browser and the express will not execute any piece of code after the response has gone.
Please correct my understanding ?
Also, Is this the correct way to handle errors as I have coded? If No, what could be the proper way, please advice? Is there any way by which we can see the complete middleware execution in sequence until the last one via any debug flags
There are a several ExpressJS things to know that apply here:
next() tells Express to continue routing to other requests. Only call next() if you want other routes to continue to be able to process this request and send a response. Or the corallary, don't call next() if you've already sent a response.
You can only send one response for a given request. Once, you've called res.send() or .res.json() or any other way of sending a response, you should be completely done with your request processing and no other code should try to send a response.
res.send() and next() are not Javascript flow control. If you want to stop further processing of the request in your function, you need to code that with traditional Javascript flow control such as return, if/else, etc... to prevent the rest of your code from continuing to run or process the request.
To that end, your code should look like this:
//upload.array is for multer npm file uplaoding....
//LIMIT is how many files you can upload via multer ....
app.post('/processFiles', upload.array('myFile', LIMIT), (req, res, next) => {
if (!validEnv) {
res.setHeader('X-Correlation-ID', corrId);
res.status(400).json({'error':'invalid env'});
return;
}
//checking how many file hvae been uplaoded
if (totalFiles > LIMIT) {
res.status(400).json({'error':'Too many files uploaded'});
return;
}
// more code here that sends some other response
}
But if I comment the line //A or //B, it gives the error response back to browser but also prints few error logs as mentioned below: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
This error occurs when you attempt to send more than one response for the same request (e.g. call res.send(...) more than once. Your code should not allow that. Once you send a response, your code should use normal Javascript flow control (such as return or if/else) to avoid executing any more code that might send a response. Typically after you send a response, you are done processing the request and would just return from the request handler.
This is happening because of the //A. See, what is going on here in your code :
If you comment line //B it reaches the next if (totalFiles > LIMIT) which is sending another response but above the return next() is called. That is why it is generating error that once they are sent in if(!validEnv) how can Headers be sent again!
You have two if conditions that mean both might get executed, in both the conditions you are setting the headers and ending the request. Once the request is ended the headers cannot be set. So if you comment [A] which has the return statement and if the second condition is also satisfied, it will set the headers again after sending the request. so it has nothing to do with the next() function or express since there is no next middleware in the stack.
Also the error handling is looks fine. however you can use the node --inspect flag while starting the script and set breakpoint in your middleware to debug entire node code.
statusRouter.route('/')
.all(function(req,res,next){
res.writeHead(200, {'Content-Type': 'application/json'});
next();
})
.get(function(req, res, next) {
res.json({
name : "xyz"
});
});
This crashes with - Header cant be set after it has been sent.
But the catch is , this works :
statusRouter.route('/')
.all(function(req,res,next){
res.writeHead(200, {'Content-Type': 'text/plain'});
next();
})
.get(function(req, res, next) {
res.end("xyz");
});
NOTE : If I remove the writeHead function in the first case where I am sending JSON it starts working as well. Why does it not work when i do a writeHead on it ? This thing is driving me crazy can anyone explain why this happens ?
P.S I'm working with express-generated app with my own router.
Both res.writeHead() and res.end() are not implemented by Express, but by the Node.js http module.
Its documentation states, for res.end():
If data is specified, it is equivalent to calling response.write(data, encoding) followed by response.end(callback)
So res.end("xyz") is short for:
res.write("xyz");
res.end();
For res.write() the documentation states:
If this method is called and response.writeHead() has not been called, it will switch to implicit header mode and flush the implicit headers.
So res.end("xyz") is actually short for:
if (! res.headersSent) {
res.writeHead(...);
}
res.write("xyz");
res.end();
This means that it's perfectly okay to issue res.writeHead() in your own code, before using res.end(). Internally, the http module will know that you already flushed the headers, so it won't do it again (therefore preventing the error you're getting). However, you can't set different headers, or change existing ones, once writeHead() has been called.
Now, res.json() is another matter: this isn't part of the http module, but of Express itself. Because it's used to send JSON responses, it will set the content type header to application/json (so you don't have to).
But this will only work if the headers haven't yet been sent already: you can't set headers when they have already been sent out. That's why you're getting the error.
If you want to set particular headers in Express, use res.set().
Express evaluates its routes in the order they are added. This means that the route handler passed to .all() will be executed first. In this handler, a response header is written and then next() tells Express to continue iterating its list of routes for another possible matching route.
It then matches the .get() route handler which tries to send a JSON response. However, .json() implicitly sends a response header as well as writes the value passed as JSON to the response and ends the response. This is why you are seeing an error about writeHead() being called more than once.
I am new to nodejs so I have a basic question and this is my scanrio
I have a javascript client which is making a http request to a node server to read a value from the database.
Once the node server receives the request it makes a simple db call and returns the data to the client in the response, and this is where the problem is.
router.get('/state', function(req, res){
var result = dbServer.makeDBCall();//Before this line executes and returns the result the next line executes
res.send(result);
}
The database call from the node server is asynchronous, therefore before the result is returned the node server has already sent a blank response to the client. What is the standard/acceptable way of getting this achieved, I know I can block the node thread using async, but then the whole purpose of node is gone right?
It depends on what kind of database node module you are using.
Other than the standard callback approach, there are also the promise way. The pg-promise library is 1 of those kind.
See sample code:
this.databaseConnection.makeDBCall('your query...')
.then(function(dbResponse) {
// Parse the response to the format you want then...
res.send(result);
})
.catch(function(error) {
// Handle error
res.send(error.message);
});
#spdev : I saw 1 of your comments about you being worried about how Node actually knows who to reply the response to, especially when there are multiple requests.
This is a very good question, and to be honest with you - I don't know much about it as well.
In short the answer is yes, Node somehow handles this by creating a corresponding ServerResponse object when a HTTP request comes through. This object seems to have some smartness to tell the Nodejs network stack how to route itself back to the caller when it gets parsed as data packets.
I tried Googling a bit for an answer but didn't got too far. I hope the ServerResponseObject documentation can provide more insight for you. Share with me if you got an answer thanks!
https://nodejs.org/api/all.html#http_class_http_serverresponse
Try below code.
router.get('/state', function(req, res){
var result = dbServer.makeDBCall(function(err,result){
if(!err) {
res.send(result);
}
});
}
Hope this Help.
The dbServer.makeDBCall(); must have a callback that runs when the statement completes executing.
Something like -
dbServer.makeDBCall({query: 'args'}, function(err, result){
if (err) // handle error
res.send(result);
})
You return the response from db from that callback function.
Learn more about callback from here-
nodeJs callbacks simple example
https://docs.nodejitsu.com/articles/getting-started/control-flow/what-are-callbacks/
3 years ago I could do multiple res.send in express.js.
even write a setTimeout to show up a live output.
response.send('<script class="jsbin" src="http://code.jquery.com/jquery-1.7.1.min.js"></script>');
response.send('<html><body><input id="text_box" /><button>submit</button></body></html>');
var initJs = function() {
$('.button').click(function() {
$.post('/input', { input: $('#text_box').val() }, function() { alert('has send');});
});
}
response.send('<script>' + initJs + '</script>');
Now it will throw:
Error: Can't set headers after they are sent
I know nodejs and express have updated. Why can't do that now? Any other idea?
Found the solution but res.write is not in api reference http://expressjs.com/4x/api.html
Maybe you need: response.write
response.write("foo");
response.write("bar");
//...
response.end()
res.send implicitly calls res.write followed by res.end. If you call res.send multiple times, it will work the first time. However, since the first res.send call ends the response, you cannot add anything to the response.
response.send sends an entire HTTP response to the client, including headers and content, which is why you are unable to call it multiple times. In fact, it even ends the response, so there is no need to call response.end explicitly when using response.send.
It appears to me that you are attempting to use send like a buffer: writing to it with the intention to flush later. This is not how the method works, however; you need to build up your response in code and then make a single send call.
Unfortunately, I cannot speak to why or when this change was made, but I know that it has been like this at least since Express 3.
res.write immediately sends bytes to the client
I just wanted to make this point about res.write clearer.
It does not build up the reply and wait for res.end(). It just sends right away.
This means that the first time you call it, it will send the HTTP reply headers including the status in order to have a meaningful response. So if you want to set a status or custom header, you have to do it before that first call, much like with send().
Note that write() is not what you usually want to do in a simple web application. The browser getting the reply little by little increases the complexity of things, so you will only want to do it it if it is really needed.
Use res.locals to build the reply across middleware
This was my original use case, and res.locals fits well. I can just store data in an Array there, and then on the very last middleware join them up and do a final send to send everything at once, something like:
async (err, req, res, next) => {
res.locals.msg = ['Custom handler']
next(err)
},
async (err, req, res, next) => {
res.locals.msg.push('Custom handler 2')
res.status(500).send(res.locals.msg.join('\n'))
}