I am using express to interact with an IoT device. The device interacts correctly with a Django server I wrote, but not with the node.js server I'm writing now. It appears that device writes a first packet with the HTTP headers (including "Content-Length") and the final "\r\n\r\n", and then a second packet with the POST body. Express is calling my post handler with no body.
Like I said, Django handles this correctly. I THINK what is being done is legal HTTP/TCP and Express is not waiting for the full length of the "Content-Length" header when it should. Is this a part of Express' design? Can I turn it off so that it waits for the whole document? Is this a bug and I should use a different framework with this device?
The express framework is unique among http frameworks in that it gets smaller each release. One of the parts that was removed from the core is body parsing, so it now handles request bodies as base node.js does unless you add middleware.
In a standard post handler the request object is a stream containing the headers which allows you to react to the request before the request has even finished sending data.
It's easiest to see with an echo handler:
echo (req, res) {
req.pipe(res);
}
But most the time you want to process the post body as a whole
postBody (req, res) {
let body = '';
req.on('data', d => body += d.toString()); // it comes in as a buffer
req.on('error', e => { res.statusCode = 400; res.end('bad post data'); });
req.on('end', () => /*do something with the body */ res.end(body));
}
Which means the minimal middleware simply calls next after assigning the body to req.body. Though in practice you should use https://github.com/expressjs/body-parser or the like since it handles edge cases etc . . .
bodyMiddleware (req, res, next) {
let body = '';
req.on('data', d => body += d.toString()); // it comes in as a buffer
req.on('error', e => { res.statusCode = 400; res.end('bad post data'); });
req.on('end', () => { req.body = body; next(); });
}
Related
From inside my expressJS application I have to verify that a cookie token is valid with a back-end server. So the relevant code involved in this is as follows:
app.get('*', (req, res, next) => {
console.log('GET: ' + req.path);
// ...
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
// build request
const restReq = https.request(authServerOptions, result => {
console.log('back-end response' + result.statusCode);
result.on('data', data => {
next(); // token is good now proceed.
});
result.on('error', error => {
res.redirect('somewhere'); // token is bad or timeout
});
});
restReq.write(token);
restReq.end();
}
So the main get function sets the REST request in motion and then just returns without calling next() or anything.
Questions:
Is this the right code for doing this? What happens if the callbacks are never called?
Is the application blocked from processing other requests until the back-end server returns or times out?
If so is there some way of freeing up the thread to process more requests?
Thanks in advance for any help. I haven't found many examples for this code pattern so if there is one a link would be appreciated.
Yes, I think the general idea of your implementation is correct.
I would also suggest, as done in the comments, to use a client such as axios to handle the request in a less verbose and more comprehensive manner, which would leave your code looking something like this:
const axios = require('axios');
app.get('*', (req, res, next) => {
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
axios.post(url, payload, opts)
.then(response => next())
.catch(error => {
console.error(error);
res.redirect('somewhere');
});
});
A bit more to the point, but functionally almost equivalent to your implementation. The one thing you are missing is the onerror callback for your request object, which currently may fail and never return a response as you correctly suspected. You should add:
restReq.on('error', error => {
console.error(error);
res.redirect('somewhere');
});
On the same vein, it would probably be more fitting to call next on result end, instead of doing so while reading response data:
result.on('end', () => {
next();
});
Then you'd be covered to guarantee that a callback would be invoked.
Neither implementation blocks the processing of future requests, as the call to the token validation service is done asynchronously in both cases.
I'm trying to access the body parameters from the request using only the http internal module from NodeJS. I've looked up everywhere and I can't seem to find how to do that other than using express or some other library. I know that with express you do that by simply using req.body.myparameter. Can I do it only with http?
You will then need to handle data Buffer manually:
if (req.method === 'POST') {
let body = '';
req.on('data', chunk => { //start reading data
body += chunk.toString(); // convert Buffer to string
});
req.on('end', () => {
console.log(body); //logging submitted data
res.end('ok');
});
}
I have some files stored on a CDN server which is not to be directly accessed from client. So I proxy the requests via the public accessible server running ExpressJS and use request module to fetch the data server-side and return it in response.
It is working and in code looks something like this:
var request = require('request');
var app = express();
var internalUrl = 'https://my.storage-cdn.com/private/info/file.xml';
app.get('/somefile.xml', function (req, res) {
request(internalUrl).pipe(res);
});
The issues I faced with above method are:
the storage/cdn server appends some response headers of its own
which include some private information and as such can be a security
issue when exposed in response. And above method of piping the res
object to request doesn't remove those headers. It passes those
headers as is to response. I want to remove those headers.
I want to add some eTag and cache-control headers so the file could get cached
properly.
I have tried changing it to something like this:
app.get('/somefile.xml', function (req, res) {
request(internalUrl, function (err, response, body) {
if (!err && response.statusCode == 200) {
res.writeHead(200, {...}); // write custom headers I need
res.end(body);
}
});
});
This allows me to overwrite the headers to my liking, but in this method I have to wait for whole file to get downloaded on the server side first before I start sending the bytes in my response and with some files being as large as 1MB, it really affects the response time adversely.
So my question is - is there a way to not have to wait for whole file to download on server side before start sending response but still be able to manipulate response headers?
You can hook onto the 'response' event:
const SECRET_HEADERS = ['Set-Cookie', 'X-Special-Token']
app.get('/somefile.xml', function (req, res) {
request(internalUrl).on('response', function (response) {
SECRET_HEADERS.forEach(function (header) {
response.removeHeader(header)
})
}).pipe(res)
})
I'm using Node.js and connect to create a simple web server. I have something similar to the following code and I can't figure out how to access the actual request message body from the request object. I'm new to this so bear with me. I'm also taking out some of the stuff that's not necessary for the example.
function startServer(dir) {
var port = 8888,
svr = connect().use(connect.static(dir, {"maxAge" : 86400000}))
.use(connect.directory(dir))
/*
* Here, I call a custom function for when
* connect.static can't find the file.
*/
.use(custom);
http.createServer(svr).listen(port);
}
function custom(req, res) {
var message = /* the message body in the req object */;
// Do some stuff with message...
}
startServer('dirName');
Make sense? I've tried logging that object to the console and it is full of TONS of stuff. I can easily see headers in there plus the request URL and method. I just can't seem to isolate the actual message body.
You should include the connect.bodyParser middleware as well:
svr = connect().use(connect.static(dir, {"maxAge" : 86400000}))
.use(connect.directory(dir))
.use(connect.bodyParser())
.use(custom);
That will provide the parsed message body as req.body to your handler.
If you want the raw message body, you shouldn't use it but instead read the req stream yourself:
function custom(req, res) {
var chunks = [];
req.on('data', function(chunk) {
chunks.push(chunk);
});
req.on('end', function() {
var rawbody = Buffer.concat(chunks);
...do stuff...
// end the request properly
res.end();
});
}
if(req.method == "POST"){
var body = '';
req.on('data', function(data){
body += data;
});
}
Then body should contain your message if you posted correctly.
A better idea would be to use Express, then use the bodyparser middleware - which will give you this functionality out of the box without worrying about somebody hammering your server. The code above has NO functionality to worry about attacks - but it will get you started.
I am still a beginner in Node.js and I am trying to explore as much as I can.
I know that Express.js is a framework used by many people for creating websites in Node.js.
But without using Express.js, I know that it is it possible to read .html files using 'fs.readFile', and then "display" this .html file in the browser.
Is there a way to get user input (say a button click, or fill in a box) from this web page into Node.js? So far, I have not found any examples of this.
Yes, this is possible. Study how the connect bodyParser's urlencoded function works.
When a request comes in from the browser, node is going to represent this as a readable data stream. For web forms, the pattern will be:
Use the request's data and end events to buffer the chunks of data from the stream into a single string.
Parse that string appropriately given its data format. In the case of a web form, this will normally urlencoded (application/x-www-form-urlencoded) MIME type
.
var qs = require('qs'); //https://github.com/visionmedia/node-querystring
function handle(req, res) {
var buf = '';
req.setEncoding('utf8');
req.on('data', function(chunk){
//assemble the request from distinct chunks into a single string
buf += chunk
});
req.on('end', function(){
//OK, you have a usable string request body, parse it and handle it
try {
var formData = qs.parse(buf);
//Yay, it parsed. Now you have your form data
//depending on your form's html, you might have formData.email, for example
} catch (err){
//oops, respond with an error
}
});
}
Tutorial
Long story short:
http.createServer(function (req, res) {
var data = '';
req.on('data', function(chunk) {
console.log("Received body data:");
console.log(chunk);
data += chunk.toString();
});
req.on('end', function() {
console.log('Received Data: ', data);
res.end();
});
}