I am using node express to process POST requests of heroku logging data with body data that is in the application/logplex-1 format (apparently syslog formatted).
In particular, I am using the body-parser module as middleware to parse the POST body.
It works OK to specify app.use(bodyParser.text({ type: 'application/logplex-1' })) to force body-parser to parse the body as text, but the text is just a big block of space-separated information without much structure other than that. Therefore I need to parse the body data further to find and extract what I want.
This is OK, but I'm wondering if there is, perhaps, a better way of parsing the logplex-1 body more directly into something more structured and easier to work with, like JSON. I'm not familiar with logplex-1 or the syslog format, and whether it does indeed have anything more useful structure/metadata in it than is apparent from the text block I'm currently getting.
Any ideas?
I have no experience with logplex or Heroku, but this seems to be working:
var syslogParser = require('glossy').Parse;
var express = require('express');
var app = express();
var server = app.listen(3012);
// Express allows arrays-of-middleware to act as a "single" middleware.
var logplexMiddleware = [
// First, read the message body into `req.body`, making sure it only
// accepts logplex "documents".
require('body-parser').text({ type: 'application/logplex-1' }),
// Next, split `req.body` into separate lines and parse each one using
// the `glossy` syslog parser.
function(req, res, next) {
req.body = (req.body || '').split(/\r*\n/).filter(function(line) {
// Make sure we only parse lines that aren't empty.
return line.length !== 0;
}).map(function(line) {
// glossy doesn't like octet counts to be prepended to the log lines,
// so remove those.
return syslogParser.parse(line.replace(/^\d+\s+/, ''));
});
next();
}
];
// Example endpoint:
app.post('/', logplexMiddleware, function(req, res) {
console.log(req.body);
return res.sendStatus(200);
});
It uses glossy to parse the syslog messages into Javascript objects.
If the amount of data being posted is considerable (>hundreds of K's), it might be better to implement a streaming solution as the code above will first read the entire message body into memory.
Related
In my express application I have a module called helpers thats is required in almost all my routes and modules. This module has a logger method that logs to fluentd (but that's unimportant). While building the data to log I'd like to add a unique identifier of the request, so that all the logs written for the same request have the same unique ID. Using a global var in the app entry point app.use doesn't work because this var would be overwritten every time a new request hits, so the global uuid will change would obviously change in case of high load or long running tasks. The res.locals is not available outside routing, so I can't use it for this matter. Is there a way to create a var that would be unique per request and available in every module or maybe a way to access the res.locals data outside routing? Thank you
EDIT
Maybe an example will help understand better the question.
Suppose I have a module called helpers.js like this:
let helpers = {};
helpers.log = (logData, logName) => {
fluentLogger.emit('', {
name: logName,
//uuid: the needed uuid,
message: logData
});
}
module.exports = helpers;
Now obviously I can do this in my app.js entry point:
app.use(function (req, res, next) {
res.locals.uuid = uuid.v4();
next();
});
and then in every loaded middleware module that requires helpers(adding a new param to the helpers.log method):
const helpers = require('helpers');
router.post('/', (req, res, next) => {
helpers.log('my log message', 'myLogName', res.locals.uuid);
next();
});
and this will normally work. But suppose a big or middle size project where there are hundreds of custom modules and models (not middlewares) and a module may require other modules that require other modules that require finally the helpers module. In this case I should pass the res.locals.uuid as a parameter to every method of every method so that I have it available in the logger method. Not a very good idea. Suppose I have a new module called dbmodel.js that is required in a middleware function:
const helpers = require('helpers');
let dbmodel = {};
dbmodel.getSomeData = (someParam) => {
//some logic
helpers.log('my log message', 'myLogName');
}
module.exports = dbmodel;
The dbmodel has no idea about the res.locals data if I don't pass it from the middleware, so the helpers.log method will also have no idea about this.
In PHP one would normally write a GLOBAL var in the application's entry point so a hypothetical logger function would have access to this global on every method request from whichever class of the application.
Hope this explanation will help :) Thank you
EDIT 2
The solution for this kind of problems is CLS. Thanks to #robertklep for the hint. A good slideshare explaining exactly the same problem (logger with unique ID) and explaining the CLS solutions can be found here: https://www.slideshare.net/isharabash/cls-and-asynclistener
I answered a very similar question here which will solve this problem.
I used to solve the problem the libraries node-uuid and continuation-local-storage. Take a look to the answer of this question and see if it helps:
NodeJS Express - Global Unique Request Id
And you want a bigger explanation, take a look here:
Express.js: Logging info with global unique request ID – Node.js
Yes you can do so by one method .
Every request comes to his routes pass that request inside the middleware.
Suppose you have
app.get('/', function(req, res) {
res.sendFile(path.join(public + "index.html"));
});
a request.
Place Middleware in it .and edit req field coming , in this way you will get the unique variable values for each request
check out this .
https://expressjs.com/en/guide/writing-middleware.html
Like this
var requestTime = function (req, res, next) {
req.requestTime = Date.now()
next()
}
app.use(requestTime)
app.get('/', function (req, res) {
var responseText = 'Hello World!<br>'
responseText += '<small>Requested at: ' + req.requestTime + '</small>'
res.send(responseText)
})
Here req.requestTime is unique for each request.
For exemple this is my server with a simple API :
var express = require('express');
var rzServer = express();
rzServer.use(bodyParser.urlencoded({extended:true}));
rzServer.use(bodyParser.json());
app.get('/url', function(req, res) {
console.log(req.query.data); // String
console.log(JSON.parse(req.query.date)); // Object
});
req.query.data is interpreted as a string but it's a JSON Object.
Is it possible with the body-parser package to parse the querystring ?
Thanks.
body-parser is a middleware to parse body (it's its name). If you want to parse the query string, so you need another middleware for that.
Another thing : GET requests normally don't take any JSON parameters (no body). If you need to send a true JSON, perhaps you're not using the good HTTP method. Try to use a POST request, or create a true query string (http://expressjs.com/fr/api.html#req.query).
I started by using express-form with my api (Express 3.3.8) in order to trim leading and trailing whitespace off of incoming POSTed fields.
However, I believe to use it I must include the form fields and rules in my middleware to my routes like so:
app.post('/api/test', form(
field("username").trim(),
field("password").trim(), function(req, res...
My question is, is there a way to do the trim automatically without specifying the fields individually? I know of the configuration option: autoTrim, but I think I still need to specify the fields on a per route/middleware basis, right? I tried leaving it out of the middleware, and just doing the form.configure({autoTrim:true}), but nothing changed with the req.body fields. Same as if I never included express-form at all.
I'm not committed to express-form. If there's another way already available to have Express always trim incoming req.body fields, please let me know.
app.use(postTrimmer);
function postTrimmer(req, res, next) {
if (req.method === 'POST') {
for (const [key, value] of Object.entries(req.body)) {
if (typeof(value) === 'string')
req.body[key] = value.trim();
}
}
next();
}
Don't forget to turn on the "body-parser" module before using postTrimmer middleware.
As it seems one must declare the fields individually using express-form, I decided to write my own whitespace trimming middleware for now as I couldn't find an existing simple solution. I use underscore.js, so you'll see its map function in use. You could otherwise do your own looping with the native Object.keys or similar. This completely rewrites all fields in req.body! Please note, this is a stop-gap for a greater validation issue. We're just doing this temporarily until we have time to clean up validation as a whole. Here's my code (put before app.use(app.router) of course):
var trimmer = function(req, res, next){
req.body = _.object(_.map(req.body, function (value, key) {
return [key, value.trim()];
}));
next();
}
app.use(trimmer);
I created a npm module for the very purpose. It provides you 4 middlewares that you can use to trim req.body, req.query, req.params or all three. It also provides a separate utility which you can use to trim any object.
This module works recursively so doesn't matter how nested the string is in object, It will be trimmed.
request_trimmer
npm i request_trimmer
const express = require('express');
const app = express();
const { trim_all,trim_body,trim_params,trim_util } = require('request_trimmer');
app.use(express.json());
#trim request body and query.
app.use(trim_all)
I am trying to build a server that can accept gzipped POST data with express. I think I could just write my own middleware to pipe the request stream to a zlib.createGunzip() stream. The question is, how can I achieve that, afterwards, the express.bodyParser() middleware is still able to parse my gunzipped POST data?
I tried to replace the original request stream methods by the ones of the zlib stream, but that just made the bodyParser return a "Bad Request" Error:
var express = require('express');
var app = express();
function gUnzip(req, res, next) {
var newReq;
if (req.headers['content-encoding'] === 'gzip') {
console.log("received gzipped body");
newReq = req.pipe(zlib.createGunzip());
Object.getOwnPropertyNames(newReq).forEach(function (p) {
req[p] = newReq[p];
});
}
next();
}
app.use(gUnzip);
app.use(express.bodyParser());
app.listen(8080);
Is there a way to make this work without rewriting the bodyParser() middleware within my own middleware?
EDIT:
This is the same question: Unzip POST body with node + express. But in the answer he just does in his own middleware what the express.bodyParser() should do, which is what I want to avoid. I am looking for a way to simply unzip the request data from the stream and then pass it to the bodyParser(), which expects a stream itself, as can be seen at http://www.senchalabs.org/connect/json.html.
compressed request bodies are generally not used because you can't negotiate content encodings between the client and server easily (there's another stackoverflow question about that i believe). most servers don't support compressed request bodies, and the only time you really need it is for APIs where the client will send large bodies.
body-parser, specifically raw-body, does not support it because the use-case is so minimal, though i've though about adding it. for now, you'll have to create your body-parser. fortunately, that's easy since you can just fork body-parser and leverage raw-body. the main code you'll add around https://github.com/expressjs/body-parser/blob/master/index.js#L80:
var zlib = require('zlib')
var stream
switch (req.headers['content-encoding'] || 'identity') {
case 'gzip':
stream = req.pipe(zlib.createGunzip())
break
case 'deflate':
stream = req.pipe(zlib.createInflate())
break
case 'identity':
break
default:
var err = new Error('encoding not supported')
err.status = 415
next(err)
return
}
getBody(stream || req, {
limit: '1mb',
// only check content-length if body is not encoded
length: !stream && req.headers['content-length'],
encoding: 'utf8'
}, function (err, buf) {
})
Have you tried using the built in compress middleware. It's documented in the expressjs reference documentation
app.use(express.compress());
Maybe you can find something useful here instead: Unzip POST body with node + express
Here's the basic setup. I'm trying to create a simple middleware component that would allow me to easily pass data from my route directly to my javascript in the client side. (Very similiar to the Gon gem in ruby). The way I'm doing it is by having a module that looks like this:
module.exports = function(){
return function(req,res,next){
var app = req.app;
if(typeof(app) == 'undefined'){
var err = new Error("The JShare module requires express");
next(err);
return;
}
res.jshare = {};
app.dynamicHelpers({
includeJShare: function(req,res){
if(typeof(res.jshare) === 'undefined'){
return "";
}
return function(){
return '<script type="text/javascript">window.jshare=' + JSON.stringify(res.jshare) + '</script>';
}
}
});
next();
};
}
Then, in my route I can do this:
exports.index = function(req, res){
res.jshare.person = {firstName : "Alex"};
res.render('index', { title: 'Express' })
};
Finally in the layout.jade:
!{includeJShare()}
What that does is in outputs a line of javascript on the client that creates the exact JSON object that was created server side.
Here's the question; it all works as expected, but being new to Express and Node.js in general, I was just curious if attaching properties onto the response object is OK, or is there something wrong with doing it that I'm simply overlooking? For some reason it doesn't pass my "smell test" but I'm not sure why.....
I know this is an old thread, but there is something else to add to this topic.
Express has a response.locals object which is meant for this purpose - extending the response from middleware to make it available to views.
You could add a property directly to the response object, and as #hasanyasin indicated, is how JavaScript is designed. But Express, more specifically, has a particular way they prefer we do it.
This may be new in express 3.x, not sure. Perhaps it didn't exist when this question was asked.
For details, see
http://expressjs.com/en/api.html#res.locals
There is also an app.locals for objects which don't vary from request to request (or response to response I suppose).
http://expressjs.com/en/api.html#app.locals
See also: req.locals vs. res.locals vs. res.data vs. req.data vs. app.locals in Express middleware
It is perfectly OK. It is how JavaScript is designed. Only thing you should be careful is to not accidentally overriding already existing properties or being overridden by others. To be safer, instead of adding everything directly to req/res objects, you might consider going a level deeper:
res.mydata={}
res.mydata.person= ...
Like that.
Use res.locals for including custom variables in your response object.