I'm trying to use mocha, request, and a SHA1 hash to write an integration test to confirm that the favicon being served from Express is the same as the one on the file system. I get two different hashes, and can't figure out why. Is it possible the encoding is changing?
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0" // Avoids DEPTH_ZERO_SELF_SIGNED_CERT error for self-signed certs
var request = require("request");
var crypto = require('crypto');
var fs = require('fs');
var favicon = crypto.createHash('sha1').update(fs.readFileSync(__dirname + '/../../public/img/favicon.ico')).digest('hex');
var app = require("../../server.js");
var expect = require('expect.js');
describe("Static tests", function () {
it("responds successfully", function (done) {
request.get("https://localhost:" + process.env.PORT + "/favicon.ico", function (err, res, body) {
// console.log(res)
expect(res.statusCode).to.be(200);
done();
});
});
it("serves out the file correctly", function (done) {
request.get("https://localhost:" + process.env.PORT + "/favicon.ico", function (err, res, body) {
// console.log(res)
expect(crypto.createHash('sha1').update(body).digest('hex')).to.be(favicon);
done();
});
});
});
Test 1 passes and then I get: "1) Server Static tests serves out the file Error: expected 'b09865f78dae40afa5f31503c208f5474e1d76a9' to equal 'd3e242e289b401c18d6e96526f586abf06385108'"
Any ideas why the same favicon might be hashing differently when being sent over HTTP versus read off the filesystem?
Assuming you are using the request module from npm, you should verify the type of the object you are receiving for the body argument is a Buffer. Looking at the source for the request module, I suspect you are getting a String instead. You might try doing the following when requiring request:
var request = require("request").defaults({ encoding: null });
That should tell the request module that you want a Buffer object by default.
Related
I am new and fail to make supertest work for me. I am wondering:
Why is the body undefined?
Is there a trick from the command line to show and inspect objects in the console?
Why doesn't the test logs "hello"?
"use strict";
const request = require('supertest');
const express = require('express');
const https = require('https');
const fs = require('fs');
const path = require('path');
const certPath = path.resolve(path.resolve(), './certs');
const app = express();
//This line is from the Node.js HTTPS documentation.
const options = {
key : fs.readFileSync(certPath+'/server-key.pem'),
cert : fs.readFileSync(certPath+'/server-crt.pem'),
ca : fs.readFileSync(certPath+'/ca-crt.pem')
};
// service
app.post('/failService', function(req, res) {
console.log('failService: '+req.body); // failService: undefined
res.send('hello');
});
describe('trial not working', function() {
it('responds with json', function(done) {
request(app)
.post('/failService')
.send({name: 'john'})
.set('Accept', /json/)
.expect(200)
.end(function(err, res) {
if (err) return done(err);
console.log('response: '+res.body); // response: [object Object]
done();
});
});
});
.... shows
$ mocha supertest.js
trial not working
failService: undefined
response: [object Object]
√ responds with json (125ms)
1 passing (171ms)
Please note that the certificates (not included) are self signed.
When isolating my code above I missed the options. As a result it does not use SSL. Sorry for that.
I fell back to starting the server and the use of a client in my test case. For that I had to fix:
CORS
The actual problem of this post by using a body parser
This is due to the self signed certificate.
i also faced similar issue there are two possible solutions
Create http server instead of https server for test environment
Replace supertest with superrequest npm package and set strictSsl as false.
I'm trying to download a file from another site from my Node app after an express HTTP get request and then return the file for download. I've tried multiple ways of getting the file, using pipe, blob, etc. but I'm grasping in the dark. The code might give you a bit more of an insight as to what I'm trying to achieve:
var router = require('express').Router();
var fs = require('fs');
var http = require('http');
router.get('/download/:file', function (req, res, next) {
http.get('http://anothersite/' + req.params.file, function(response) {
res.setHeader('Content-disposition', 'attachment; filename=' + req.params.file);
res.setHeader('Content-type', 'application/octet-stream');
res.download(fs.createWriteStream(req.params.file).pipe(response));
});
});
This gives me an error "Cannot pipe. Not Readable". The file itself is not a regular file format (it's a file from our customized software with its own extension).
What am I missing?
For one you need to use readable stream here, not writable
The Express res object is derived from node's http.ServerResponse, which is itself implementing node's WritableStream interface. See docs here and here.
Since that is the case, I think you can use response argument passed to your callback directly, since that is already a ReadableStream (see here). Try using readable stream like this:
router.get('/download/:file', function (req, res, next) {
http.get('http://anothersite/' + req.params.file, function(response) {
res.setHeader('Content-disposition', 'attachment; filename=' + req.params.file);
res.setHeader('Content-type', 'application/octet-stream');
response.pipe(res); // <-- change here
});
});
This code is working, with node v5.0.0 and latest chrome:
const express = require('express');
const app = express();
const http = require('http');
app.get('/', (req, res) => {
http.get('http://www.fillmurray.com/200/300', (response) => {
res.setHeader('Content-disposition', 'attachment; filename=' + 'hello.jpg');
res.setHeader('Content-type', 'application/octet-stream');
response.pipe(res)
});
});
app.listen(3001, () => console.log(('listening :)')))
you can use request library as:
request('http://anothersite/').pipe(fs.createWriteStream('filename.extension'))
Update:
If you are willing to do it by http you can create a write stream to as follow. After saving the file successfully I think you can do res.download() correctly.
var http = require('http')
http.get('http://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png').on('response', function (response) {
var writeStream = fs.createWriteStream('output.png');
response.pipe(writeStream);
response.on('end', function () {
console.log('stream completed!')
});
// This is here incase any errors occur
writeStream.on('error', function (err) {
console.log(err);
});
});
Is it possible to define a unique request Id that is included in each log statement without handing the logger to each method/function call?
Technologies in use: NodeJS, Express, Winston
Edited
Finally, I have created a library that makes all the work.
https://github.com/davicente/express-logger-unique-req-id
It is a wrapper of Winston library, so you can use it the same way.
Let me know if it helps you
We had this same problem in several projects, and I couldn't finde any complete solution for this question. We are using same technologies (Node.js, Express.js and Winston for logs)
I found a solution to this using a couple of libraries and wrapping Winston library:
- node-uuid for creating unique identificators for each request
- continuation-local-storage for sharing this IDs among different modules without sending req object in all the calls.
First I need to create and set the unique identificator with each request. I do it in the middleware:
var uuid = require('node-uuid');
var createNamespace = require('continuation-local-storage').createNamespace;
var myRequest = createNamespace('my request');
// Run the context for each request. Assign a unique identifier to each request
app.use(function(req, res, next) {
myRequest.run(function() {
myRequest.set('reqId', uuid.v1());
next();
});
});
After that I had to wrap Winston library, recovering the id from the context and adding to the message of the log:
var winston = require('winston');
var getNamespace = require('continuation-local-storage').getNamespace;
// Wrap Winston logger to print reqId in each log
var formatMessage = function(message) {
var myRequest = getNamespace('my request');
message = myRequest && myRequest.get('reqId') ? message + " reqId: " + myRequest.get('reqId') : message;
return message;
};
var logger = {
log: function(level, message) {
winstonLogger.log(level, formatMessage(message));
},
error: function(message) {
winstonLogger.error(formatMessage(message));
},
warn: function(message) {
winstonLogger.warn(formatMessage(message));
},
verbose: function(message) {
winstonLogger.verbose(formatMessage(message));
},
info: function(message) {
winstonLogger.info(formatMessage(message));
},
debug: function(message) {
winstonLogger.debug(formatMessage(message));
},
silly: function(message) {
winstonLogger.silly(formatMessage(message));
}
};
module.exports = logger;
I think it was a little bit complex, so I decided to write it down in a post. You can get more information from there: Express.js: Logging info with global unique request ID – Node.js
I hope this helps with your problem.
This answer has a problem: the counter goes back to 0 every time the node process is restarted. Turns out there is fairly simple to work around. You simply add an express middleware that tags each request called with a UUID using the uuid package.
For uuid Pre-2.0
const uuid = require('uuid');
const app = express();
app.use(function (req, next) {
req.id = uuid.v4();
next();
});
For uuid 3.0+
const uuidv4 = require('uuid/v4');
const app = express();
app.use(function (req, next) {
req.id = uuidv4();
next();
});
At the very beginning of your request handling add something like the following (or put it in its own file):
var makeID = (function() {
var index = 0;
return function() {
return index++;
}
})();
app.use(function(req, res, next) {
req.id = makeID()
next()
})
This will give every request a unique (sequential) id. Do not allow people to identify themselves with it, only use it internally!
When logging in Winston, you can enter metadata, to be displayed after the message (in the form ${name}=${value}), which looks like this
app.use(function(req, res) {
winston.log('info', 'Test Log Message', { id: req.id });
res.end("Done.")
});
Hope this is helpful.
I am writing a custom middleware that generates a cryptographic signature of every request (it is very similiar to the authentication mechanism used by AWS API v4). In order for this signature to be correctly generated, I must fetch the entire raw body of the HTTP request.
I am also using BodyParser, which is registered after my custom middleware.
My custom middleware can be represented like this:
// libs/simplifiedSignatureCheckerMiddleware.js
module.exports = function (req, res, next){
// simple and fast hashing stuff
var payload = '';
req.on('data', function(chunk) { payload += chunk }, null);
req.on('end', function(){
// hmac stuff
console.log(payload);
var ok = true; // ...
if(ok)
next();
else
next("Bad")
});
}
This is how I use it on the server.
// simpleServer.js
// BASE SETUP
// =============================================================================
var express = require('express');
var app = express();
var bodyParser = require('body-parser');
var jsonStream = require('express-jsonstream');
var nconf = require('nconf');
var https = require('https');
var fs = require('fs');
// load configurations
nconf.argv().env();
nconf.file({file: 'config.json'});
app.use(require('./libs/simplifiedSignatureCheckerMiddleware'));
// configure app to use bodyParser()
// this will let us get the data from a POST
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({extended: true}));
app.use(jsonStream());
// ROUTES FOR API
// =============================================================================
var router = express.Router();
router.post('/api/', function (req, res) {
var param1 = req.body.param1 || "";
var param2 = req.body.param2 || "";
res.json({message: 'welcome', one: param1, two: param2 });
});
// REGISTER ROUTES
app.use(router);
// START THE SERVER
// =============================================================================
https.createServer({
key: fs.readFileSync('./key.pem'),
cert: fs.readFileSync('./cert.pem')
}, app).listen(nconf.get('http:port'));
console.log("APIs listening on port " + nconf.get('http:port'));
As you can verify, the raw body is written successfully to the console by the middleware, BUT the request will never be processed by the registered route and the connection hangs forever.
Do you have any clue on how to solve this problem?
Thanks in advance.
Ok, since the only feasible way to solve this problem seems to be by modifying the original source code of bodyParser, I have forked it.
https://github.com/emanuelecasadio/body-parser-rawbody
This fork exposes the raw body of the request as a field named rawBody. As you can see, there is only ONE extra line of code.
You can install it by using npm install body-parser-rawbody.
EDIT
Another option is to use the bodyParser like this, as noted by dougwilson here: https://github.com/expressjs/body-parser/issues/83#issuecomment-80784100
app.use(bodyParser.json({verify:function(req,res,buf){req.rawBody=buf}}))
I haven't personally tried this option and I do not know if it works.
I am building a cross system admin app, which will be used as an admin tool for multiple backend systems. The app is built on top of Mean.js.
I have setup a /proxy route using "express-http-proxy" to send all sub-routes to their respective backend system endpoints. However, I need to have each request authenticated within my admin app and then decorated with the targeted backendSystem credentials before the "express-http-proxy" can continue. Here's an example of my /proxy route...
app.use('/proxy', users.requiresLogin, expressHttpProxy(config.backendSystem.host, {
forwardPath: function (req) {
return '/1.0' + require('url').parse(req.url).path;
},
decorateRequest: function (req) {
req.headers['content-type'] = 'application/json';
req.headers['backend-system-id'] = config.backendSystem.id;
req.headers['backend-system-key'] = config.backendSystem.key;
return req;
}
}));
NOTE:
Currently the backendSystem credentials are stored based on the environment my admin app is ran in. However, in the future the backendSystem credentials will be specified by the user, and this /proxy route will differently than what is currently shown.
THE ISSUE:
Proxy routes that require data within the request body don't work.
e.g. POST /comments {"user": user_id, "text": "rabble rabble rabble"}
WHAT I'VE FOUND:
bodyParser.json() and "express-https-proxy" don't play nice. I've confirmed this by removing bodyParser.json() from express.js.
However, this isn't a full solution since almost all of my other routes need bodyParser.json, e.g. /auth/signin.
Does anyone have a clean way that I can make a route exception for my /proxy route so that bodyParser.json won't be called for it?
As far as I understand, the root of problem is so:
if you were reading a POST request by pure node, you should be using a code like this
if (req.method == 'POST') {
console.log("POST");
var body = '';
req.on('data', function (data) {
body += data;
console.log("Partial body: " + body);
});
req.on('end', function () {
console.log("Body: " + body);
});
res.writeHead(200, {'Content-Type': 'text/html'});
res.end('post received');
}
in other words, you need to use the req.on('data') & req.on('end') events.
but the problem is,that you can use this code only once. after the 'end' is called, the request is consumed.
so then you use bodyParser , it consumes the request, and the proxy have nothing to do with it.
actually, in my opinion, the proxy wait for the 'data' event to appear , but it will newer happen, so the code halts.
The solution:
you need to 're-enable' the events. I used this code and it works for me.
var express = require('express');
var bodyParser = require('body-parser');
var http = require('http');
//call for proxy package
var devRest = require('dev-rest-proxy');
//init express (as default)
var users = require('./routes/users');
var app = express();
app.use(bodyParser.json());
//set the proxy listening port
app.set('port', 8080);
//process the POST request
app.post('/users/*', function(req, res) {
//just print the body. do some logic with it
console.log("req.body: ",req.body);
//remove listeners set by bodyParser
req.removeAllListeners('data');
req.removeAllListeners('end');
//add new listeners for the proxy to use
process.nextTick(function () {
if(req.body) {
req.emit('data', JSON.stringify(req.body));
}
req.emit('end');
});
//forward the request to another server
devRest.proxy(req,res, 'localhost', 3000);
});
//start the proxy server
http.createServer(app).listen(app.get('port'), function(){
console.log("Express server listening on port " + app.get('port'));
});
module.exports = app;
the solution found on schumacher-m post (github of nodejitsu)
I was able to resolve my issue by adding a regex that excluded my /proxy route to where bodyParser.json was being added within express.js. I found that from this answer
While this approach doesn't scale well, it solved my immediate issue.
I get it works by converting the data into query string using 3rd party query-string as follows:
proxyReqBodyDecorator: function(bodyContent, srcReq) {
return (queryString.stringify(bodyContent));
}
Have tried JSON.stringify but not working, need the data in the following format
array_field=val1&array_field=val2&array_field=val3......
To modify the request body, do this with the latest express-http-proxy v1.6.2:
const express = require('express');
const proxy = require('express-http-proxy');
const bodyParser = require('body-parser');
const conf = {
proxyHost: 'some.example.net:9200',
proxyOptions: {
proxyReqBodyDecorator: modifyRequestBody,
preserveHostHdr: true,
parseReqBody: true
},
port: 8073
};
var app = express();
app.use('/proxy', proxy(conf.proxyHost, conf.proxyOptions));
function modifyRequestBody(body, srcReq) {
if(srcReq.method.match(/^(GET|POST)$/i)) {
try {
// convert buffer to string, then to object
var str = Buffer.from(body).toString('utf-8');
var reqBody = JSON.parse(str);
if(someCondition)) {
reqBody.addStuff = 'whatever';
body = reqBody; // return modified body as object
}
} catch(error) {
console.log('- error: ' + JSON.stringify(error));
}
}
return body; // return original buffer, or modified object
}
app.listen(conf.port, function () {
log('app listening on port ' + conf.port);
});
You can fill the proxyReq.bodyContent inside the decorateRequest method with the JSON-ed data from originalReq.body to be correctly POST'ed:
app.use('/proxy', users.requiresLogin, expressHttpProxy(config.backendSystem.host, {
...
...
decorateRequest: function (proxyReq, originalReq) {
...
...
if (originalReq.body) {
proxyReq.bodyContent = JSON.stringify(originalReq.body);
}
return proxyReq;
}
...
...
}));