I'm currently trying to log the full end response of every Express request with a simple middleware function like this:
module.exports = (req, res, next) => {
const startTime = new Date();
const oldEnd = res.end;
res.end = (chunks, encoding) => {
const responseTime = new Date() - startTime;
res.set('Server-Timing', `total;dur=${responseTime}`);
console.log(req.path, `Response Time: ${responseTime}`);
res.end = oldEnd;
res.end(chunks, encoding);
};
next();
}
This code works fine with normal Express endpoints but when I try to serve a static file like this: app.use('/static/path', express.static('path')) I get the following error:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
This happens because of the res.set for the server timing but this means express.static uses .end() twice? When I console.log in my middleware function it only gets called once.
I'm using NodeJS 10 and Express 4.16.4
Does anyone know how to solve this problem?
res.end is not called twice.
serve-static is streaming the file to the client and when the first chunk of the file is written to the stream, the headers will be sent. From the nodejs docs:
response.writeHead
If response.write() or response.end() are called before calling this, the implicit/mutable headers will be calculated and call this function.
So it is not possible to set headers after the stream has started to send data to the client. But it is possible to pass a setHeader function in the options to serve-static.
express.static('./public', {
setHeaders: (res, path, stat) => {
const responseTime = new Date() - res.locals.startTime;
res.set('Server-Timing', `total;dur=${responseTime}`);
},
});
However, since the headers are sent off at the start of the steam this is not accurate response time. More of a response time for the just the headers.
Related
so I have the following Scenario; I have a private API key that Angular will show in XHR request. To combat this, I decided to use Express as a proxy and make server side requests. However, I cannot seem to find documentation on how to make my own get requests.
Architecture:
Angular makes request to /api/external-api --> Express handles the route and makes request to externalURL with params in req.body.params and attaches API key from config.apiKey. The following is pseudocode to imitate what I'm trying to accomplish:
router.get('/external-api', (req, res) => {
externalRestGetRequest(externalURL, req.body.params, config.apiKey)
res.send({ /* get response here */})
}
You are half way there! You need something to make that request for you. Such as the npm library request.
In your route something like
var request = require('request');
router.get('/external-api', function(req, res){
request('http://www.google.com', function (error, response, body) {
console.log('error:', error); // Print the error if one occurred and handle it
console.log('statusCode:', response && response.statusCode); // Print the response status code if a response was received
res.send(body)
});
})
This allows you to make any type of request using whatever URL or API keys you need. However it's important to note you also need to handle any errors or bad response codes.
The accepted answer is good, but in case anyone comes across this question later, let's keep in mind that as of February, 2020, request is now deprecated.
So what can we do? We can use another library. I would suggest Axios.
Install it and do something like:
const axios = require('axios')
const url = "https://example.com"
const getData = async (url) => {
try {
const response = await axios.get(url)
const data = response.data
console.log(data)
} catch (error) {
console.log(error)
}
}
getData(url)
I have some files stored on a CDN server which is not to be directly accessed from client. So I proxy the requests via the public accessible server running ExpressJS and use request module to fetch the data server-side and return it in response.
It is working and in code looks something like this:
var request = require('request');
var app = express();
var internalUrl = 'https://my.storage-cdn.com/private/info/file.xml';
app.get('/somefile.xml', function (req, res) {
request(internalUrl).pipe(res);
});
The issues I faced with above method are:
the storage/cdn server appends some response headers of its own
which include some private information and as such can be a security
issue when exposed in response. And above method of piping the res
object to request doesn't remove those headers. It passes those
headers as is to response. I want to remove those headers.
I want to add some eTag and cache-control headers so the file could get cached
properly.
I have tried changing it to something like this:
app.get('/somefile.xml', function (req, res) {
request(internalUrl, function (err, response, body) {
if (!err && response.statusCode == 200) {
res.writeHead(200, {...}); // write custom headers I need
res.end(body);
}
});
});
This allows me to overwrite the headers to my liking, but in this method I have to wait for whole file to get downloaded on the server side first before I start sending the bytes in my response and with some files being as large as 1MB, it really affects the response time adversely.
So my question is - is there a way to not have to wait for whole file to download on server side before start sending response but still be able to manipulate response headers?
You can hook onto the 'response' event:
const SECRET_HEADERS = ['Set-Cookie', 'X-Special-Token']
app.get('/somefile.xml', function (req, res) {
request(internalUrl).on('response', function (response) {
SECRET_HEADERS.forEach(function (header) {
response.removeHeader(header)
})
}).pipe(res)
})
I'm trying to mock request and response objects for my node/express handlers. I've tried a few mocking libraries and have run into issues with API compatibility, which has made them too unreliable for testing purposes.
What I would like to do is create the raw request and response objects myself and direct the output somewhere other than a live connection.
Here's what I have so far:
env.mockReq = function(o){
o = o || {};
o.hostname = 'www.tenor.co';
o.protocol = 'https';
o.path = o.url;
o.createConnection = function(){
console.log('mockReq createConnection');
};
var req = new http.ClientRequest(o);
req.url = o.url;
req.method = o.method;
req.headers = o.headers || {};
return req;
};
env.mockRes = function(o){
var res = new http.ServerResponse({
createConnection: function(){
console.log('mockRes createConnection');
}
});
return res;
};
Here's some test code:
var req = env.mockReq({method: 'GET', url: '/'});
var res = env.mockRes();
res.on('end', function(arguments){
expect(this.statusCode).toBe(200);
expect(this._getData().substr(-7)).toEqual('</html>');
scope.done();
done();
});
// my express app
app.handle(req, res);
My handler is piping a stream data source to the response:
stream.pipe(response);
It works fine when I load the requests in a browser, but my test times out because the response end event never gets fired. I should note that I have logging statements in my handler that's under test and it completes right to the end.
To complicate matters, I'm using nock to mock out some API requests. I had to add the following to prevent an error:
// Prevents "Error: Protocol "https" not supported. Expected "http:""
nock('http://www.example.com')
.persist()
.get('/non-existant-path')
.reply(function(uri, requestBody) {
console.log('nock path:', this.req.path);
return ''
});
That nock callback never actually gets called though. But without this code I get that error, even if I don't use https. The live version of my site redirects all traffic to https, so maybe a live connection is being made, but then why is my handler executing?
I'm trying to use streams to send data to the browser with Hapi, but can't figure our how. Specifically I am using the request module. According to the docs the reply object accepts a stream so I have tried:
reply(request.get('https://google.com'));
The throws an error. In the docs it says the stream object must be compatible with streams2, so then I tried:
reply(streams2(request.get('https://google.com')));
Now that does not throw a server side error, but in the browser the request never loads (using chrome).
I then tried this:
var stream = request.get('https://google.com');
stream.on('data', data => console.log(data));
reply(streams2(stream));
And in the console data was outputted, so I know the stream is not the issue, but rather Hapi. How can I get streaming in Hapi to work?
Try using Readable.wrap:
var Readable = require('stream').Readable;
...
function (request, reply) {
var s = Request('http://www.google.com');
reply(new Readable().wrap(s));
}
Tested using Node 0.10.x and hapi 8.x.x. In my code example Request is the node-request module and request is the incoming hapi request object.
UPDATE
Another possible solution would be to listen for the 'response' event from Request and then reply with the http.IncomingMessage which is a proper read stream.
function (request, reply) {
Request('http://www.google.com')
.on('response', function (response) {
reply(response);
});
}
This requires fewer steps and also allows the developer to attach user defined properties to the stream before transmission. This can be useful in setting status codes other than 200.
2020
I found it !! the problem was the gzip compression
to disable it just for event-stream you need provide the next config to Happi server
const server = Hapi.server({
port: 3000,
...
mime:{
override:{
'text/event-stream':{
compressible: false
}
}
}
});
in the handler I use axios because it support the new stream 2 protocol
async function handler(req, h) {
const response = await axios({
url: `http://some/url`,
headers: req.headers,
responseType: 'stream'
});
return response.data.on('data',function (chunk) {
console.log(chunk.toString());
})
/* Another option with h2o2, not fully checked */
// return h.proxy({
// passThrough:true,
// localStatePassThrough:true,
// uri:`http://some/url`
// });
};
I'm using Node.js and connect to create a simple web server. I have something similar to the following code and I can't figure out how to access the actual request message body from the request object. I'm new to this so bear with me. I'm also taking out some of the stuff that's not necessary for the example.
function startServer(dir) {
var port = 8888,
svr = connect().use(connect.static(dir, {"maxAge" : 86400000}))
.use(connect.directory(dir))
/*
* Here, I call a custom function for when
* connect.static can't find the file.
*/
.use(custom);
http.createServer(svr).listen(port);
}
function custom(req, res) {
var message = /* the message body in the req object */;
// Do some stuff with message...
}
startServer('dirName');
Make sense? I've tried logging that object to the console and it is full of TONS of stuff. I can easily see headers in there plus the request URL and method. I just can't seem to isolate the actual message body.
You should include the connect.bodyParser middleware as well:
svr = connect().use(connect.static(dir, {"maxAge" : 86400000}))
.use(connect.directory(dir))
.use(connect.bodyParser())
.use(custom);
That will provide the parsed message body as req.body to your handler.
If you want the raw message body, you shouldn't use it but instead read the req stream yourself:
function custom(req, res) {
var chunks = [];
req.on('data', function(chunk) {
chunks.push(chunk);
});
req.on('end', function() {
var rawbody = Buffer.concat(chunks);
...do stuff...
// end the request properly
res.end();
});
}
if(req.method == "POST"){
var body = '';
req.on('data', function(data){
body += data;
});
}
Then body should contain your message if you posted correctly.
A better idea would be to use Express, then use the bodyparser middleware - which will give you this functionality out of the box without worrying about somebody hammering your server. The code above has NO functionality to worry about attacks - but it will get you started.