I have some files stored on a CDN server which is not to be directly accessed from client. So I proxy the requests via the public accessible server running ExpressJS and use request module to fetch the data server-side and return it in response.
It is working and in code looks something like this:
var request = require('request');
var app = express();
var internalUrl = 'https://my.storage-cdn.com/private/info/file.xml';
app.get('/somefile.xml', function (req, res) {
request(internalUrl).pipe(res);
});
The issues I faced with above method are:
the storage/cdn server appends some response headers of its own
which include some private information and as such can be a security
issue when exposed in response. And above method of piping the res
object to request doesn't remove those headers. It passes those
headers as is to response. I want to remove those headers.
I want to add some eTag and cache-control headers so the file could get cached
properly.
I have tried changing it to something like this:
app.get('/somefile.xml', function (req, res) {
request(internalUrl, function (err, response, body) {
if (!err && response.statusCode == 200) {
res.writeHead(200, {...}); // write custom headers I need
res.end(body);
}
});
});
This allows me to overwrite the headers to my liking, but in this method I have to wait for whole file to get downloaded on the server side first before I start sending the bytes in my response and with some files being as large as 1MB, it really affects the response time adversely.
So my question is - is there a way to not have to wait for whole file to download on server side before start sending response but still be able to manipulate response headers?
You can hook onto the 'response' event:
const SECRET_HEADERS = ['Set-Cookie', 'X-Special-Token']
app.get('/somefile.xml', function (req, res) {
request(internalUrl).on('response', function (response) {
SECRET_HEADERS.forEach(function (header) {
response.removeHeader(header)
})
}).pipe(res)
})
Related
I'm currently trying to log the full end response of every Express request with a simple middleware function like this:
module.exports = (req, res, next) => {
const startTime = new Date();
const oldEnd = res.end;
res.end = (chunks, encoding) => {
const responseTime = new Date() - startTime;
res.set('Server-Timing', `total;dur=${responseTime}`);
console.log(req.path, `Response Time: ${responseTime}`);
res.end = oldEnd;
res.end(chunks, encoding);
};
next();
}
This code works fine with normal Express endpoints but when I try to serve a static file like this: app.use('/static/path', express.static('path')) I get the following error:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
This happens because of the res.set for the server timing but this means express.static uses .end() twice? When I console.log in my middleware function it only gets called once.
I'm using NodeJS 10 and Express 4.16.4
Does anyone know how to solve this problem?
res.end is not called twice.
serve-static is streaming the file to the client and when the first chunk of the file is written to the stream, the headers will be sent. From the nodejs docs:
response.writeHead
If response.write() or response.end() are called before calling this, the implicit/mutable headers will be calculated and call this function.
So it is not possible to set headers after the stream has started to send data to the client. But it is possible to pass a setHeader function in the options to serve-static.
express.static('./public', {
setHeaders: (res, path, stat) => {
const responseTime = new Date() - res.locals.startTime;
res.set('Server-Timing', `total;dur=${responseTime}`);
},
});
However, since the headers are sent off at the start of the steam this is not accurate response time. More of a response time for the just the headers.
I am using express to interact with an IoT device. The device interacts correctly with a Django server I wrote, but not with the node.js server I'm writing now. It appears that device writes a first packet with the HTTP headers (including "Content-Length") and the final "\r\n\r\n", and then a second packet with the POST body. Express is calling my post handler with no body.
Like I said, Django handles this correctly. I THINK what is being done is legal HTTP/TCP and Express is not waiting for the full length of the "Content-Length" header when it should. Is this a part of Express' design? Can I turn it off so that it waits for the whole document? Is this a bug and I should use a different framework with this device?
The express framework is unique among http frameworks in that it gets smaller each release. One of the parts that was removed from the core is body parsing, so it now handles request bodies as base node.js does unless you add middleware.
In a standard post handler the request object is a stream containing the headers which allows you to react to the request before the request has even finished sending data.
It's easiest to see with an echo handler:
echo (req, res) {
req.pipe(res);
}
But most the time you want to process the post body as a whole
postBody (req, res) {
let body = '';
req.on('data', d => body += d.toString()); // it comes in as a buffer
req.on('error', e => { res.statusCode = 400; res.end('bad post data'); });
req.on('end', () => /*do something with the body */ res.end(body));
}
Which means the minimal middleware simply calls next after assigning the body to req.body. Though in practice you should use https://github.com/expressjs/body-parser or the like since it handles edge cases etc . . .
bodyMiddleware (req, res, next) {
let body = '';
req.on('data', d => body += d.toString()); // it comes in as a buffer
req.on('error', e => { res.statusCode = 400; res.end('bad post data'); });
req.on('end', () => { req.body = body; next(); });
}
I am trying to make a slack app and to complete Oauth2, I have to send the URI below and get a JSON response back in the body.
The problem is, every time I am trying to use the function request() in my app.get() function, ejs is always trying to go and get my views. Now I tried rendering my specific view for app.get() but then when I use request() again, ejs is again trying to get a view.
How can I redirect to another url from my app.get and receive the JSON. I can use req.redirect() but I don't know how to get the response back.
Please please help! Thanks
app.get('/', (req, res) =>{
var options = {
uri: 'https://slack.com/api/oauth.access code='+req.query.code+'&client_id='+client_id+'&client_secret='+client_secret,
method: 'GET'
}
request(options, (error, response, body) => {
var JSONresponse = JSON.parse(body)
if (!JSONresponse.ok){
console.log(JSONresponse)
res.send("Error encountered: \n"+JSON.stringify(JSONresponse)).status(200).end()
}else{
console.log(JSONresponse)
res.send("Success!")
}
})
})
so I have the following Scenario; I have a private API key that Angular will show in XHR request. To combat this, I decided to use Express as a proxy and make server side requests. However, I cannot seem to find documentation on how to make my own get requests.
Architecture:
Angular makes request to /api/external-api --> Express handles the route and makes request to externalURL with params in req.body.params and attaches API key from config.apiKey. The following is pseudocode to imitate what I'm trying to accomplish:
router.get('/external-api', (req, res) => {
externalRestGetRequest(externalURL, req.body.params, config.apiKey)
res.send({ /* get response here */})
}
You are half way there! You need something to make that request for you. Such as the npm library request.
In your route something like
var request = require('request');
router.get('/external-api', function(req, res){
request('http://www.google.com', function (error, response, body) {
console.log('error:', error); // Print the error if one occurred and handle it
console.log('statusCode:', response && response.statusCode); // Print the response status code if a response was received
res.send(body)
});
})
This allows you to make any type of request using whatever URL or API keys you need. However it's important to note you also need to handle any errors or bad response codes.
The accepted answer is good, but in case anyone comes across this question later, let's keep in mind that as of February, 2020, request is now deprecated.
So what can we do? We can use another library. I would suggest Axios.
Install it and do something like:
const axios = require('axios')
const url = "https://example.com"
const getData = async (url) => {
try {
const response = await axios.get(url)
const data = response.data
console.log(data)
} catch (error) {
console.log(error)
}
}
getData(url)
So, I am creating a REST API using node and I have to create a route.
Purpose of the route: Act as a proxy server and make a call to a different external website and return the response it gets to the original request.
So far, I have the following code and it works:
app.post('/v1/something/:_id/proxy',
function(req, res, next) {
// Basically make a request call to some external website and return
// the response I get from that as my own response
var opts = {/*json containing proper uri, mehtod and json*/}
request(opts, function (error, responseNS, b) {
if(error) return callback(error)
if(!responseNS) return callback(new Error('!response'))
return res.json(responseNS.body)
})
}
)
My question is, how can I stream this http response that I am getting from the external website. By that, I mean that I want to get the response as a stream and keep returning it as soon as it comes in chunks.
Is this possible?
You can pipe the incoming response from an external source straight to a response that your app sends to the browser, like this:
app.post('/v1/something/:_id/proxy',
function(req, res, next) {
// Basically make a request call to some external website and return
// the response I get from that as my own response
var opts = {/*json containing proper uri, mehtod and json*/}
request(opts, function (error, responseNS, b) {
if(error) return callback(error)
if(!responseNS) return callback(new Error('!response'))
return res.json(responseNS.body)
}).pipe(res);
});
With request you can directly pipe incoming response to either file stream, to other requests or to the response that your api sends to the browser. Like
function (req, res, next) {
request
.get('http://example.com/doodle.png')
.pipe(res)
}
Similary in your case just pipe to response.
app.post('/v1/something/:_id/proxy',
function(req, res, next) {
// Basically make a request call to some external website and return
// the response I get from that as my own response
var opts = {/*json containing proper uri, mehtod and json*/}
request(opts, function (error, responseNS, b) {
if(error) return callback(error)
if(!responseNS) return callback(new Error('!response'))
}).pipe(res);
}
)