I am using node js as reverse proxy mostly using http and http-proxy module. While sending the request to to nodejs to redirect to one of my application, i have to pass request headers which will all be in upper case. However, nodejs or rather http is converting all upper case to lower case, because of which one of the validation of my application is failing.
My code snippet is:
http.createServer(function (request, response) {
var redirection = 'http://localhost:8000';
var path = url.parse(request.url).path;
switch (path) {
case '/health':
proxy.web(request, response, { target: redirection });
break;
}).listen(8080);
Request headers passed are:
curl -H "X-AUTH: PBxqcEm5sU743Cpk" -X GET http://localhost:8080/health
Now what is happening is, header "X-AUTH" is getting transformed into "x-auth" and my application is not able to validate it. In my application the header matching is case sensitive.
The request headers printed from node js request object are:
{ host: 'localhost:8080',
'user-agent': 'curl/7.47.1',
accept: '*/*',
'x-auth': 'PBxqcEm5sU743Cpk' }
My requirement is to retain the upper case of the header passed in request so that my application can validate and authorize it.
Please let me know if there is any way to achieve this
Thanks a lot
FWIW HTTP header field names are case-insensitive so the case really should not matter.
However, node does provide access to the raw headers (including duplicates) via req.rawHeaders. Since req.rawHeaders is an array (format is [name1, value1, name2, value2, ...]), you will need to iterate over it to find the header(s) you are looking for.
Related
I am new to the whole backend stuff I understood that both bodyparser and express.json() will parse the incoming request(body from the client) into the request object.
But what happens if I do not parse the incoming request from the client ?
without middleware parsing your requests, your req.body will not be populated. You will then need to manually go research on the req variable and find out how to get the values you want.
Your bodyParser acts as an interpreter, transforming http request, in to an easily accessible format base on your needs.
You may read more on HTTP request here ( You can even write your own http server )
https://nodejs.org/api/http.html#http_class_http_incomingmessage
You will just lose the data, and request.body field will be empty.
Though the data is still sent to you, so it is transferred to the server, but you have not processed it so you won't have access to the data.
You can parse it yourself, by the way. The request is a Readable stream, so you can listen data and end events to collect and then parse the data.
You shall receive what you asked for in scenarios where you do not convert the data you get the raw data that looks somewhat like this username=scott&password=secret&website=stackabuse.com, Now this ain't that bad but you will manually have to filter out which is params, what is a query and inside of those 2 where is the data..
unless it is a project requirement all that heavy lifting is taken care of by express and you get a nicely formatted object looking like this
{
username: 'scott',
password: 'secret',
website: 'stackabuse.com'
}
For Situation where you DO need to use the raw data express gives you a convenient way of accessing that as well all you need to do is use this line of code
express.raw( [options] ) along with express.json( [options] )
I want to do some operations with response from python requests library. After I use below function;
response = requests.get(f'{AUTHORIZE_URL}?client_id={CLIENT_ID}&response_type=code&state={STATE}&redirect_uri={REDIRECT_URI}')
I need to get an URL something like this in return;
http://127.0.0.1:8000/products/auth/?state=2b33fdd45jbevd6nam&code=MGY1MTMyNWY0YjQ0MzEwNmMxMjY2ZjcwMWE2MWY5ZDE5MzJlMjA1YjdkNWExNGRhYjIzOGI5NzQ5OWZkNTA5NA
While doing it, it will be easier to use JSON in order to get state and code values from URL but I cannot use it because I think the content type does not allow this.
See this for Content-Type explanation: Content-Type
In short the "content-type" in the headers of response got by using requests.get tells you what kind of the content server did send, in your case you'we got a response in the form of the HTML (like .html document) and you can read that response with response.text, if the "content-type" is "application/json" then you can read it as JSON like this response.json().
I see that you use some local server, your local server should send in headers "Content-Type": "application/json" and then you should be able to read JSON from response like this (you need to send JSON not hmtl or text from server):
targetURL = 'http://127.0.0.1:8000/products/auth/?state=2b33fdd45jbevd6nam&code=MGY1MTMyNWY0YjQ0MzEwNmMxMjY2ZjcwMWE2MWY5ZDE5MzJlMjA1YjdkNWExNGRhYjIzOGI5NzQ5OWZkNTA5NA'
response.get(targetURL).json()
I try to make an REST-API, but varnish returns always the first called response and I have no idea why.
If I open a page with a Browser, Varnish returns HTML -> is OK.
If I curl the same page curl -i https://example.com -H "Accept: application/json" Varnish also returns HTML -> which is False.
As I see, Varnish always returns the first cached item, If this is JSON varnish returns JSON, if this is HTML Varnish returns HTML.
Without Varnish everything works like expected.
If you're serving different content type on the same URL, you you might want to tell Varnish to partition cache accordingly.
In fact, Varnish doesn't do much special about it, and it behaves like other proxies would. If they see a URL without information specifying how a resource's cache should partition, then no matter if it is a JSON or a regular request, the first request will be cached and served the same irrespective of request type.
So you need to tell Varnish how to partition cache for a resource.
The "Vary" header
The most straightforward and "HTTP" compatible way for other proxies in the wild, is Vary response header.
It tells the proxy cache (Varnish in this case), to partition, vary cache for a resource based on a header value coming from a client.
E.g. client sends header X: some-value and your app sends header Vary: X is what it takes to make the cache different between different value of X.
For Varnish 3, there is an example with Accept-Encoding.
The article details an implementation challenge with Vary - different clients may be sending quite different values for varied header thus resulting in severely partitioned cache. So you typically want to normalize the varying header's value, to a set of known, expected values.
In your case you want to Vary (and normalize) the Accept header. So something along the lines of (in vcl_recv procedure):
if (req.http.Accept) {
if (req.http.Accept ~ "application/json") {
set req.http.Accept = "application/json";
} else {
set req.http.Accept = "text/html";
}
}
Next you need to have your app actually send Vary: Accept (inside your app source files). Alternatively, you can throw some Varnish VCL instead, if modiying app source files is not feasible:
sub vcl_fetch {
if (!beresp.http.Vary) { # no Vary at all
set beresp.http.Vary = "Accept";
} elseif (beresp.http.Vary !~ "Accept") { # add to existing Vary
set beresp.http.Vary = beresp.http.Vary + ", Accept";
}
}
statusRouter.route('/')
.all(function(req,res,next){
res.writeHead(200, {'Content-Type': 'application/json'});
next();
})
.get(function(req, res, next) {
res.json({
name : "xyz"
});
});
This crashes with - Header cant be set after it has been sent.
But the catch is , this works :
statusRouter.route('/')
.all(function(req,res,next){
res.writeHead(200, {'Content-Type': 'text/plain'});
next();
})
.get(function(req, res, next) {
res.end("xyz");
});
NOTE : If I remove the writeHead function in the first case where I am sending JSON it starts working as well. Why does it not work when i do a writeHead on it ? This thing is driving me crazy can anyone explain why this happens ?
P.S I'm working with express-generated app with my own router.
Both res.writeHead() and res.end() are not implemented by Express, but by the Node.js http module.
Its documentation states, for res.end():
If data is specified, it is equivalent to calling response.write(data, encoding) followed by response.end(callback)
So res.end("xyz") is short for:
res.write("xyz");
res.end();
For res.write() the documentation states:
If this method is called and response.writeHead() has not been called, it will switch to implicit header mode and flush the implicit headers.
So res.end("xyz") is actually short for:
if (! res.headersSent) {
res.writeHead(...);
}
res.write("xyz");
res.end();
This means that it's perfectly okay to issue res.writeHead() in your own code, before using res.end(). Internally, the http module will know that you already flushed the headers, so it won't do it again (therefore preventing the error you're getting). However, you can't set different headers, or change existing ones, once writeHead() has been called.
Now, res.json() is another matter: this isn't part of the http module, but of Express itself. Because it's used to send JSON responses, it will set the content type header to application/json (so you don't have to).
But this will only work if the headers haven't yet been sent already: you can't set headers when they have already been sent out. That's why you're getting the error.
If you want to set particular headers in Express, use res.set().
Express evaluates its routes in the order they are added. This means that the route handler passed to .all() will be executed first. In this handler, a response header is written and then next() tells Express to continue iterating its list of routes for another possible matching route.
It then matches the .get() route handler which tries to send a JSON response. However, .json() implicitly sends a response header as well as writes the value passed as JSON to the response and ends the response. This is why you are seeing an error about writeHead() being called more than once.
Is it possible to change the url of a HTTP-request without redirection?
For example instead of:
request 1
GET /user/abc123/ HTTP/1.1
HTTP/1.1 301 Moved Permanently
Location: /files/abc123
request 2
GET /files/abc123 HTTP/1.1
HTTP/1.1 200 OK
.
.
[filecontent]
I could could respond the file directly, but letting the client know that he got redirected:
single request
GET /user/abc123/ HTTP/1.1
HTTP/1.1 200 OK
Location: /files/abc123
.
.
[filecontent]
As far as I know, it's not possible to do this with HTTP. Redirection in HTTP specifically means the the client is supposed to send a second a request.
I think what you want is more akin to specifying a "canonical url" for some resources, and then having this canonical url displayed in the browsers location bar.
RFC 6596 specifies a way to specify canonical urls with <link rel="canonical">. However, it does not specify what a browser should do with it, if anything. Google uses it to make better choices about which urls to index.
Other than using <link> tags, it's also possible to specify relationships between resources via the HTTP Link header, i.e. Link: </better-url>; rel=canonical. See http://www.w3.org/wiki/LinkHeader . I'm not sure if this would be picked up by Google though. The page at http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394 doesn't mention Google supports it. Browsers surely will disregard it, as they do with practically any link tag, stylesheets being the notable exception.
If the content in question is a HTML document, you could use the HTML5 history API for this. Specifically, use the history.replaceState method. I don't think achieving something similar is possible with other types of content.
Edit
Content-Location header may actually fit what you want quite well.
From section 14.14 of HTTP 1.1 RFC:
The Content-Location entity-header field MAY be used to supply the resource location for the entity enclosed in the message when that entity is accessible from a location separate from the requested resource's URI. A server SHOULD provide a Content-Location for the variant corresponding to the response entity; especially in the case where a resource has multiple entities associated with it, and those entities actually have separate locations by which they might be individually accessed, the server SHOULD provide a Content-Location for the particular variant which is returned.
Content-Location = "Content-Location" ":"
( absoluteURI | relativeURI )
The value of Content-Location also defines the base URI for the entity.
The Content-Location value is not a replacement for the original requested URI; it is only a statement of the location of the resource corresponding to this particular entity at the time of the request. Future requests MAY specify the Content-Location URI as the request- URI if the desire is to identify the source of that particular entity.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
See also What is the purpose of the HTTP header field “Content-Location”?
Well, it is possible, but it feels a bit dirty though.
A quick demo:
var express = require('express');
var app = express();
app.get('/user/abc123', function(req, res, next) {
req.path = req.url = '/files/abc123';
next();
});
app.get('/files/abc123', function(req, res) {
res.set('Location', req.url);
res.send('files!');
});
app.listen(3012);
This is my simple approach, not just change the originalUrl but the path too. My method suggestions:
app.use(function(req, res, next) {
console.log("request", req.originalUrl);
const removeOnRoutes = '/not-wanted-route-part';
req.originalUrl = req.originalUrl.replace(removeOnRoutes,'');
req.path = req.path.replace(removeOnRoutes,'');
return next();
});
By this way /not-wanted-route-part/users will became /users