I have a Varnish cache setup in front of my application. I want some routes to be not cached, but completely passed through to other service.
I need to preserve headers like X-Forwarded-For so underlying service won't notice anything.
I've tried "pipe" configuration but it didn't worked as I excepted.
I'm looking for complete config section.
Config I've tried looked like:
backend blog {
.host = "127.0.0.1";
.port = "8083";
}
sub vcl_recv {
if (req.url ~ "^/blog/") {
set req.backend_hint = blog;
return (pipe);
} else {
set req.backend_hint = default;
}
}
You're looking for return(pass), which will not serve the content from the cache and connect directly to the backend.
The return(pipe) logic is only useful if you're unsure whether or not the incoming request is actually an HTTP request. By piping a request, you take away all notion of HTTP and reduce the data to plain TCP.
Thanks to return(pass) all headers are maintained.
Here's an updated version of your code example:
vcl 4.1;
backend blog {
.host = "127.0.0.1";
.port = "8083";
}
sub vcl_recv {
if (req.url ~ "^/blog/") {
set req.backend_hint = blog;
return (pass);
} else {
set req.backend_hint = default;
}
}
I want to use ESI in Varnish to combine content from different sides.
Every side is a small micro service with a small frontend snippet.
ESI should construct the page with the different snippets.
I'll be using Varnish 4.0.5.
As long I'll use it for content from my side its works fine.
<html>
<body>
<esi:include src="/hello"/> <!-- works -->
<esi:include src="http://www.example.org/index.html"/> <!-- doesn't works -->
</body>
</html>
Here's my vcl
vcl 4.0;
backend default {
.host = "localhost";
.port = "8080";
}
sub vcl_recv {
# Only a single backend
set req.backend_hint= default;
# Setting http headers for backend
set req.http.X-Forwarded-For = client.ip;
# Unset headers that might cause us to cache duplicate infos
unset req.http.Accept-Language;
unset req.http.User-Agent;
# drop cookies and params from static assets
if (req.url ~ "\.(gif|jpg|jpeg|swf|ttf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
unset req.http.cookie;
set req.url = regsub(req.url, "\?.*$", "");
}
# drop tracking params
if (req.url ~ "\?(utm_(campaign|medium|source|term)|adParams|client|cx|eid|fbid|feed|ref(id|src)?|v(er|iew))=") {
set req.url = regsub(req.url, "\?.*$", "");
}
}
sub vcl_backend_response {
set beresp.do_esi = true;
}
I'll get the following result in the browser
hello
Cannot GET /index.html
When I define the external host also in the VCL
backend otherbackend {
.host = "www.example.org";
.port = "80";
}
and
sub vcl_recv {
# Only a single backend
set req.backend_hint= default;
if (req.http.host == "www.example.org") {
set req.backend_hint = otherbackend;
}
I'll get some content from the external site (static assets will no be served and therefore leads to an error in the browser)
Question
- Is there a way to get content from an external site without defining every external site as a backend?
Since a little while I am using Varnish Cache solutions and as long as the configuration was more or less the same as it was after the installation everything worked out very well.
But now I like to do a little bit more with Varnish. Currently I am using the following setup for my servers :
Visitors -> CloudFlare -> HaProxy -> Varnish (separate servers) -> Apache2 Content.
What I like to know is how am I able to make the right vcl script to accept let's say an incoming request from ip A on port B and redirect this to ip C on port D. (And this more than once.)
Example :
Default.Varnish works well like this:
DAEMON_OPTS="-a :8085,:8087 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,768m"
But now about the Varnish.Default :
backend default_1 { .host = "11.22.333.444"; .port = "8885"; }
backend default_2 { .host = "11.22.333.444"; .port = "8887"; }
And I tried something like this :
sub vcl_recv {
if (server.port == 8885) { set req.backend = default_1; }
if (server.port == 8887) { set req.backend = default_2; }
}
(Please be aware of the fact that both requests are going to the
same outgoing server. Only the port is different!)
Someone who knows enough from Varnish already knows what I want I guess. I just like to use Varnish to proxy separate 'channels' based on different ports.
Try to check Varnish bind port, not backend port
sub vcl_recv {
if (server.port == 8085) { set req.backend = default_1; }
if (server.port == 8087) { set req.backend = default_2; }
}
I would like to route subdomains in varnish using regex capture. Here is my attempt:
backend gitlab {
.host = "127.0.0.1";
.port = "82";
}
backend jenkins {
.host = "127.0.0.1";
.port = "83";
}
sub vcl_recv {
if (req.http.host ~ "^((gitlab|jenkins|ruby))\.") {
set req.backend = $1;
return(pass);
}
error 405 "No service.";
}
How can this kind of construct be achieved in VCL? I'd rather not use a less elegant "if-then" pattern.
You can not refer to a backend with a string. The VCL compiler doesn't support it. You might be able to write a special director (a vmod) that does the work for you, but this would require you to bring out your copy of K&R and start digging around in the Varnish source tree.
It would not surprise me if somebody will write a VMOD to do something like this some day. It will be very useful.
I have a node application on an instance-store amazon machine behind the elastic load balancer (elb). However, the remote IP adress seems to always be the same. I used this code to get the client's IP address in node (via connect/express):
req.socket.remoteAddress
I didn't get anything else from the node documentation. Any hint?
Here's a solution in case you are using express:
According to the documentation, you can enable trust proxy for your express instance and then req.ip will be populated with the correct ip address.
By enabling the "trust proxy" setting via app.enable('trust proxy'),
Express will have knowledge that it's sitting behind a proxy and that
the X-Forwarded-* header fields may be trusted, which otherwise may be
easily spoofed.
Enabling this setting has several subtle effects. The first of which
is that X-Forwarded-Proto may be set by the reverse proxy to tell the
app that it is https or simply http. This value is reflected by
req.protocol.
The second change this makes is the req.ip and req.ips values will be
populated with X-Forwarded-For's list of addresses.
Here's an example:
var app = express();
app.enable('trust proxy');
// ...
app.use(function(req, res, next) {
console.log('client ip address:', req.ip);
return next();
});
The answer worked for me, thanks. But you may just try:
var ip_address = null;
if(req.headers['x-forwarded-for']){
ip_address = req.headers['x-forwarded-for'];
}
else {
ip_address = req.connection.remoteAddress;
}
sys.puts( ip_address );
Your receiving the IP of the ELB instance and you'll need to get the x-forwarded-for value from the headers. Since I'm not a node.js guru, I found this code at http://forum.webfaction.com/viewtopic.php?id=4500
Example:
var http = require( 'http' ),
sys = require( 'sys' );
http.createServer(
function( req, res ) {
var ip_address = null;
try {
ip_address = req.headers['x-forwarded-for'];
}
catch ( error ) {
ip_address = req.connection.remoteAddress;
}
sys.puts( ip_address );
}
);
The selected correct answer here is dangerous, because AWS ELBs switch the order as expected: https://github.com/koajs/koa/issues/1094#issuecomment-345861282
Express, koa, etc. typically take the left-most item, while ELB makes it the right-most item
(express docs):
If true, the client’s IP address is understood as the left-most entry in the X-Forwarded-For header.
In case if express.js is in use:
app.set('trust proxy', 2)
Instead of
app.enable('trust proxy')
Because the app.enable('trust proxy') uses the leftmost ip from the x-forwarded-for header and so can be easily spoofed by just providing x-forwarded-for header manually.
While the app.set('trust proxy', 2) has the number of hops specified that being counted from right to left of the x-forwarded-for header. I.e. if there is an AWS load balancer than 2 will be the right number to count because each new hop ip is added to the end of the x-forwarded-for header.
If you're using something else then do the similar way. Just get the req.headers['x-forwarded-for'], split by coma and then count hops from right to left until load balancer ip is not excluded.