How to purge Varnish cache when using Cloudflare? - varnish

Its seems that purging doesn't work when using Cloudflare(Cloudflare returns 403 Forbidden).
This what i got when i searched for a solution online:
"The problem is that when you are using cloudflare, varinsh does not get the original IP of the sender. Instead it gets the IP of the cloudflare. So purging can not be done. We need to tell the varnish the original IP of the sender."
Add these following lines inside vcl_recv
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.method == "PURGE" || req.url == "/purge") {
# Replace these IP with your IP
if ( req.http.X-Forwarded-For !~ "(209.152.41.21|105.45.120.37)") {
return(synth(405, "This IP is not allowed to send PURGE requests."));
}
ban("req.url ~ /");
return (purge);
}
I tried this solution but it didn't work.

This question is old but could still use an answer. :-) The quote you found is partially correct. Here's why:
Your setup is likely similar to the below:
------ --------- ------------ ------
| WP | <- | Varnish | <- | CloudFlare | <--- | User |
------ --------- ------------ ------
There are two ways that the purge can happen:
User -> CloudFlare -> Varnish, or
User -> CloudFlare -> Varnish -> WP -> WP plugin -> Varnish.
The second situation can successfully cause a purge. If you have a plugin which triggers a cache purge/invalidate, it will come from a predictable IP address. In fact, if you run varnish on the same server as WP, the IP address will be [127.0.0.1]. There's a nice implementation for this situation.
The problem with the first situation (purging directly from CloudFlare) is that CloudFlare has many IP addresses which you would have to keep up to date. But more importantly, there's nothing that would prevent a bad actor from also creating a service using CloudFlare which would also be allowed to send a purge request to your server, basically rendering this security worthless.
Since X-Forwarded-For is basically just a list of all of the previous IP addresses along the way, this would also be easy to fake and bypass.
Since filtering on IP address is not useful directly through CloudFlare, you could alternatively use a token/secret that you send as part of the purge request. (i.e. only allow a special URL like:
if (req.url == "/purge-719179c7-6226-4b87-9503-1b6d54d5fea5") {... with some other Guid of course). One could argue that this is still not secure, but perhaps better than allowing all CloudFlare IPs.

Related

Varnish: Purge says it works but doesn't remove old content

I'm running a stand alone instance of varnish on a Digital Ocean Ubuntu VM which basically works fine. The setup is used to take load of an older wordpress server that sits anyhwere else. That works quite well but i'm having a hard time getting content purged. And when talking about purge i mean to invalidate the cache for a URL to force varnish to fetch a fresh version from the backend (just to make sure as i've seen some irritation about purge/ban).
I have setup an ACL for purge and as far as i can see with varnishlog the purges get accepted - on one side from the WordPress blog (where W3TC handles the purges) as well es from the local console where i tried to purge with curl -X PURGE http://url.to.purge
The problem is that i still get the old versions of the URL in the browser on matter what i do locally.
This is how i handle purge in vcl_recv:
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
return (purge);
}
and i get VCL_error(200, Purged) on every purge so i guess it's probably ok.
Looks like i'm still doing things wrong. After giving service varnish a restart the full cache refreshes and the pages refresh too - until then varnish keeps everything for ages - no matter how much i'm purging.
my Varnish version is 4.0.3.
Any idea?
Thanks,
Frank
Got same behavior on Varnish 6 with vcl 4.1.
The only way to solve it was explicitly define sub vcl_purge like this:
sub vcl_purge {
set req.method = "GET";
set req.http.X-Purger = "Purged";
return (restart);
}
Didn't find the reason and this may not be exactly what you want because after purge it will get content from the backend without waiting for client request.
But still didn't find another way and this is good enough for me.

Set Varnish headers conditionally based on client.ip

How would I use client.ip as a conditional in setting headers in the fetch section of a Varnish 3.0 VCL? I have some troubleshooting headers that I like to set to solve caching issues, however, I don't want them publicly visible. I'd love to be able to whitelist the headers for my ip address only.
Is there any way to access client.ip in _fetch?
You can best set all troubleshooting headers in you _recv without any conditions and remove them in you vcl deliver. this way you dont need to add the same ip check on every conditional header
if you want to use an ip range you can use the following code
acl debug {
"your ip adress1";
"you ip adress 2";
}
in you vcl_recv
if (!client.ip ~ debug) {
set req.http.x-debug = "debug";
}
in you vcl_deliver
if(!req.htt.x-debug){
remove resp.http.debugheader1;
remove resp.http.debugheader2;
}

How can i modify the host header

I am trying to develop a chrome extension that would set the "host" header on certain requests. But the documentation is contradicting as to if the "host" header can be modified or not.
Both of these issues indicate that a) it should not be possible and b) it is impossible
https://code.google.com/p/chromium/issues/detail?id=154900
https://code.google.com/p/chromium/issues/detail?id=158073
Yet multiple extensions in the gallery state they do modify the "host" header.
e.g.
https://chrome.google.com/webstore/detail/header-hacker/phnffahgegfkcobeaapbenpmdnkifigc?hl=en
https://chrome.google.com/webstore/detail/change-http-request-heade/ppmibgfeefcglejjlpeihfdimbkfbbnm
Is it possible to modify the "host" header in the windows version of chrome, and if so how?
Background: I want to be able to test load balanced web instances hitting each host directly via ip address. The "hosts" file is to cumbersome for a large number of hosts. At the moment I use curl to pass the modified "host" header, but I really need the solution in the browser and available for others.
#kzahel was right, and the note about redirection was spot on, here is my basic working code.
chrome.webRequest.onBeforeSendHeaders.addListener(function (details) {
if (details.url.indexOf('toast.txt') <= -1)
return;
details.requestHeaders.push({
name: 'Host',
value: 'testhost:80'
});
return { requestHeaders: details.requestHeaders };
}, {
urls: ['http://*/*']
}, ['requestHeaders', 'blocking']);
chrome.webRequest.onBeforeRequest.addListener(function (details) {
if (details.url.indexOf('sauce') <= -1)
return;
var url = 'http://127.0.0.1/toast.txt';
return { redirectUrl: url };
}, {
urls: ['http://*/*']
}, ['blocking']);
Admittedly a slightly odd example but it goes like this.
My local IIS has a site created that points to a folder that has a file "toast.txt", which is bound to "testhost".
Windows can no way of knowing about "testhost" e.g. cannot ping it.
With the extension running.
Enter the address http://testhost/sauce
The extension notes the "sauce" in the "onBeforeRequest" method and redirects to "http://127.0.0.1/toast.txt" which in turn is picked up on by the "onBeforeSendHeaders" method where the "Host" header is added containing "testhost:80". All this occurs before the request leaves Chrome.
Now IIS receives the request "/toast.txt" that by itself would result in a 404, but because the "Host" header is set it pushes the request to the new website that is bound to "testhost" which then returns the "toast.txt" file.
Phew!
It looks like you shouldn't have difficulty doing this. Use onBeforeRequest
onBeforeRequest: Fires when a request is about to occur. This event is sent before any TCP connection is made and can be used to cancel or redirect requests.
Since this is triggered before any connection to the server is made, you should be able to modify the host header then [edit: if host header is not available, then use a redirect]. Make sure you have the "requestHeaders" permission in the manifest or else you won't see the request headers at all.

Disable caching for certian IP's using varnish

Is it possible to let clients with certain IP's pass thru to the backend and not cache using varnish? I don't see this in any of the example configs.
i think better way is described here https://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-October/021278.html
if you have list of IP's you should create an acl list:
acl passem {
"192.168.55.0/24";
}
and then in vcl.recv you should
if (client.ip ~ passem) {
return(pass);
}
I received this answer from the mailing list.
Yes, you can:
if (client.ip == IP)
{
return(pass);
}

DNS: How do resource records work for an Authoritative DNS server?

But please bear with me. I do not need help with ndns or JavaScript. I need help with DNS Resource Records.
I can already send resource records. I just need to know how to send the right ones for an Authoritative DNS Server.
I am writing the DNS server using ndns. Ndns is supposed to do the low level communications for me, but I still have to know the basics of DNS. Ndns is not documented except for this example. It is in JavaScript, but it should be pretty easy to read anyway. When a request is received, it adds a resource record to the response and sends the response
function handleDnsRequest(request, response) {
response.addRR(
ndns.ns_s.ar, // Section AR
'node.js', // Name
ndns.ns_t.txt, // Type TXT
ndns.ns_c.in, // Class IN
1991, // TTL
'http://nodejs.org/' // Value
);
response.send();
}
So, no matter what the request, this handler adds a response record as follows
Section AR (Additional Records)
Name "node.js"
Type TXT (Text String)
Class IN (Internet)
TTL 1991 (~33 minutes)
Value (Text String)
Which gives this output on Windows nslookup
C:\>nslookup - 127.0.0.1
node.js text =
"http://nodejs.org/"
Default Server: UnKnown
Address: 127.0.0.1
> google.com
Server: UnKnown
Address: 127.0.0.1
Name: google.com
>
How can I send correct responses? I want to start off by sending a fixed IP address for all A records no matter what and to deny most everything else as unsupported or whatnot.
In a typical log in to nslookup, ask for an a record What would be the typical list of Resource Records that would come out of the DNS server?
I want to start off by sending a fixed
IP address for all A records no matter
what and to deny most everything else
as unsupported or whatnot.
Aha, now we're getting somewhere.
You need to return an RR in the answer section that has the same "owner name" as that in the (first) question, with the appropriate fields.
Try this:
function listener (req, res)
{
res.addRR(
ndns.ns_s.an, // answer section
req.question[0].name, // name
ndns.ns_t.a, // type
ndns.ns_c.in, // class
3600, // TTL
'127.0.0.1' // RDATA
);
res.header.aa = 1; // authoritative answer
res.header.ra = 0; // recursion not available
res.send ();
}
This only handles the default response, and doesn't check whether the inbound query was for an A record or not.
To refuse other queries you'll want to check for:
req.question.length == 1
req.question[0].type == ndns.ns_t.a
req.question[0].class == ndns.ns_c.in
and then set res.header.rcode to something non-zero.
A real authoritative server would also send DNS server names in the authority section, but you should be able to get away without doing so here.

Resources