Right, I'm going to be honest, I don't know varnish vcl, I can work out some basic stuff but I don't know it very well which is obviously why I'm having issue's.
I'm trying to set up cache banning via a http request, however the request can't come in via the DNS but rather through the IP address of the varnish box otherwise I can't be sure that every varnish box cache will have the target flushed; this is because we have several varnish boxes all behind an ELB so you can't guarantee that a ban request will not go to the same box twice, hence doing this via IPs.
I'm using this to insure that only the allowed IP's are allowed to ban but this isn't working:
sub vcl_hit {
if (req.request == "BAN") {
ban("req.url ==" + req.url);
error 200 "Purged";
}
}
I don't really know what to do to get this working and I've looked but most of the tutorials I've found seem to be for full URLS rather than just ip + pattern_to_purge
from your config example i expect you use varnish 3
you can add a list of ips that is allowed to do the purge as followed
acl ban_allowed_ip {
"127.0.0.1";
"127.0.0.2";
}
inside your if(req.request =="BAN") add the following
if (!client.ip ~ ban_allowed_ip) {
error 405 "Not allowed.";
}
The answer is to use:
if (req.request == "BAN") {
if (req.http.X-Debug != "True") {
error 405 "Not allowed.";
}
ban("obj.http.x-url ~ " + req.url);
error 200 "ban added";
}
Whilst this will return 200 regardless if the item in the cache exists or not, it does add the ban.
Related
I'm assuming that I must have made a mistake in my Varnish configuration. (I'm running version 6.0.7.) Here is the relevant section:
####----SECTION THREE: PURGE RULES----####
# Access Control List to define which IPs can purge content
acl purge {
"localhost";
"127.0.0.1";
"<my home IP address>";
"::1";
}
####----SECTION FOUR: PROCESSING RULES----####
sub vcl_recv {
# Do not allow purging from non-approved IPs
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return (synth(405, "This IP is not allowed to send PURGE requests."));
}
return (purge);
}
# Allow banning regexes
if (req.method == "BAN") {
if (!client.ip ~ purge) {
return (synth(405, "This IP is not allowed to send BAN requests."));
}
ban("req.http.host == " + req.http.host + " && req.url ~ ^" + req.url);
return (synth(200, "Ban added"));
}
# Allow purging regexes
if (req.method == "PURGE") {
if (req.http.X-Purge-Method == "regex") {
ban("req.url ~ " + req.url + " && req.http.host ~ " + req.http.host);
return (synth(200, "Banned"));
} else {
return (purge);
}
(N.B.: In my setup, Varnish is running on port 6081, with HAProxy in front of it.)
Using this setup, from either my home IP address or the server Varnish is running on, I am able to clear individual pages from the Varnish cache by running, e.g.:
curl -i -X PURGE https://example.com/page/
I can also clear the entire domain from the Varnish cache by running, e.g.:
curl -i -X BAN https://example.com
Likewise, I can clear individual pages from the Varnish cache by running, e.g.:
curl -i -X PURGE http://<IP of Varnish server>:6081/page/ -H "Host: example.com”
I can also clear the entire domain from the Varnish cache by running, e.g.:
curl -i -X BAN http://<IP of Varnish server>:6081 -H "Host: example.com”
But I have noticed that I cannot clear the entire domain by running either:
curl -i -X PURGE https://example.com
or:
curl -i -X PURGE http://<IP of Varnish server>:6081 -H "Host: example.com”
To clear the entire domain, I have to use BAN instead. I do not know if this is a problem or a misconfiguration, but it seems to be, because although I have read that PURGE is not used to clear everything, the Wordpress plugin I am using seems to be trying to, by sending a PURGE request to domain.com/.* when I hit the “Clear ALL Varnish Cache” button.
This, frankly, confuses me. Because it doesn’t work either in the plug-in itself or if I try it from the command line. What am I missing? Is the plugin just out of date? Did Varnish used to be able to clear the entire cache this way? I'd appreciate any pointers. Thanks!
Here's a cleaned up version of the purging logic in your VCL:
acl purge {
"localhost";
"127.0.0.1";
"::1";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return (synth(405, "This IP is not allowed to send PURGE requests."));
}
if (req.http.X-Purge-Method == "regex") {
ban("obj.http.x-url ~ " + req.url + " && obj.http.x-host ~ " + req.http.host);
return (synth(200, "Banned"));
}
return (purge);
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
Purging
The PURGE HTTP method is what triggers the cache invalidation. Without any headers return(purge); will be called, which will invalidate the exact URL that was used.
If the X-Purge-method request header is added, and is set to regex, a regular expression match will occur. Invalidating multiple objects using a regular expression cannot be done with return(purge);, and requires banning.
Bans
The ban() function adds a ban expression to the ban list. Expressions on this list are matched to all the objects that are stored in cache. The matching ones are removed.
You can consult the ban list using the following command on your Varnish server:
varnishadm ban.list
Lurker-friendly bans
The ban lurker, a special thread that processes the ban list, doesn't have access to the request context. If you want this thread to remove the objects from cache, based on request parameters like the URL or the hostname, you need to apply som trickery.
As you can see, the pattern I suggest, matches on obj.http.x-url & obj.http.x-host. These are set in vcl_backend_response via 2 custom response headers.
If you don't do this, the ban lurker cannot match the objects, and the responsibility will shift to the next request. Even without these so-called lurker-friendly bans, banning will still work, but it is not as efficient.
Testing it
The first example will invalidate http://example.com/page from the cache
curl -i -XPURGE -H "Host: example.com" "http://localhost:6081/page/"
The next example will invalidate all objects whose URL starts with http://example.com/page:
curl -i -XPURGE -H "Host: example.com" -H "X-Purge-Method: regex" "http://localhost:6081/page/"
The last example will invalidate all objects from cache for the example.com domain:
curl -i -XPURGE -H "Host: example.com" -H "X-Purge-Method: regex" "http://localhost:6081/.*"
Here's the ban list item for that last example:
root#varnish:/etc/varnish# varnishadm ban.list
Present bans:
1606393598.913178 0 - obj.http.x-url ~ /.* && obj.http.x-host ~ example.com
1606393484.785268 3 C
As you can see, it matches all URLs for the example.com host. When you perform an HTTP request after that, you'll see the Age: 0 response header. This indicates that this response has been stored in cache for zero seconds. This means the ban was successful.
I'm very new on varnish and I've a business on my hands recently. It's a local magazine website http caching (Tech Stack is Javascript + PHP). I'm trying to use varnish 4 for caching the website. What they want me to do is; any new articles should be appeared on FE immediately, any deleted articles should be erased from the FE immediately, any changes on website's current appereance should be applied directly (changing articles' current locations, they can be dragged anywhere on the website based on articles' popularity change.) and finally any changes on existing articles should be applied to website immediately. As you see on the config below, in sub vcl_recv block I tried to use return(purge) for POST requests, because new articles and article changes is applied via POST request. But it doesn't work at all. When I try create a new dummy content or make changes on existing articles, it's not purging the cache and showing the fresh content even if POST request is successful. Also, on the BE side, I tried to use if (beresp.status == 404) for deleted articles, but it doesn't work too. When I delete the dummy article I created, it's not being deleted too, I'm still seein the stale content. How should I change my config to get all these things done? Thank you.
my varnish config is ;
import directors;
import std;
backend server1 {
.host = "<some ip>";
.port = "<some port>";
}
sub vcl_init {
new bar = directors.round_robin();
bar.add_backend(server1);
}
sub vcl_recv {
set req.backend_hint = bar.backend();
if (req.http.Cookie == "") {
unset req.http.Cookie;
}
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");
if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
unset req.http.cookie;
}
if (req.url ~ "\.*") {
unset req.http.cookie;
}
if (req.method == "POST") {
return(purge);
}
}
sub vcl_deliver {
# A bit of debugging info.
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
}
else {
set resp.http.X-Cache = "MISS";
}
}
sub vcl_backend_response {
set beresp.grace = 1h;
set beresp.ttl = 120s;
if (bereq.url ~ "\.*") {
unset beresp.http.Set-Cookie;
unset beresp.http.Cache-Control;
}
if (bereq.method == "POST") {
return(abandon);
}
if (beresp.status == 404) {
return(abandon);
}
return (deliver);
}
No need to use the director if you only have one backend. Varnish will automatically select the backend you declared if there's only 1 backend.
Purging content
The POST purge call you're doing is not ideal. Please have a look at the following page to learn more about content invalidation in Varnish: https://varnish-cache.org/docs/6.0/users-guide/purging.html#http-purging
The snippet on that page contains an ACL to protect your platform from unauthorized purges.
It's important to know that you'll need to create a hook into your CMS or your MVC controller, that does the purge call.Here's a simple example using curl in PHP:
$curl = curl_init("http://your.varnish.cache/url-to-purge");
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "PURGE");
curl_exec($curl);
As you can see, this is an HTTP request done in cURL that uses the custom PURGE HTTP request method. This call needs to be executed in your good right after the changes are stored in the database. This post-publishing hook will ensure that Varnish clears this specific object from cache.
VCL cleanup
The statement below doesn't look like a reliable way to remove cookies, because the expression will remove cookies from all pages dat contain a dot:
if (req.url ~ "\.*") {
unset req.http.cookie;
}
The same applies to the following statement coming from the vcl_backend_response hook:
if (bereq.url ~ "\.*") {
unset beresp.http.Set-Cookie;
unset beresp.http.Cache-Control;
}
I assume some pages do actually need cookies to properly function. An admin panel for example, or the CMS, or maybe even a header that indicates whether or not you're logged in.
The best way forward is to define a blacklist or whitelist of URL patterns that can or cannot be cached.
Here's an example:
if(req.url !~ "^/(admin|user)" {
unset req.http.Cookie;
}
The example above will only keep cookies for pages that start with /admin or /user. There are other ways as well.
Conclusion
I hope the purging part is clear. If not, please take a closer look at https://varnish-cache.org/docs/6.0/users-guide/purging.html#http-purging.
In regards to the VCL cleanup: purging can only work if the right things are stored in cache. Dealing with cookies can be tricky in Varnish.
Just try to define under what circumstances cookies should be kept for specific pages. Otherwise, you can just remove the cookies.
Hope that helps. Good luck.
Thijs
I've already asked and received an awesome answer about how to get an error from the backend to force serving from stale cache (grace) found here: varnish 4: serve graced object if beresp.status is an error?
but now that logic needs an extra step: i include the following code currently
sub vcl_backend_fetch {
if (bereq.retries == 0) {
unset bereq.http.X-Varnish-Backend-5xx;
unset bereq.http.X-Varnish-Backend-206;
} else {
if (bereq.http.X-Varnish-Backend-5xx) {
return (abandon);
}
if (bereq.http.X-Varnish-Backend-206) {
return (abandon);
}
}
}
sub vcl_synth {
if (resp.status == 503 &&
req.method != "POST" &&
!req.http.X-Varnish-Restarted-5xx) {
set req.http.X-Varnish-Restarted-5xx = "1";
return (restart);
}
if (resp.status == 503 &&
req.method != "POST" &&
!req.http.X-Varnish-Restarted-206) {
set req.http.X-Varnish-Restarted-206 = "1";
return (restart);
}
}
obviously the second if statement in the vcl_synth is virtually identical to the first one, with the exception of the header it's looking for. I need to differentiate the 206 to restart with a different request header, but I am not sure how. the issue is that, if the backend returns a 206, the rest of the logic abandons the backend fetch (which hands off to vcl_synth with a 503), and vcl_synth restarts the request to force serving graced cached objects. however, if there's no graced cache object to return to the user, then the user sees a 503 instead of a 206.
Before realizing that this line of thinking was not possible, i tried to have vcl_backend_fetch return a synth(206), so that vcl_synth could use resp.status to differentiate, and add a different request header before restarting the request. then i would be able to look for that header in vcl_miss, and if it's there, i could restart the whole request a second time, and force it to serve the 206 from the usual backend.
TL;DR: how do I differentiate two different cases in a vcl_backend_fetch abandon, to get vcl_synth to restart the request with two different headers?
I'm pretty sure it's not possible to do anything in vcl_backend_fetch in order to abandon the request and transfer some kind of flag to vcl_synth.
I'm not sure if I've completely understood your use case, but it seems that would need something like:
sub vcl_synth {
if (resp.status == 503 &&
req.method != "POST" &&
!req.http.X-Varnish-Restarted-5xx) {
# Let's restart the request for the first time and try to
# serve grace content. This could select an always-sick
# backend during vcl_recv.
set req.http.X-Varnish-Restarted-5xx = "1";
return (restart);
}
if (resp.status == 503 &&
req.method != "POST" &&
req.http.X-Varnish-Restarted-5xx &&
!req.http.X-Varnish-Restarted-206) {
# Grace content was not available after the first restart. Let's
# restart again the request and try to serve content using some
# other backend.
set req.http.X-Varnish-Restarted-206 = "1";
return (restart);
}
}
(Varnish 2.1.5)
I've got some strange situation in my Varnish. I'm trying to invalidate cache objects through PURGE requests initiated from NodeJS.
My testing consists of requesting the object, letting it cache, then do a purge request, then request it again (resulting in a fetch), and then request it again resulting in a hit of the refreshed cache object.
When I test this through the Firefox debug console, it works fine. All steps seem to work as expected. When I test the entire process in NodeJS, it works as expected, just fine. However, when I let the object cache through Firefox, and then try to invalidate it through NodeJS, it reports a 404 Not in cache.
I'm a 100% sure I'm using the same URI, and I have no idea why it acts this way. Has anyone else experienced this problem? And if yes, what is the solution to this problem?
This is my VCL:
backend default {
.host = "127.0.0.1";
.port = "80";
}
acl purge {
"localhost";
"*loadbalancer-ip*";
}
sub vcl_recv {
if (req.request == "PURGE") {
if(!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
} else if (req.url ~ "(?i)\.(jpeg|jpg|png|gif|ico|js|css|xml)$") {
unset req.http.Cookie;
return (lookup);
} else {
return (pass);
}
}
sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
error 404 "Not in cache.";
}
}
sub vcl_fetch {
if (req.url ~ "(?i)\.(jpeg|jpg|png|gif|ico|js|css|xml)$") {
unset beresp.http.set-cookie;
return (deliver);
}
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}
As you can see, my configuration is pretty straight forward. This configuration is for testing purposes, I know using the loadbalancer IP is not safe and I will change it to use the Forwarded-For IP once everything works.
With a little help of this thread:
What is the function of the "Vary: Accept" HTTP header?
I came to know that it considers the Vary header when determining whether to cache or not, and which version to get from cache.
In my case, the Vary header contained the User-Agent, which is why I was getting different results from different methods.
Can I configure Varnish in such a way it shows the original page from the backend when the backend throws a 500 error page?
It's the default. I have some if (beresp.status == 500) in it :s
I assume you want to show the original 500 error only in some environments, like development.
If so, then you can assign Varnish an identity:
$ varnishd -i development
And then check that identity in your VCL:
sub vcl_fetch {
if (server.identity ~ "^development") {
return (deliver);
}
if (beresp.status == 500) {
# ...
}
}