This is my first time experimenting with Varnish.
We noticed our CDN not doing any request coalescing, so as an experiment I'm allowed to try out Varnish.
However I can't seem to figure out how to modify the response headers when stale content is served.
Image the following vcl and all backend responses have max-age=60, public, s-maxage=600 as the cache-control directive.
sub vcl_backend_response {
set beresp.grace = 3600s;
return (deliver);
}
When I:
visit /foo
do a varnish soft purge and a subsequent CDN (hard) purge on /foo
visit /foo again
When visiting /foo for the second time I'll immediately get a (stale) response. However I don't want it to have the same cache-control directive anymore, because then my CDN will continue to serve this stale response for another 10 minutes.
Is it possible to change the cache-control directive for stale content to max-age=0, public, s-maxage=10?
Here's the VCL code you need to modify the Cache-Control header for stale content:
vcl 4.1;
sub vcl_deliver {
//set resp.http.x-ttl = obj.ttl;
//set resp.http.x-grace = obj.grace;
if(obj.ttl <= 0s && obj.grace > 0s ) {
set resp.http.Cache-Control = "max-age=0, public, s-maxage=10";
}
}
If you want to debug the process, just uncomment the 2 lines and look for the x-ttl and x-grace response headers to see the remaining values for both timers.
Although you can control the staleness by setting beresp.grace in VCL, you can also set the staleness through the stale-while-revalidate directive in the Cache-Control header.
Here's an example:
Cache-Control: max-age=60, public, s-maxage=600, stale-while-revalidate=3600
This will allow the browser to cache for a minute, allow intermediary servers to cache for 10 minutes and set an allowed staleness of an hour while revalidating outdated content.
Related
Is there any way to cache request with auth headers in varnish?
I want to ignore the auth headers while caching the request
There are various ways to approach this, depending on the importance of auth headers.
1. You don't care about auth
If you don't care about the auth part and if you want to risk serving cached content to unauthorized users, you can just use the following VCL code:
sub vcl_recv {
unset req.http.Authorization;
}
2. Ignore authorization to some extent
It is also possible to care about auth a bit, but not too much.
The following VCL snippet will allow caching even if there is an Authorization header:
sub vcl_recv {
if(req.http.Authorization) {
return(hash);
}
}
The consequence of this is that the initial cache miss will pass through to the backend and will be processed there. Potential unauthorized access will be handled there.
But as soon as the has been dealt with, the object is stored in the cache and the next requests will get cached content regardless of the authorization status of that request.
3. Perform auth on the edge
It is also possible to handle the auth part in Varnish while caching the content.
The following VCL code will handle this:
sub vcl_recv {
if(req.http.Authorization != "Basic YWRtaW46c2VjcmV0") {
return (synth(401, "Restricted"));
}
unset req.http.Authorization;
}
sub vcl_synth {
if (resp.status == 401) {
set resp.http.WWW-Authenticate = {"Basic realm="Restricted area""};
}
}
This code will actively inspect the content of the Authorization header and will ensure the username admin is used with password secret.
The YWRtaW46c2VjcmV0 string is nothing more than a base64 encoding of admin:secret.
4. Use vmod_basicauth
A more advanced and flexible way to terminate auth on the edge is by using https://git.gnu.org.ua/vmod-basicauth.git/. This VMOD can be compiled from source and can be downloaded from ftp://download.gnu.org.ua/release/vmod-basicauth.
Assuming the credentials are stored in /var/www/.htpasswd, you can leverage this VMOD to match the Authorization header to the content of the .htpasswd file.
Here's the VCL:
vcl 4.1;
import basicauth;
sub vcl_recv {
if (!basicauth.match("/var/www/.htpasswd",req.http.Authorization)) {
return (synth(401, "Restricted"));
}
unset req.http.Authorization;
}
sub vcl_synth {
if (resp.status == 401) {
set resp.http.WWW-Authenticate = {"Basic realm="Restricted area""};
}
}
This is entirely possible but also extremely dangerous: Varnish would return the same cached (authorized) content to all requests.
Example:
User A requests resource Z with proper authentication. Varnish relays the request to backend, caches the response and returns the resource.
User B requests resource Z with proper authentication. They will get the cached resource Z even if Z contains user A's content.
User X requests resource Z with invalid authentication. They will too get the cached resource anyway since the backend is bypassed.
Having said that, you can override Varnish's built-in VCL. Details are documented but the main idea is:
Copy default vcl_recv VCL (for your version) from source and add it to the end of your vcl_recv.
Remove the safeguards from vcl_recv: Just remove vcl_req_authorization which disables caching:
sub vcl_req_authorization {
if (req.http.Authorization) {
# Not cacheable by default.
return (pass);
}
}
In your vcl file issue a return statement at the end so built-in vcl is not used.
My requirement is to enable "secure" flag in one of the header called "_adminv2_session" but which contains a dynamic value. I was trying to enable the flag using the below configurations in nginx, but it'll create a new header with same name and assign value "/"
add_header Set-Cookie "_adminv2_session=/; HttpOnly; Secure";
But when I try without value, it gives errors,
add_header Set-Cookie "_adminv2_session; HttpOnly; Secure";
Can anyone help me on enable secure flag on the so called header in nginx ?
Screenshot of current status,
Thanks.
You might need to use proxy_cookie_path and add_header directives like this.
As far as I understood, you need to set your _adminv2_session variable value first, then to add Secure flag to your Set-Cookie header.
Also you can use directive more_set_headers from 3rd-party headers_more module to do you task in one directive.
more_set_headers 'Set-Cookie: $sent_http_set_cookie; HttpOnly; Secure';
I am testing my webpage with webpagetest.org
On my page, there are a bunch of images. I can see them well cached in the repeated run: (304 response is marked as yellow in WebPageTest waterfall result)
However, in cache static content, there's no check on those resources
I found the difference is that those scripts and styles have cache-control: max-age=2592000, while those media resources have cache-control: max-age=0 in the server response. Does it mean that WebPageTest will neglect these responses with max-age=0 in static cache checking?
Does it mean that WebPageTest will neglect these responses with max-age=0 in static cache checking?
The documentation states that resources which include a specific indication of non-cacheability will not be subject to the 'Cache Static' check:
Applicable Objects
Any non-html object with a mime type of "text/*", "*javascript*" or "image/*" that does not explicitly have an Expires header of 0 or -1, a cache-control header of "private", "no-store" or "no-cache" or a pragma header of "no-cache"
While max-age=0 isn't included in that list, it should be treated the same as no-cache, and is likely being treated the same here and excluding those objects from this check.
We're currently using the s-maxage directive in the Cache-Control header from our origin to control the TTL in Varnish. However, I'd like to remove it from the response before delivery, so that no other caches in the request chain act on it.
I'm currently looking at the header VMOD, to remove s-maxage from the header, but leave the rest of it intact. I believe this could be achieved with something like this:
sub vcl_deliver {
header.regsub(resp, "s-maxage=[0-9]+,?\s?", "")
}
As a newcomer to Varnish, I wanted to sanity-check this approach and make sure there isn't a better way to tackle it?
Appreciate any support or advice.
Replace header at delivery time
The following VCL snippet will strip off the s-maxage attribute from the Cache-Control header before it is sent to the client.
sub vcl_deliver {
set resp.http.cache-control = regsub(resp.http.cache-control,
"(,\s*s-maxage=[0-9]+\s*$)|(\s*s-maxage=[0-9]+\s*,)","");
}
Replace header at storage time
It is also possible to strip off this attribute from the Cache-Control header before it gets stored into a cache object. In that case, you'll use the beresp.http.cache-control variable inside vcl_backend_response.
sub vcl_backend_response {
set beresp.http.cache-control = regsub(beresp.http.cache-control,
"(,\s*s-maxage=[0-9]+\s*$)|(\s*s-maxage=[0-9]+\s*,)","");
}
Using vmod_headerplus
If you're using Varnish Enterprise, you can use the vmod_headerplus module to easily delete header attributes:
vcl 4.1;
import headerplus;
sub vcl_deliver {
headerplus.init(resp);
headerplus.attr_delete("Cache-Control","s-maxage",",");
headerplus.write();
}
vcl 4.1;
import headerplus;
sub vcl_backend_response {
headerplus.init(beresp);
headerplus.attr_delete("Cache-Control","s-maxage",",");
headerplus.write();
}
Although Varnish Enterprise is the commercial version of Varnish Cache, you can still use it without upfront license payments if you use it on AWS, Azure or GCP.
Varnish Enterprise on AWS
Varnish Enterprise on Azure
Varnish Enterprise on GCP
I'm trying to get CORS working on my trigger.io app:
I've got the following setup in my .htaccess
Header set Access-Control-Allow-Headers: "Accept,Origin,Content-Type,X-Requested-With"
Header set Access-Control-Allow-Methods "GET,PUT,POST,DELETE,OPTIONS"
Header set Access-Control-Allow-Credentials: "true"
Header set Access-Control-Allow-Origin "http://localhost:3000,content://io.trigger.forge99d5a0b8621e11e28cc2123139286d0c"
Running the trigger App in the web (localhost:3000) works fine.
But when I deploy it to an (android) device I see the following error in the debug output:
[ERROR] XMLHttpRequest cannot load {link}http://mydevtest.lan/api/auth/currentuser.{/link} Origin content://io.trigger.forge99d5a0b8621e11e28cc2123139286d0c is not allowed by Access-Control-Allow-Origin. -- From line 1 of null
I'm fearing that setting content:// in the Access-Control-Allow-Origin header is not legal.
The Access-Control-Allow-Origin header as you have it is invalid. Valid values are either '*', or a space separated list of origins. One of the following should work:
Header set Access-Control-Allow-Origin "*"
or
Header set Access-Control-Allow-Origin "http://localhost:3000 content://io.trigger.forge99d5a0b8621e11e28cc2123139286d0c"
Note that I've never tested the latter form (with multiple origins). While the CORS spec allows it, browsers may not yet support it.
One other thing you could do is read in the value of the Origin header, validate it on your server (i.e. manually check that the value equals either "http://localhost:3000" or "content://io.trigger.forge99d5a0b8621e11e28cc2123139286d0c"), and then echo only that value in the Access-Control-Allow-Origin response header. However this requires a little more work since it introduces some server-side conditional processing.
I also fear that content:// is not allowed in CORS, could you try setting Access-Control-Allow_origin to *, if that works then that is probably the problem.
A better solution would be to avoid doing XHR requests and use forge.request.ajax which will make the request from native code and avoid any cross domain restrictions. You can find the documentation for that here http://docs.trigger.io/en/v1.4/modules/request.html#modules-request