If I set maxAge when I called sendFile(), just like below:
res.sendFile('public/index.html', {maxAge: 100000})
Does this mean the file 'public/index.html' will cached in server's memory for 100 seconds? Or, This is just a message send to client side, and server side's does nothing with memory cache?
maxAge is a directive for the Cache-Control header. In your case, this tells the client that index.html will be considered 'fresh' for 100000ms, so it is unnecessary to ask the server again for that file until maxAge has elapsed. The client keeps this file for the duration, it has nothing to do with server caching.
However, who caches what does depend on which way the directive is going.
As stated on MDN:
max-age=
Specifies the maximum amount of time a resource will be considered fresh. Contrary to Expires, this directive is relative to the time of the request.
And
The Cache-Control general-header field is used to specify directives for caching mechanisms in both requests and responses. Caching directives are unidirectional, meaning that a given directive in a request is not implying that the same directive is to be given in the response.
Related
It working but not affect performance. Do I use it right?
/etc/varnish/default.vcl
backend default {
.host = "127.0.0.1";
.port = "4000"; }
i was add vanish port instead 4000 in nginx config
location / {
proxy_pass http://localhost:6081;
}
My Angular application (google pagespeed) desktop performance is 99% but the mobile performance is 40-60%.
Varnish's out-of-the-box behavior respects HTTP caching best practices.
This means:
Only cache HTTP GET & HTTP HEAD calls
Don't serve responses from cache when the request contains cookie headers
Don't serve responses from cache when the request contains authorization headers
Don't store responses in cache when set-cookie headers are present
Don't store responses in cache when the cache-control header is a zero TTL or when it contains the following: no-cache, or no-store, or private
Under all circumstances Varnish will try to serve from cache or store in cache.
This is that behavior written in VCL: https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl
Adapting to the real world
Although these caching best practices make sense, they are not realistic when you look at the real world. In the real world we use cookies all the time.
That's why you'll probably have to write some VCL code to change the behavior of the cache. In order to do so, you have to be quite familiar with the HTTP endpoints of your app, but also the parts where cookies are used.
Parts of your app where cookie values are used on the server-side will have to be excluded from caching
Parts of your app where cookie values aren't used will be stored in cache
Tracking cookies that are only used at the client side will have to be stripped
How to examine what's going on
The varnishlog binary will help you understand the kind of traffic that is going through Varnish and how Varnish behaves with that traffic.
I've written an in-depth blog post about this, please have a look: https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/
Writing VCL
Once you've figured out what is causing the drop in performance, you can write VCL to mitigate. Please have a look at the docs site to learn about VCL: https://varnish-cache.org/docs/6.0/index.html
The is reference material in there, a user guide and even a tutorial.
Good luck
In some situations when my application responds with 404 Not Found code it also returns Set-Cookie directive with session identifier, but there is no Cache-Controle or Pragma directive. Does this mean that session identifier can be stored in browser cache and does this influence the security of the application? I am not sure if all responses with Set-Cookie should contain caching directives.
Whether a cookie is permanently stored in the browser or not is controlled by the Expires and Max-Age properties. Cache-Control and Pragma headers only affect page contents. So I think you're good on 404 pages even without explicit cache headers (* but see the edit below).
Session cookies should always be set without an explicit expiry date, in which case they won't normally be stored on disk and will be removed when the user quits the browser.
(Note that there are cases beyond your control when such data from memory will still be persisted to disk, like for example when a user decides to hibernate, or the computer runs out of memory and starts to swap.)
Edit (see comments):
In case of normal pages that set cookies, you usually have headers to prevent caching of sensitive info like Cache-control: no-cache,no-store,must-revalidate. This I think inherently includes not caching cookie responses either, so you don't need to explicitly set it on normal pages.
So the question is then, what cookie is set on a 404 page? If an unauthenticated user downloads the 404 page and gets a session cookie, that cookie is useless for an attacker, as the application should not be vulnerable to session fixation (the cookie value should change upon logon anyway). If it is an authenticated user, why would the application set the session cookie again on a 404 page? If it does though, you should send headers to prevent caching, that's a good catch by Skipfish. (In fact, you can do this for unauthenticated users too, but I would rate that a very low risk.)
When reading the spec for HSTS (Strict-Transport-Security), I see an injunction in section 7.2 against sending the header when accessed over http instead of https:
An HSTS Host MUST NOT include the STS header field in HTTP responses
conveyed over non-secure transport.
Why is this? What are the risks if this is violated?
The danger is to the availability of the website itself. If the website is able to respond (either now or in the future) over HTTP but not over HTTPS, it will semi-permanently prevent browsers from accessing the site:
Browser: "I want http://example.com"
ExampleCom: "You should go to the https:// URL now and for the next 3 months!"
Browser: "I want https://example.com"
ExampleCom: [nothing]
By only serving the STS header over HTTPS connections, the site guarantees that at least right now it is not pointing browsers to an inaccessible site. Of course, if the max-age is set to 3 months and the HTTPS site breaks tomorrow, the effect is the same. This is merely an incremental protection.
If your server cannot positively tell from request characteristics whether it is being accessed over HTTP vs. HTTPS, but you believe you have set up your website to only be accessible over HTTPS anyhow (e.g. due to SSL/TLS termination in an nginx proxy), it should be safe to serve the header all the time. But if you want to serve both, e.g. if you wish to serve HTTP->HTTPS redirects from your server, find out how your proxy tells you about the connection and start gating the STS header response on that.
(Thanks to Deirdre Connolly for this explanation!)
Not sure if you have a specific issue you are trying to solve, or are only asking for curiosity sake but this might be better asked on http://security.stackexchange.com
Like you I can't see the threat from the server sending this over HTTP. It doesn't really make sense, but I'm not sure if there is a risk to be honest. Except to say if you can't set up the header properly then perhaps you're not ready to implement HSTS as it can be dangerous if misconfigured!
The far bigger danger is if a browser was to process a HSTS header received over HTTP, which section 8.1 explicitly states it MUST ignore:
If an HTTP response is received over insecure transport, the UA MUST
ignore any present STS header field(s).
The risk here is that a malicious attacker (or an accidentally misconfigured header) could take a HTTP-only website offline (or the HTTP-only parts of a mixed site) if a browser incorrectly processed it. This would effectively cause a DoS for that user(s) until either the header expiries or the site implements HTTPS.
Additionally if a browser did accept this header over HTTP rather than HTTPS, it could be possible for a MITM attacker to expire the header by setting it to a max-age of 0. For example if you have a long HSTS header set on https://www.example.com but attacker was able to publish a max-age=0 header with includeSubDomain over http://example.com, and the browser incorrectly processed that, then it could effectively remove the protection HTTPS gives to your www site.
For these reasons it's very important that clients (i.e. webbrowsers) implement this correctly and ignore the HSTS header if served over HTTP and only process it over HTTPS. This could be another reason the RFC states servers must not send this over HTTP - just in case a browser implements this wrong but, to be honest, if that happens then that browser is putting all HTTP only websites at risk as a MITM attacker could add it as per above.
I have a very simple site and am setting up varnish cache on it. The server is nginx.
The cache seems to get automatically purged after 120 seconds as when I go on the site i see the Age header being reset.
Can anyone point me towards where to remove this and have pages cached indefinitely or until i manually purge varnish?
You did not mention your OS or distribution, but for example on CentOS /etc/sysconfig/varnish sets the defaults for Varnish. Amongst those defaults is VARNISH_TTL=120, which sets the default TTL to 120 seconds.
If you only wish to set a high TTL for all objects, you can just edit the default one in /etc/sysconfig/varnish.
If the backend sends to the Varnish age headers, the Varnish will consider them as a real expiration date just like a web browser and will purge it's content when the header expires.
You should make sure that the backend doesn't send cache-control headers to the varnish and only the varnish will add cache-control headers when sending data to the browsers.
I've noticed an issue on one of my sites whereby my content pages (which shouldn't set any cookies, should all be returning "Cache-Control: public" with a max-age set, and don't require authorization).
My issue is that somehow HitPass objects are making it into my cache, removing the caching from that page. I need to debug this, but am confused at exactly how best to do this particularly as I'm unable to replicate the issue.
I notice that varnish gives me an ID beside the HitPass in the varnish log. I assume this is the varnish ID for the request that generated the HitPass, and that searching back in a varnish log would tell me exactly what was wrong with the response?
Would it be better to just remove the SetCookie header from pages that I want to cache? The problem is that vcl_fetch is called even if a URL is passed... Is there any way to tell in vcl_fetch whether or not the current request has been passed by vcl_recv?
SetCookie is indeed a reason why you get hit-for-pass objects in your cache. This is an important optimization for non-prepared sites. A hit-for-pass will let varnish go straight to the backend for each of these request instead of stall them and wait for the response of the previous one.
I'm not sure as to exactly what you are wanting to debug. If it's the set-cookie, you should probably either remove that from the backend or make your own rules on what ones to cache or what one's to ignore in your cache. If you still need the set-cookie and it has unique values, hit-for-pass is the way to do that best.