Can't configure NGINX to cache the images - linux

I have been trying to cache images that are stored in nginx server, I am quite new to nginx I have installed it and configure it with php5-fpm, I have been reading many tutorial regarding caching images and php files. I have succeeded in caching the PHP files but I can't cache the images here is part of the nginx config file:
fastcgi_cache_path /etc/nginx/cache levels=1:2 keys_zone=MYAPP:100m inactive=60m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";
fastcgi_cache_use_stale error timeout invalid_header http_500;
add_header X-Cache $upstream_cache_status;
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
root /usr/share/nginx/html;
index index.html index.htm;
# Make site accessible from http://localhost/
server_name localhost;
set $skip_cache 0;
# POST requests and urls with a query string should always go to PHP
if ($request_method = POST) {
set $skip_cache 1;
}
if ($query_string != "") {
set $skip_cache 1;
}
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_cache MYAPP;
fastcgi_cache_valid 200 60m;
}
location ~* ^.+\.(jpg|jpeg|gif|png)$ {
access_log off; log_not_found off; expires 1d;
}
location ~ /\. { deny all; access_log off; log_not_found off; }
}
now when I run curl -I http://xxx.xxx.xxx.xxx/script.php I can see the md5 of the script created in /etc/nginx/cache folder and I can see the X-Cache : HIT
However, when I run curl -I http://xxx.xxx.xxx.xxx/image.jpg no file created in the /etc/nginx/cache and I have the following result in the console:
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Tue, 17 Jun 2014 08:14:37 GMT
Content-Type: image/jpeg
Content-Length: 55936
Last-Modified: Tue, 17 Jun 2014 04:30:11 GMT
Connection: keep-alive
ETag: "539fc453-da80"
Expires: Wed, 18 Jun 2014 08:14:37 GMT
Cache-Control: max-age=86400
Accept-Ranges: bytes
now it looks fine however when I run it again the expiry date will keep changing as if it is calling new image not the cache
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Tue, 17 Jun 2014 08:16:34 GMT
Content-Type: image/jpeg
Content-Length: 55936
Last-Modified: Tue, 17 Jun 2014 04:30:11 GMT
Connection: keep-alive
ETag: "539fc453-da80"
Expires: Wed, 18 Jun 2014 08:16:34 GMT
Cache-Control: max-age=86400
Accept-Ranges: bytes
my questions are:
1- should the cached images be shown in the cache folder?
2- does fastcgi capable of caching the images?
3- what can I do to fix the image caching problem?
Sorry for the beginner questions, but I can't find the answer.

Related

add security headers in NGINX while using fastcgi caching

I am using nginx with fastcgi cache. I want to use security headers on my site. I have already added add header field in my virtual host configurations but I can not get any headers unless I disable add_header X-fastcgi cache $upstream cache status in my fastcgi_main.conf file.
virualhost file :
}
include /etc/nginx/bots.d/blockbots.conf;
include /etc/nginx/bots.d/ddos.conf;
include /etc/nginx/skip_cache.conf ;
include /etc/nginx/purge_location.conf ;
include /etc/nginx/gzip_location.conf ;
include /etc/nginx/security_wp.conf;
add_header Referrer-Policy 'origin';
add_header "X-Frame-Options: sameorigin" always;
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php7.3-fpm.sock;
include "/etc/nginx/customfastcgi" ;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
# underscores_in_headers on;
client_max_body_size 256M;
include /etc/nginx/fastcgi_main.conf ;
}
}
FASTCGI_main.conf
fastcgi_no_cache $skip_cache;
fastcgi_cache phpcache;
fastcgi_cache_valid 200 1m;
fastcgi_cache_valid 301 1m;
fastcgi_cache_valid 302 1m;
fastcgi_cache_valid 307 1m;
fastcgi_cache_valid 404 1m;
fastcgi_cache_use_stale error timeout invalid_header http_500 http_503;
fastcgi_cache_min_uses 1;
fastcgi_cache_methods GET HEAD;
fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
add_header X-FastCGI-Cache $upstream_cache_status;
```
RESULT:
curl -I https://dev-kuhicbury.$domain
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Fri, 09 Oct 2020 11:39:35 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
rel="https://api.w.org/"
X-FastCGI-Cache: HIT
You've stepped onto a very common configuration pitfall of the add_header directive.
Similar to all other array-like directives in NGINX, it is only inherited, if there is no other add_header in the current context.
The typical solution is to copy-paste (through inevitable duplication), the desired headers in a specific location:
In FASTCGI_main.conf:
fastcgi_no_cache $skip_cache;
fastcgi_cache phpcache;
fastcgi_cache_valid 200 1m;
fastcgi_cache_valid 301 1m;
fastcgi_cache_valid 302 1m;
fastcgi_cache_valid 307 1m;
fastcgi_cache_valid 404 1m;
fastcgi_cache_use_stale error timeout invalid_header http_500 http_503;
fastcgi_cache_min_uses 1;
fastcgi_cache_methods GET HEAD;
fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
add_header X-FastCGI-Cache $upstream_cache_status;
add_header Referrer-Policy 'origin';
add_header "X-Frame-Options: sameorigin" always;
This unintuitive behavior of NGINX has been a source of trouble for many.
Here are some modules of interest, which address the same issue (as in, "better add_header"):
ngx_headers_more
ngx_security_headers, more applicable to your case

Cherrypy NGINX error: 403 directory index of /some/path is forbidden

I am running NGINX on an Ubuntu 18.04 x64 Digital Ocean server. I have a Cherrypy app running directly on the Ubuntu server. I am trying to use NGINX to proxy_pass to my Cherrypy app for a specific route. The proxy_pass appears to be working, but I am getting a 403 Forbidden error when I try to POST to the route. The Cherrypy route works with the Python requests POST request when I test it locally, but doesn't work when I send the request through NGINX with the proxy_pass.
Cherrypy is in a Pipenv virtual environment. To run it I run Python3 app.py.
Here's the error in the NGINX error log:
2019/10/17 20:51:50 [error] 29574#29574: *51 directory index of "/mnt/media_storage/media_root/media/monday/monday-file-upload/" is forbidden, client: 73.14.140.118, server: media.bscs.org, request: "GET /monday/monday-file-upload/ HTTP/1.1", host: "media.bscs.org"
Here's my NGINX config:
# Microcaching
proxy_cache_path /tmp/cache keys_zone=cache:10m levels=1:2 inactive=600s max_size=100m;
# Cache in browser
# Expires map
map $sent_http_content_type $expires {
default off;
text/html epoch;
text/css 30d;
application/javascript 30d;
~image/ 30d;
}
upstream apps {
server 127.0.0.1:8080;
}
server {
listen 80;
listen [::]:80;
server_name media.bscs.org;
rewrite ^/(.*) https://media.bscs.org/$1 permanent;
}
server {
listen *:443 ssl http2;
listen [::]:443 ssl http2;
server_name media.bscs.org;
root /mnt/media_storage/media_root/media;
charset utf-8;
client_max_body_size 1000M;
# Gzip/compress text-based assets
gzip on;
gzip_http_version 1.0;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/html text/css application/json application/x-javascript text/xml application/xml text/javascript application/javascript image/svg+xml;
gzip_disable "MSIE [1-6]\.";
# make sure gzip does not lose large gzipped js or css files
# see http://blog.leetsoft.com/2007/7/25/nginx-gzip-ssl
gzip_buffers 16 8k;
# Microcaching
proxy_cache cache;
proxy_cache_valid 200 1s;
# Cache in browser
expires $expires;
ssl on;
ssl_ciphers "my-cipher";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Content-Type-Options nosniff;
add_header 'Access-Control-Allow-Origin' '*';
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver_timeout 5s;
ssl_certificate /etc/nginx/ssl/cert_chain.crt;
ssl_certificate_key /etc/nginx/ssl/STAR.bscs.org.key;
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
location = /favicon.ico {
access_log off;
log_not_found off;
sendfile on;
sendfile_max_chunk 1m;
}
location ~* \.(gif|jpg|jpeg|png|js|css)$ {
log_not_found off;
access_log off;
sendfile on;
sendfile_max_chunk 1m;
}
location /media/ {
alias /mnt/media_storage/media_root/media/;
location /media/monday/monday-file-upload/ {
alias /mnt/media_storage/media_root/media/monday/monday-file-upload/;
proxy_pass http://apps/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}
# Redirects
rewrite ^/tech-report/2018-1/2018-1.html$ https://bscs.org/resources/reports/designing-citizen-science-for-both-science-and-education-a-workshop-report/$1 permanent;
}
Here's my post request in with Python requests:
def uploadFileToMediaServer(uploaded_file_local_path):
with open(uploaded_file_local_path, 'rb') as f:
files = {'uploaded_file': f}
r = requests.post('https://media.bscs.org/monday/monday-file-upload', files=files)
print(r.request.url, file=sys.stderr)
print(r.request.headers, file=sys.stderr)
return r
Here's my Cherrypy app:
import cherrypy
from cherrypy.process.plugins import Daemonizer
config = {
'global': {
'server.socket_host': '127.0.0.1',
'server.socket_port': 8080,
'server.thread_pool': 8,
'server.max_request_body_size': 0,
'server.socket_timeout': 60
}
}
class App:
#cherrypy.expose
def index(self, uploaded_file):
try:
with open('../uploads/{}'.format(uploaded_file.filename), 'wb') as f:
while True:
data = uploaded_file.file.read(8192)
if not data:
return {'message': 'File failed to upload'}
f.write(data)
return {'message': 'File uploaded successfully'}
except Exception:
cherrypy.log(Exception, traceback=True)
if __name__ == '__main__':
d = Daemonizer(cherrypy.engine)
d.subscribe()
cherrypy.tree.mount(App(), "/", config)
cherrypy.engine.start()
cherrypy.engine.block()

Nginx not Caching my Application

I'm trying to setup the caching of my ExpressApp and its rendered Jade files with Nginx.
When I have a look at the pm2 logs of my application while running the google Pagespeed Insight test, I see that the Images and HTML are still served by the app instead of the Nginx proxy.
Two Questions:
How do I have to change my configuration to have the desired effect?
How do I effectively validate that the Setup/Caching is working?
Thanks!
My current Nginx Configuration:
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=backcache:8m max_size=50m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
proxy_cache_valid 200 302 10m;
proxy_cache_valid 404 1m;
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
expires 1w;
proxy_cache backcache;
#proxy_cache_bypass $http_cache_control;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://localhost:3030;
}
}
Headers:
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: ###
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET / HTTP/1.1
> Host: ###
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.10.0 (Ubuntu)
< Date: Thu, 24 Nov 2016 13:15:37 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 66761
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
< ETag: W/"104c9-8+4lo531tUk9k3SJq4sPHg"
< Expires: Thu, 01 Dec 2016 13:15:37 GMT
< Cache-Control: max-age=604800
< X-Proxy-Cache: MISS
Headers of two consecutive Requests to an Image
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:02 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:02 GMT
X-Proxy-Cache: MISS
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:08 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:08 GMT
X-Proxy-Cache: MISS
The headers you show as the request headers are most probably not the original ones, because it says "Provisional headers shown".
The actual request headers probably contain the "cache-control" header and your nginx configuration uses that header to bypass the cache, as indicated by the 'X-Proxy-Cache:BYPASS' response header.
If there is no reason to keep it, you should remove the proxy_cache_bypass-directive from your config.

Nginx set ssl for URLs

Good day. I got nginx server and it runs https connections.
For now all URLs run with https. All i need is - to exclude some URLs from https, so they could be accessed with simple http.
Here is my NGINX config file:
server {
listen 80;
server_name my-fin.ru www.my-fin.ru;
root /usr/server/finance/abacus/webapp;
location ~ ^/.+\.(eot|ttf|woff)$ {
expires max;
add_header Cache-Control public;
add_header Access-Control-Allow-Origin *;
}
location ~ ^/.+\.(ico|jpg|jpeg|gif|pdf|jar|png|js|css|txt|epf|svg)$ {
expires max;
add_header Cache-Control public;
}
location / {
return 301 https://my-fin.ru;
}
}
server {
listen *:443;
server_name my-fin.ru;
client_max_body_size 10m;
gzip on;
gzip_min_length 500;
gzip_buffers 4 8k;
gzip_types text/plain text/xml application/xml application/x-javascript text/javascript text/css text/json application/json;
access_log /var/log/nginx/finance.access.log;
error_log /var/log/nginx/finance.error.log;
ssl on;
ssl_certificate /usr/server/myfin.crt;
ssl_certificate_key /usr/server/myfin.key;
charset utf-8;
root /usr/server/finance/abacus/webapp;
location ~ ^/.+\.(eot|ttf|woff)$ {
expires max;
add_header Cache-Control public;
add_header Access-Control-Allow-Origin *;
}
location ~ ^/.+\.(ico|jpg|jpeg|gif|pdf|jar|png|js|css|txt|epf|svg)$ {
expires max;
add_header Cache-Control public;
}
location / {
# give site more time to respond
proxy_read_timeout 120;
proxy_pass http://127.0.0.1:8087;
proxy_redirect http:// $scheme://;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr ;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for ;
}
}
Please help to configure nginx.
According to this comment, Here's how i would do it.
## non https server
server {
#non ssl server
listen 80;
server_name example.com;
root /path/to/root;
location /features {
#handle /features
}
location /info {
# handle /info
}
location /help {
#handle /help
}
location /
return 301 https://example.com$request_uri;
}
}
## https server
server {
# handle ssl
listen 443 ssl;
server_name example.com subdomain1.example.com;
root /path/to/root;
location ~ /(features|help|info) {
# redirect those 3 subfolders to http
return 301 http://example.com$request_uri;
}
location / {
#handle ssl requests;
}
}
## https subdomain
server {
listen 443 ssl;
server_name subdomain2.example.com;
root /path/to/root;
location ~ /(features|help|info) {
# redirect those 3 subfolders to http
return 301 http://example.com$request_uri;
}
location / {
# subdomain handling
}
}
Please note that https wont work on subdomains unless you have a wildcard SSL certificate, otherwise the browser will issue a warning.

Why isn't Varnish sending 304 unmodified when If-Modified-Since header is sent?

When sending a GET request directly to the backend with If-Modified-Since: Wed, 15 Feb 2012 07:25:00 CET set, Apache correctly returns a 304 with no content.
When I send the same request through Varnish 3.0.2, it responds with a 200 and resends all the content even though the client already has it. Obviously, this isn't a good use of bandwidth. My understanding is that Varnish supports intelligent handling of this header and should be sending a 304, so I figure I'd done something wrong with my .vcl file.
Varnishlog gives this:
16 SessionOpen c 84.97.17.233 64416 :80
16 ReqStart c 84.97.17.233 64416 1597323690
16 RxRequest c GET
16 RxURL c /fr/CS/CS_AU-Maboreke-6-6-2004.pdf
16 RxProtocol c HTTP/1.0
16 RxHeader c Host: www.quotaproject.org
16 RxHeader c User-Agent: Sprawk/1.3 (http://www.sprawk.com/)
16 RxHeader c Accept: */*
16 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
16 RxHeader c Connection: close
16 RxHeader c If-Modified-Since: Wed, 15 Feb 2012 07:25:00 CET
16 VCL_call c recv lookup
16 VCL_call c hash
16 Hash c /fr/CS/CS_AU-Maboreke-6-6-2004.pdf
16 Hash c www.quotaproject.org
16 VCL_return c hash
16 Hit c 1597322756
16 VCL_call c hit
16 VCL_acl c NO_MATCH CTRLF5
16 VCL_return c deliver
16 VCL_call c deliver deliver
16 TxProtocol c HTTP/1.1
16 TxStatus c 200
16 TxResponse c OK
16 TxHeader c Server: Apache
16 TxHeader c Last-Modified: Wed, 09 Jun 2004 16:07:50 GMT
16 TxHeader c Vary: Accept-Encoding
16 TxHeader c Content-Type: application/pdf
16 TxHeader c Date: Wed, 22 Feb 2012 18:25:05 GMT
16 TxHeader c Age: 12432
16 TxHeader c Connection: close
16 Gzip c U D - 107685 115763 80 796748 861415
16 Length c 98304
16 ReqEnd c 1597323690 1329935105.713264704 1329935106.208528996 0.000071526 0.000068426 0.495195866
16 SessionClose c EOF mode
16 StatSess c 84.97.17.233 64416 0 1 1 0 0 0 203 98304
If I understand this correctly, the object is already in Varnish's cache so it doesn't need to contact the backend, but it already knows the Last-Modified so why would it not respond with 304?
And here's my VCL file:
backend idea {
# .host = "www.idea.int";
.host = "83.145.60.235"; # IDEA's public website IP
.port = "80";
}
backend qp {
# .host = "www.quotaproject.org";
.host = "83.145.60.235"; # IDEA's public website IP
.port = "80";
}
#
#Below is a commented-out copy of the default VCL logic. If you
#redefine any of these subroutines, the built-in logic will be
#appended to your code.
#
sub vcl_recv {
# force domain so that Apache handles the VH correctly
if (req.http.host ~ "^qp" || req.http.host ~ "quotaproject.org$") {
set req.http.Host = "www.quotaproject.org";
set req.backend = qp;
} else {
# default to idea.int
set req.http.Host = "www.idea.int";
set req.backend = idea;
}
# Before anything else we need to fix gzip compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} else if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else if (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unknown algorithm
remove req.http.Accept-Encoding;
}
}
# ajax requests bypass cache. TODO: Make sure you Javascript implementation for AJAX actually sets XMLHttpRequest
if (req.http.X-Requested-With == "XMLHttpRequest") {
return(pass);
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# Purge everything url - this isn't the squid way, but works
if (req.url ~ "^/varnishpurge") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
if (req.url == "/varnishpurge") {
ban("req.http.host == " + req.http.host + " && req.url ~ ^/");
error 841 "Purged site.";
}
else {
ban("req.http.host == " + req.http.host + " && req.url ~ ^" + regsub( req.url, "^/varnishpurge(.*)$", "\1" ) + "$");
error 842 "Purged page.";
}
}
# spoof the client IP (taken from http://utvbloggen.se/snabb-guide-till-varnish/)
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
# Force delivery from cache even if other things indicate otherwise
if (req.url ~ "\.(flv)") {
# pipe flash start away
return(pipe);
}
if (req.url ~ "\.(jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar|rar|odc|odb|odf|odg|odi|odp|ods|odt|sxc|sxd|sxi|sxw|dmg|torrent|deb|msi|iso|rpm)$") {
# cookies are irrelevant here
unset req.http.Cookie;
unset req.http.Authorization;
}
# Force short-circuit to the real site for these dynamic pages
if (req.url ~ "/customcf/" || req.url ~ "/uid/editData.cfm" || req.url ~ "^/private/") {
return(pass);
}
# Remove user agent, since Apache will server these resources the same way
if (req.http.User-Agent) {
set req.http.User-Agent = "";
}
if (req.http.Cookie) {
# removes all cookies named __utm? (utma, utmb...) - tracking thing
set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", "\1");
# remove cStates for RHM boxes (the server doesn't need to know these, JS will handle this client-side)
set req.http.cookie = regsub(req.http.cookie, "(; )?cStates=[^;]*", ""); #cStates might sometimes have a blank value
# remove ColdFusion session cookie stuff
if (!req.url ~ "^/publications/" && !req.url ~ "^/uid/admin/") {
set req.http.cookie = regsub(req.http.cookie, "(; )?CFID=[^;]+", "");
set req.http.cookie = regsub(req.http.cookie, "(; )?CFTOKEN=[^;]+", "");
}
# Remove the cookie header if it's empty after cleanup
if (req.http.cookie ~ "^;? *$") {
# The only cookie data left is a semicolon or spaces
remove req.http.cookie;
}
}
}
#
# Called when the requested object was not found in the cache
#
sub vcl_hit {
# Allow administrators to easily flush the cache from their browser
if (client.ip ~ CTRLF5) {
if (req.http.pragma ~ "no-cache" || req.http.Cache-Control ~ "no-cache") {
set obj.ttl = 0s;
return(pass);
}
}
}
#
# Called when the requested object has been retrieved from the
# backend, or the request to the backend has failed
#
sub vcl_fetch {
set beresp.grace = 1h;
# strip the cookie before the image is inserted into cache.
if (req.url ~ "\.(jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar|rar|odc|odb|odf|odg|odi|odp|ods|odt|sxc|sxd|sxi|sxw|dmg|torrent|deb|msi|iso|rpm)$") {
remove beresp.http.set-cookie;
set beresp.ttl = 100w;
}
# Remove CF session cookies for everything but the publications subsite
if (!req.url ~ "^/publications/" && !req.url ~ "/customcf/" && !req.url ~ "^/uid/admin/" && !req.url ~ "^/uid/editData.cfm") {
remove beresp.http.set-cookie;
}
if (beresp.ttl < 48h) {
set beresp.ttl = 48h;
}
}
#
# Called before a cached object is delivered to the client
#
sub vcl_deliver {
# We'll be hiding some headers added by Varnish. We want to make sure people are not seeing we're using Varnish.
remove resp.http.X-Varnish;
remove resp.http.Via;
# We'd like to hide the X-Powered-By headers. Nobody has to know we can run PHP and have version xyz of it.
remove resp.http.X-Powered-By;
}
Can anyone see the problem or problems?
Update: According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.3
Note: When handling an If-Modified-Since header field, some
servers will use an exact date comparison function, rather than a
less-than function, for deciding whether to send a 304 (Not
Modified) response.
It seems this may be Varnish's behaviour. I'm sending another date which is previous to the real file's last modified date, but not exactly what is cached in Varnish.
The problem is the non-GMT time zone in the If-Modified-Since request header:
If-Modified-Since: Wed, 15 Feb 2012 07:25:00 CET
According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3
All HTTP date/time stamps MUST be represented in Greenwich Mean Time (GMT), without exception.
Varnish implements this as a strict requirement, whereas Apache handles nonstandard date formats more robustly. This is why you observed different behavior when querying Apache directly.
Since this question is still open with no answers and several up votes, I'll post an answer.
This does not seem to be an issue with Varnish 3.0.0 (which we are using) or the current version of Varnish you are running on your site.
200 OK response when requesting content with an expired If-Modified-Since header:
# curl -z "Wed, 09 Jun 2010 16:07:50 GMT" --head "www.quotaproject.org/robots.txt"
HTTP/1.1 200 OK
Server: Apache
Last-Modified: Tue, 22 Jan 2013 13:23:41 GMT
Vary: Accept-Encoding
Cache-Control: public
Content-Type: text/plain; charset=UTF-8
Date: Mon, 25 Nov 2013 15:00:45 GMT
Age: 69236
Connection: keep-alive
X-Cache: HIT
304 response when If-Modified-Since is after Last-Modified date:
# curl -z "Wed, 09 Jun 2013 16:07:50 GMT" --head "www.quotaproject.org/robots.txt"
HTTP/1.1 304 Not Modified
Server: Apache
Last-Modified: Tue, 22 Jan 2013 13:23:41 GMT
Vary: Accept-Encoding
Cache-Control: public
Content-Type: text/plain; charset=UTF-8
Date: Mon, 25 Nov 2013 15:00:52 GMT
Age: 69243
Connection: keep-alive
X-Cache: HIT
The same with the example you gave in varnishlog output:
# curl -z "Wed, 15 Feb 2012 07:25:00 CET" --head "www.quotaproject.org/fr/CS/CS_AU-Maboreke-6-6-2004.pdf"
HTTP/1.1 304 Not Modified
Server: Apache
Last-Modified: Wed, 09 Jun 2004 16:07:50 GMT
Cache-Control: public
Content-Type: application/pdf
Accept-Ranges: bytes
Date: Mon, 25 Nov 2013 15:08:48 GMT
Age: 335802
Connection: keep-alive
X-Cache: HIT
I would say Varnish works as expected. Maybe this was a problem with the Varnish build you were using or there was something amiss with the testing methodology. I couldn't see any problems with your VCL either.

Resources