I have excluded the url like this from my varnish cache
if (req.url ~ "^/folder_name/") {
return (pass);
}
but still when i access curl -I http://ip/folder_name i see the below response
HTTP/1.1 301 Moved Permanently
Date: Mon, 04 Jul 2016 08:48:46 GMT
Server: Apache/2.2.15 (CentOS)
Location: http://ip/folder_name/
Content-Length: 319
Content-Type: text/html; charset=iso-8859-1
X-Varnish: 294958
Age: 0
Via: 1.1 varnish-v4
Connection: keep-alive
Can anyone please tell me what am i doing wrong, i need to exclude
"all urls that start with /folder_name" from being cached in varnisg
Your regex ends in a slash and "http://ip/folder_name does" not - could this be it?
Related
Wondered if someone can help me. I'm trying to remove a 302 redirect from one of my sites but I can't get the code correct. It works for the first http -> https but the redirect to /tt-rss/ is still being done using 302
This is what I'm using......can anyone help?
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
>>> http://rss.oliroe.com
> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------
Status: 301 Moved Permanently
Code: 301
Server: nginx
Date: Wed, 05 Oct 2022 12:14:02 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 231
Connection: close
Location: https://rss.oliroe.com/
>>> https://rss.oliroe.com/
> --------------------------------------------
> 302 Found
> --------------------------------------------
Status: 302 Found
Code: 302
Server: nginx
Date: Wed, 05 Oct 2022 12:14:03 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
X-Powered-By: PHP/8.1.11
Location: /tt-rss/
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
>>> /tt-rss/
> --------------------------------------------
> 200 OK
> --------------------------------------------
Status: 200 OK
Code: 200
Server: nginx
Date: Wed, 05 Oct 2022 12:14:03 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
X-Powered-By: PHP/8.1.11
Cache-Control: public
Strict-Transport-Security: max-age=15768000; includeSubdomains; preload
I've solved this, the website had this code in the index.php header("Location: /tt-rss/");, I didn't realise redirects in php defaulted to 302. Changing the line to header("Location: /tt-rss/", true, 301); solved my issues. Many thanks
I'm noticing this behavior on Varnish 6.5 where it's not making backend calls per the max-age cache control origin response, if the request is not frequently requested by clients.
Below's the expected behavior I see for a cache requested every 1 second. It has 20 seconds max-age cache-control header from origin:
Request 1:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:02 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 1183681 1512819
age: 17
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
Request 2:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:04 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 891620 1512819
age: 19
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
Request 3:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:05 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 1183687 1512819
age: 20
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
Request 4:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:06 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 854039 1183688
age: 1
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
You can see the Request #4 above makes a new origin request with the cache request id being 1183688.
Now if I wait a long while and make that same request, the cache age is pretty old and varnish does not make an origin request to cache a fresh object:
Request 5 after a while:
HTTP/2 200
date: Tue, 20 Jul 2021 02:10:08 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 1512998 1183688
age: 482
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
I suppose I could start adding the Expires header from origin, but looking for explanation why varnish behaves this way if the request is infrequent. Thanks.
TTL header precedence in Varnish
Varnish does check the max-age directive, but there might be other factors can cause the TTL to be an unexpected value.
Here's the TTL precedence:
The Cache-Control header's s-maxage directive is checked.
When there's no s-maxage, Varnish will look for max-age to set its TTL.
When there's no Cache-Control header being returned, Varnish will use the Expires header to set its TTL.
When none of the above apply, Varnish will use the default_ttl runtime parameter as the TTL value. Its default value is 120 seconds.
Only then will Varnish enter vcl_backend_response, letting you change the TTL.
Any TTL being set in VCL using set beresp.ttl will get the upper hand, regardless of any other value being set via response headers.
Your specific situation
The best way to figure out what's going on is by running varnishlog and adding a filter for the URL you want to track.
Here's an example for the homepage:
varnishlog -g request -q "ReqUrl eq '/'"
The output will be extremely verbose, but will contain all the info you need.
Tags that are of particular interest are:
TTL see https://varnish-cache.org/docs/6.5/reference/vsl.html#varnish-shared-memory-logging
BerespHeader (specifically the Cache-Control backend response header)
RespHeader (specifically the Cache-Control response header)
Please also have a look at your VCL and check whether or not the TTL is changed by set beresp.ttl =.
What do I need to help you
In summary, if you want further assistance, please provide your full VCL, as well as a varnishlog extract for the transactions that is giving you to unexpected behavior.
Based on that information, we'll have a pretty good idea what's going on.
While running the curl command using both (credentials & without) I always get location output which is correct & due to this I get
HTTP/1.1 302 Found this output but in reality application is down.
any idea/help how to by pass or check the correct output.
[root#VDCLP3213 ~]# curl -Ik http://grid-net.gs.ec.ge.com/GestionHeures --user (username:Password)
HTTP/1.1 302 Found
Date: Tue, 23 Jan 2018 10:14:52 GMT
Expires: Wed, 01 Jan 1997 12:00:00 GMT
Cache-Control: private,no-store,no-cache,max-age=0
Location: https://fss.gecompany.com/fss/idp/SSO.saml2?SAMLRequest=fZHBbsIwEER%2FJfI9cRJCQRaJlMKhSLSghvbQS%2BU4S7Dk2KnXKeXva6BV6YWrPfN2ZneGvFM9Kwe318%2FwMQC64KtTGtn5IyeD1cxwlMg07wCZE6wqH1csjWLWW%2BOMMIoEJSJYJ42eG41DB7YC%2BykFvDyvcrJ3rkdGaWtlE2pwUYsRiKiFSJiOVntZ10aB20eIhp7gKd2sqy0JFj6N1PzE%2FaPsEL3VO3uuj2eCf6Gy6WlVraNT6pQEy0VO3rPxpG5EPJnuGj6eTtLdiIsk4TGI%2BI5Ps8zLEAdYanRcu5ykcTIN4yRMR9skZknGxukbCTY%2FJe%2BlbqRub2%2BkvoiQPWy3m%2FDS4hUsnht4ASlmp4TsPNhebfo2lv%2BulxQthl4fHkD56hD2xjouVdjbZkav0Jc5PXvyrOViY5QUx6BUyhzmFriDnCSEFhfL%2F%2FMX3w%3D%3D&RelayState=ss%3Amem%3A7871d5ec2f67dc36f0c796d589df7cc5f38664a8a79eb7daa3d8f80059eb8259
Connection: close
Content-Type: text/html; charset=iso-8859-1
Please help
Use the -L or --location option to follow redirects.
Note that this will still show the headers for all the intermediate sites, you'll need to parse out the last HTTP/1.1 line and its following headers to get the headers from the final target.
$ curl -s -I -L online.bridgebase.com/purchase/pay.php
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.6.2
Date: Tue, 23 Jan 2018 11:04:23 GMT
Content-Type: text/html
Content-Length: 160
Connection: close
Location: https://www.bridgebase.com/purchase/pay.php
Set-Cookie: SRV=www2.dal06.sl; path=/; domain=.bridgebase.com
HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Tue, 23 Jan 2018 11:04:23 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Vary: Accept-Encoding
X-Powered-By: PHP/5.4.45-0+deb7u11
Set-Cookie: PHPSESSID=og3dirjhdi4lhtm17iav8kgm67; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: SRV=www1.dal09.sl; path=/; domain=.bridgebase.com
imac:barmar $ curl --version
curl 7.54.0 (x86_64-apple-darwin14.5.0) libcurl/7.54.0 OpenSSL/1.0.2k zlib/1.2.5 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy
I'm seeing a situation where requests through Cloudfront have a different Cache-Control than my origin. I have Object Caching set to "Use Origin Cache Headers" and (I don't think this is relevant) Compress Objects Automatically set to "No"
I've found that if I change Object Caching to "Customize" and change the value around that does in fact change the headers returned from the CDN. That's okay and all... but I'm curious to know why with my existing settings this header isn't being passed through.
Thanks!
Compressed Request from Origin - shows Cache-Control of '31536000'
(05:34 PM) jsharpe#mbp:~ curl -I https://staging.testing.com/assets/application-0d5691ba401c3f5a305fda52745a831376545a605a6c16e50fc838fdaa567e57.css --compressed
HTTP/1.1 200 OK
Server: Cowboy
Date: Wed, 16 Aug 2017 21:34:22 GMT
Connection: keep-alive
Last-Modified: Wed, 16 Aug 2017 05:05:25 GMT
Content-Type: text/css
Cache-Control: public, max-age=31536000
Content-Encoding: gzip
Vary: Accept-Encoding, Origin
Content-Length: 33563
Via: 1.1 vegur
Compressed Request from CDN - shows Cache-Control of '86400'
(05:34 PM) jsharpe#mbp:~ curl -I https://staging-cdn.testing.com/assets/application-0d5691ba401c3f5a305fda52745a831376545a605a6c16e50fc838fdaa567e57.css --compressed
HTTP/1.1 200 OK
Content-Type: text/css
Content-Length: 33563
Connection: keep-alive
Server: Cowboy
Date: Wed, 16 Aug 2017 05:07:12 GMT
Last-Modified: Wed, 16 Aug 2017 05:05:25 GMT
Cache-Control: public, max-age=86400
Content-Encoding: gzip
Via: 1.1 vegur, 1.1 7d327ef7e21429ba6a44eb6374c976f3.cloudfront.net (CloudFront)
Vary: Accept-Encoding
Age: 59233
X-Cache: Hit from cloudfront
X-Amz-Cf-Id: TEqKbQ5ZYySY7m8rDft_MAlygEiam6gYvzrXBpS7D2DrBNbVUZ1y3Q==
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 11 years ago.
The community reviewed whether to reopen this question 8 months ago and left it closed:
Original close reason(s) were not resolved
I have a problem with my server redirecting http://www.mylesgray.com:8080/ -> http://www.mylesgray.com/.
Here are my Nginx default and fastcgi_params config files:
https://gist.github.com/1745271
https://gist.github.com/1745313
This is rather a nusance as I am trying to run a benchmark of Nginx w/ caching vs Varnish w/ caching on top of Nginx to see if there is any performance benefit of one over the other.
As such I have straight Nginx w/ caching listening on port 8080 and varnish on port 80 which forwards any non-cached requests to Nginx on localhost:8080, so obviously what I want to do is run an ab benchmark on http://www.mylesgray.com:8080/ and on http://www.mylesgray.com/ to see the difference.
Here are the results of curl -I on various addresses.
# curl -I http://www.mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
Server: nginx/0.7.65
Date: Sun, 05 Feb 2012 12:07:34 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Location: http://www.mylesgray.com/
# curl -I http://mylesgray.com
HTTP/1.1 301 Moved Permanently
Server: nginx/0.7.65
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Location: http://www.mylesgray.com/
Content-Length: 0
Date: Sun, 05 Feb 2012 12:15:51 GMT
X-Varnish: 1419774165 1419774163
Age: 15
Via: 1.1 varnish
Connection: keep-alive
# curl -I http://mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
Server: nginx/0.7.65
Date: Sun, 05 Feb 2012 12:16:08 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Location: http://www.mylesgray.com/
Then running curl -I http://www.mylesgray.com gives:
# curl -I http://www.mylesgray.com
HTTP/1.1 200 OK
Server: nginx/0.7.65
Content-Type: text/html; charset=UTF-8
X-Powered-By: PHP/5.3.2-1ubuntu4.7ppa5~lucid1
X-Pingback: http://www.mylesgray.com/xmlrpc.php
Content-Length: 5132
Date: Sun, 05 Feb 2012 12:07:29 GMT
X-Varnish: 1419774133 1419774124
Age: 30
Via: 1.1 varnish
Connection: keep-alive
So as you can see 80 is served by Varnish and 8080 by Nginx but I cannot find anywhere anything that does a 301 redirect, not in nginx.conf or in the sites-enabled/default file and I don't believe it is caused by Wordpress itself but an very much open to correction.
Please help, this is driving me nuts!
Myles
You should add a '/' at then end of your URLs. Furthermore if you run ab http://foo.com it will return you a "ab: invalid URL" error. If you do "ab -t 10 http://example.com/" everything will work fine. You should always use '/' in your URLs otherwize your webserver will try to redirect the page to the home page automatically for you which generates an undesirable extra load on the server and some extra bytes on the wire.
You web server told you what it did:
'/' is missing and something is incorrect with the port numer:
# curl -I http://www.mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
[...]
======> Location: http://www.mylesgray.com/
'www' and '/' are missing:
# curl -I http://mylesgray.com
HTTP/1.1 301 Moved Permanently
[...]
=======> Location: http://www.mylesgray.com/
[...]
'/' and 'www' are missing:
# curl -I http://mylesgray.com:8080
HTTP/1.1 301 Moved Permanently
[...]
========> Location: http://www.mylesgray.com/
'hope that helps :)
The presence of an X-Powered-By: PHP header means that wordpress is issuing the 301. It's due to wordpress forcing www.mylesgray.com. When you use a nonstandard port, user agents will generally include the port in the Host: header. Try adding
fastcgi_param HTTP_HOST $host;
with the rest of your fastcgi_param directives (or alog with your "include fastcgi_params;") and it should fix this.