I want to filter the access logs on:
User-agent contains "Googlebot"
Referer contains "google"
I use this varnishlog command:
varnishlog -q "ReqHeader ~ 'User-Agent.*Googlebot'"
this is my output:
* << Request >> 564834158
- Begin req 564834144 rxreq
- Timestamp Start: 1626180326.557796 0.000000 0.000000
- Timestamp Req: 1626180326.557796 0.000000 0.000000
- ReqStart xx.xxx.xx.xxx 45253
- ReqMethod GET
- ReqURL /xx/yy/xxx-yyy
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.yyyyyy.com
- ReqHeader AMP-Cache-Transform: google;v="1..7"
- ReqHeader Connection: keep-alive
- ReqHeader Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader From: googlebot(at)googlebot.com
- ReqHeader User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader If-Modified-Since: Mon, 12 Jul 2021 10:56:33 GMT
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: xx.xxx.xx.xxx
- VCL_call RECV
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqUnset Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader accept: application/vnd.fos.user-context-hash
- ReqHeader X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqURL /userContext.php
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 561267092 34.193790 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Tue, 13 Jul 2021 12:44:00 GMT
- RespHeader Server: Apache
- RespHeader Access-Control-Allow-Origin: https://www.yyyyyy.com
- RespHeader Access-Control-Allow-Credentials: true
- RespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT
- RespHeader Pragma: no-cache
- RespHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- RespHeader Cache-Control: max-age=120, s-max-age=120
- RespHeader Vary: Cookie
- RespHeader Set-Cookie: mainMenuId=1002323; expires=Thu, 12-Aug-2021 12:44:00 GMT; Max-Age=2592000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: PHPSESSID=qhn0l4m4f18g4g3cer3d8e5pg3; path=/; domain=.xxxx.yyyy; HttpOnly
- RespHeader Set-Cookie: connexion_id=1898574699; expires=Sun, 09-Jan-2022 12:44:00 GMT; Max-Age=15552000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: connexion_id=1898574699; expires=Sun, 09-Jan-2022 12:44:00 GMT; Max-Age=15552000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: connexion_id=1898574699; expires=Sun, 09-Jan-2022 12:44:00 GMT; Max-Age=15552000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: membre_statut_id=60; expires=Mon, 11-Oct-2021 12:44:00 GMT; Max-Age=7776000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: statutSolde=pub; expires=Thu, 12-Aug-2021 12:44:00 GMT; Max-Age=2592000; path=/; domain=.xxxx.yyyy
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Content-Type: application/vnd.fos.user-context-hash
- RespHeader X-Varnish: 564834158 561267092
- RespHeader Age: 85
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespHeader X-Cache-Hits: 261
- ReqHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- VCL_return restart
- Timestamp Process: 1626180326.558006 0.000209 0.000209
- Timestamp Restart: 1626180326.558009 0.000213 0.000003
- Link req 564834159 restart
- End
* << Request >> 562977018
- Begin req 562977017 restart
- Timestamp Start: 1626180326.055452 0.000223 0.000000
- ReqStart xx.xxx.xx.xxx 47603
- ReqMethod GET
- ReqURL /userContext.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.yyyyyy.com
- ReqHeader Connection: keep-alive
- ReqHeader From: googlebot(at)googlebot.com
- ReqHeader User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: xx.xxx.xx.xxx
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader accept: application/vnd.fos.user-context-hash
- ReqHeader X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- VCL_call RECV
- ReqUnset Accept-Encoding: gzip
- ReqHeader Accept-Encoding: gzip
- ReqURL /be/fr/isseymiyake-sac-seau-lucent-rose-femme-4850263
- ReqUnset X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqUnset accept: application/vnd.fos.user-context-hash
- ReqHeader accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqUnset X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 562977019 fetch
- Timestamp Fetch: 1626180328.191066 2.135837 2.135614
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Tue, 13 Jul 2021 12:45:26 GMT
- RespHeader Access-Control-Allow-Origin: https://www.yyyyyy.com
- RespHeader Access-Control-Allow-Credentials: true
- RespHeader isVarnish: 1
- RespHeader Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader templateName: FICHE_PRODUIT_TPL_ID
- RespHeader Content-Encoding: gzip
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Content-Length: 41620
- RespHeader Content-Type: text/html; charset=ISO-8859-1
- RespHeader Cache-Control: max-age=4
- RespHeader X-Varnish: 562977018
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespHeader X-Cache: MISS
- RespUnset Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader Vary: ,Accept-Encoding
- RespUnset Vary: ,Accept-Encoding
- RespHeader Vary: Accept-Encoding
- VCL_return deliver
- Timestamp Process: 1626180328.191101 2.135873 0.000036
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1626180328.192502 2.137274 0.001400
- ReqAcct 407 0 407 499 41620 42119
- End
* << Request >> 564834159
- Begin req 564834158 restart
- Timestamp Start: 1626180326.558009 0.000213 0.000000
- ReqStart xx.xxx.xx.xxx 45253
- ReqMethod GET
- ReqURL /userContext.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.yyyyyy.com
- ReqHeader AMP-Cache-Transform: google;v="1..7"
- ReqHeader Connection: keep-alive
- ReqHeader From: googlebot(at)googlebot.com
- ReqHeader User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader If-Modified-Since: Mon, 12 Jul 2021 10:56:33 GMT
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: xx.xxx.xx.xxx
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader accept: application/vnd.fos.user-context-hash
- ReqHeader X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- VCL_call RECV
- ReqUnset Accept-Encoding: gzip
- ReqHeader Accept-Encoding: gzip
- ReqURL /xx/yy/xxx-yyy
- ReqUnset X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqUnset accept: application/vnd.fos.user-context-hash
- ReqHeader accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqUnset X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 564834160 fetch
- Timestamp Fetch: 1626180328.405520 1.847724 1.847511
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Tue, 13 Jul 2021 12:45:26 GMT
- RespHeader Access-Control-Allow-Origin: https://www.yyyyyy.com
- RespHeader Access-Control-Allow-Credentials: true
- RespHeader isVarnish: 1
- RespHeader Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader templateName: CARROUSEL_MARQUE_TPL_ID
- RespHeader Content-Encoding: gzip
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Content-Type: text/html; charset=ISO-8859-1
- RespHeader Cache-Control: max-age=4
- RespHeader X-Varnish: 564834159
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespHeader X-Cache: MISS
- RespUnset Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader Vary: ,Accept-Encoding
- RespUnset Vary: ,Accept-Encoding
- RespHeader Vary: Accept-Encoding
- VCL_return deliver
- Timestamp Process: 1626180328.405548 1.847751 0.000027
- RespHeader Accept-Ranges: bytes
- RespHeader Transfer-Encoding: chunked
- RespHeader Connection: keep-alive
- Timestamp Resp: 1626180328.409407 1.851611 0.003860
- ReqAcct 586 0 586 507 63033 63540
- End
I want this Data Format:
Date / time or timestamp
Hostname
IP client
Referer
Path
User-agent
Status code
Bytes
LoadTime (in Milliseconds)
The scheme (http or htps)
Varnishlog
If you want to use varnishlog, this is the command you need:
varnishlog -c -g request -I Timestamp:Start -I ReqHeader:Host \
-i ReqStart -I ReqHeader:Referer -i ReqUrl -I ReqHeader:User-Agent \
-i RespStatus -i ReqAcct -I Timestamp:Resp -I ReqHeader:X-Forwarded-Proto \
-q "ReqHeader:User-Agent ~ 'Googlebot' and ReqHeader:Referer ~ 'google'"
Here's some potential output for this command:
* << Request >> 46
- Timestamp Start: 1626253862.684183 0.000000 0.000000
- ReqStart 172.17.0.1 58432 a0
- ReqURL /
- ReqHeader Host: localhost
- ReqHeader X-Forwarded-Proto: https
- ReqHeader Referer: google
- ReqHeader User-Agent: Googlebot
- RespStatus 200
- Timestamp Resp: 1626253862.684398 0.000215 0.000139
- ReqAcct 114 0 114 326 612 938
See https://varnish-cache.org/docs/trunk/reference/vsl.html for the meaning of every VSL field. See https://varnish-cache.org/docs/trunk/reference/vsl-query.html for more information on VSL queries.
The output is multiline and not so easy to parse. On the other hand, it's very verbose. As usual: it's a tradeoff.
Varnishncsa
It's also possible to use varnishncsa, which is the Apache-style logformat. The output is single-line and less verbose.
The following command can be used to retrieve the information you need:
varnishncsa -c -g request \
-q "ReqHeader:User-Agent ~ 'Googlebot' and ReqHeader:Referer ~ 'google'" \
-F "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %T"
The format you get is an extension of the standard format, which is the following:
%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
I just added %T to display the time it took to retrieve the content. This is expressed in seconds, and will be equal to 0 most of the time. If you want to have microsecond precision, you can use %{VSL:Timestamp:Resp[2]}x instead.
Here's some potential output for this command:
172.17.0.1 - - [14/Jul/2021:09:10:23 +0000] "GET http://localhost/ HTTP/1.1" 200 612 "google" "Googlebot" 0
However, if you want the exact order and fields you specified in the list, you'll end up with the following varnishncsa command:
varnishncsa -c -g request \
-q "ReqHeader:User-Agent ~ 'Googlebot' and ReqHeader:Referer ~ 'google'" \
-F "%t %{Host}i %h %{Referer}i %U %{User-agent}i %s %b %{VSL:Timestamp:Resp[2]}x %{X-Forwarded-Proto}i"
FYI: I'm retrieving the scheme via the X-Forwarded-Proto.
Here's some potential output:
[14/Jul/2021:09:09:19 +0000] localhost 172.17.0.1 google / Googlebot 200 612 0.001057 https
See https://varnish-cache.org/docs/trunk/reference/varnishncsa.html#display-varnish-logs-in-apache-ncsa-combined-log-format for more information.
Related
I'm trying to do a soft purge only for certain req.url values, all other invalidations are managed with a ban.
While ban is working, purge.soft(0s,30s) does not modify anythig in the cache, TTL remains the standard (7200s) and the cache remains active.
What am I doing wrong?
Full VCL code:
https://pastebin.com/QLmBh0hw
GET request log:
* << Request >> 524366
- Begin req 524315 rxreq
- Timestamp Start: 1664176259.698910 0.000000 0.000000
- Timestamp Req: 1664176259.698910 0.000000 0.000000
- ReqStart 172.16.1.194 23952 a0
- ReqMethod GET
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqProtocol HTTP/1.1
- ReqHeader X-Forwarded-For: 84.247.245.84, 130.176.221.197
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader Host: test-prod.luminalpark.com
- ReqHeader X-Amzn-Trace-Id: Root=1-63315083-50faab2c2fbacda82eba1f9c
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
- ReqHeader X-Amz-Cf-Id: ToerPgpur_0Y4SvPsU3hkSSzfK-PywWzlX-nlhnQasaji9IGXsFb5g==
- ReqHeader Via: 2.0 0f03c98743d9ffe79330c1f694241fc2.cloudfront.net (CloudFront)
- ReqHeader Cookie: _dc_gtm_UA-830149-18=1; _fbp=fb.1.1663062717375.1186628968; _ga=GA1.1.2073680112.1662976397; _ga_499EG6CXZD=GS1.1.1664175973.166.1.1664176257.0.0.0; _gcl_au=1.1.1770949614.1662976397; _gid=GA1.2.1336671677.1662976397; _hjSessionUser_21623=eyJpZCI
- ReqHeader Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
- ReqHeader Referer: https://test-prod.luminalpark.com/it-it/
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader cache-control: max-age=0
- ReqHeader sec-ch-ua: "Google Chrome";v="105", "Not)A;Brand";v="8", "Chromium";v="105"
- ReqHeader sec-ch-ua-mobile: ?0
- ReqHeader sec-ch-ua-platform: "Windows"
- ReqHeader dnt: 1
- ReqHeader upgrade-insecure-requests: 1
- ReqHeader sec-fetch-site: same-origin
- ReqHeader sec-fetch-mode: navigate
- ReqHeader sec-fetch-user: ?1
- ReqHeader sec-fetch-dest: document
- ReqHeader CloudFront-Viewer-HTTP-Version: 2.0
- ReqHeader CloudFront-Forwarded-Proto: https
- ReqHeader CloudFront-Viewer-Address: 84.247.245.84:61581
- ReqHeader CloudFront-Viewer-TLS: TLSv1.3:TLS_AES_128_GCM_SHA256:connectionReused
- ReqHeader X-Cloudfront-Origin: VC5ZNQ588QNE3S
- ReqUnset X-Forwarded-For: 84.247.245.84, 130.176.221.197
- ReqHeader X-Forwarded-For: 84.247.245.84, 130.176.221.197, 172.16.1.194
- VCL_call RECV
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 884747 7171.150022 60.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Server: nginx/1.18.0 (Ubuntu)
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader Cache-Control: must-revalidate, public, s-maxage=7200
- RespHeader Date: Mon, 26 Sep 2022 07:10:30 GMT
- RespHeader Strict-Transport-Security: max-age=31536000; includeSubDomains
- RespHeader X-Frame-Options: deny
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Referrer-Policy: strict-origin-when-cross-origin
- RespHeader sw-invalidation-states:
- RespHeader Set-Cookie: sw-states=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; secure; httponly; samesite=lax
- RespHeader Set-Cookie: sw-cache-hash=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; httponly
- RespHeader Content-Encoding: gzip
- RespHeader Vary: Accept-Encoding
- RespHeader x-url: /it-it/prodotti/catene/catene-di-luci/
- RespHeader X-Cacheable: YES
- RespHeader X-Varnish: 524366 884747
- RespHeader Age: 28
- RespHeader Via: 1.1 varnish (Varnish/6.0)
- VCL_call DELIVER
- RespUnset Cache-Control: must-revalidate, public, s-maxage=7200
- RespHeader Cache-Control: max-age=0, private
- RespUnset Set-Cookie: sw-states=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; secure; httponly; samesite=lax
- RespUnset Set-Cookie: sw-cache-hash=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; httponly
- RespHeader X-Cache: HIT
- RespUnset sw-invalidation-states:
- RespHeader X-Cache-Hits: 2
- VCL_return deliver
- Timestamp Process: 1664176259.698985 0.000076 0.000076
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 110181
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664176259.699106 0.000196 0.000120
- ReqAcct 3158 0 3158 612 110181 110793
- End
and here's varnishlog during PURGE
curl -XPURGE http://<varnish-host>/it-it/prodotti/catene/catene-di-luci/
* << Request >> 622680
- Begin req 622679 rxreq
- Timestamp Start: 1664176453.712271 0.000000 0.000000
- Timestamp Req: 1664176453.712271 0.000000 0.000000
- ReqStart 172.16.2.136 47728 a0
- ReqMethod PURGE
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqProtocol HTTP/1.1
- ReqHeader Host: 172.16.3.37
- ReqHeader User-Agent: curl/7.68.0
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 172.16.2.136
- VCL_call RECV
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- VCL_acl MATCH purgers "ecommerce-node1-prod"
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- ReqHeader purged: 0
- VCL_return synth
- Timestamp Process: 1664176453.712326 0.000055 0.000055
- RespProtocol HTTP/1.1
- RespStatus 404
- RespReason Not Found
- RespReason Not Found
- RespHeader Date: Mon, 26 Sep 2022 07:14:13 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 622680
- VCL_call SYNTH
- RespHeader purged: 0
- VCL_return deliver
- RespHeader Content-Length: 0
- Storage malloc Transient
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664176453.712364 0.000093 0.000038
- ReqAcct 114 0 114 153 0 153
- End
I've tested it myself and I'm not experiencing any issues.
The VCL code
Here's the standard vmod_purge example VCL code I took from https://varnish-cache.org/docs/6.0/reference/vmod_generated.html#example and adapted:
I removed the ACL check
I converted purge.hard() to purge.soft(0s,30s)
sub vcl_recv {
if (req.method == "PURGE") {
return (hash);
}
}
sub my_purge {
set req.http.purged = purge.soft(0s,30s);
if (req.http.purged == "0") {
return (synth(404));
}
else {
return (synth(200));
}
}
sub vcl_hit {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_miss {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_synth {
if (req.method == "PURGE") {
if (req.http.purged) {
set resp.http.purged = req.http.purged;
}
return (deliver);
}
}
What soft purging does
The way you've configured soft purging will ensure the object is considered stale, however because you set the grace time to 30s, async revalidation may still happen until 30 seconds past the lifetime of the object.
Example HTTP calls
Here are some example calls that illustrate what happens when soft purging takes place.
Step 1: call the endpoint and receive a fresh object
➜ ~ curl -I localhost
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 595
ETag: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
Date: Thu, 22 Sep 2022 07:51:22 GMT
X-Varnish: 65546 12
Age: 18
Via: 1.1 varnish (Varnish/6.6)
Accept-Ranges: bytes
Connection: keep-alive
As you can see we're calling the http://localhost endpoint and get a result with an Age: 18 header. This means the object has been cached for 18 seconds.
Step 2: purge
➜ ~ curl -XPURGE -I localhost
HTTP/1.1 200 OK
Date: Thu, 22 Sep 2022 07:51:42 GMT
Server: Varnish
X-Varnish: 32784
purged: 1
Content-Length: 0
Accept-Ranges: bytes
Connection: keep-alive
In step 2 we're performing the soft purge. The purged: 1 header implies that 1 object was purged.
Step 3: grace mode kicks in
➜ ~ curl -I localhost
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 595
ETag: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
Date: Thu, 22 Sep 2022 07:51:22 GMT
X-Varnish: 65552 12
Age: 26
Via: 1.1 varnish (Varnish/6.6)
Accept-Ranges: bytes
Connection: keep-alive
After the purge the we're still seeing cached content being served because the Age: 26 header that implies the object has been cached for 26 seconds.
But the output from varnishlog -g request show that the stale content is served while an asynchronous fetch happens for the new content. This is a direct result of calling purge.soft(0s, 30s):
* << Request >> 65552
- Begin req 65551 rxreq
- Timestamp Start: 1663833108.524685 0.000000 0.000000
- Timestamp Req: 1663833108.524685 0.000000 0.000000
- VCL_use boot
- ReqStart 172.24.0.1 58300 http
- ReqMethod HEAD
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.79.1
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 172.24.0.1
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 12 -1.375188 30.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- Link bereq 65553 bgfetch
- Timestamp Fetch: 1663833108.524864 0.000179 0.000179
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Content-Type: application/json; charset=utf-8
- RespHeader Content-Length: 595
- RespHeader ETag: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
- RespHeader Date: Thu, 22 Sep 2022 07:51:22 GMT
- RespHeader X-Varnish: 65552 12
- RespHeader Age: 26
- RespHeader Via: 1.1 varnish (Varnish/6.6)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1663833108.524876 0.000191 0.000011
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1663833108.524937 0.000252 0.000061
- ReqAcct 74 0 74 275 0 275
** << BeReq >> 65553
-- Begin bereq 65552 bgfetch
-- VCL_use boot
-- Timestamp Start: 1663833108.524815 0.000000 0.000000
-- BereqMethod HEAD
-- BereqURL /
-- BereqProtocol HTTP/1.1
-- BereqHeader Host: localhost
-- BereqHeader User-Agent: curl/7.79.1
-- BereqHeader Accept: */*
-- BereqHeader X-Forwarded-For: 172.24.0.1
-- BereqMethod GET
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader If-None-Match: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
-- BereqHeader X-Varnish: 65553
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- Timestamp Fetch: 1663833108.524843 0.000027 0.000027
-- Timestamp Connected: 1663833108.525006 0.000191 0.000163
-- BackendOpen 31 default 172.24.2.11 80 172.24.2.14 52026 connect
-- Timestamp Bereq: 1663833108.525047 0.000231 0.000040
-- Timestamp Beresp: 1663833108.530071 0.005256 0.005024
-- BerespProtocol HTTP/1.1
-- BerespStatus 200
-- BerespReason OK
-- BerespHeader Content-Type: application/json; charset=utf-8
-- BerespHeader Content-Length: 598
-- BerespHeader ETag: W/"256-1rfBjX0LanGZOnmzdwpQfzIjU30"
-- BerespHeader Date: Thu, 22 Sep 2022 07:51:48 GMT
-- BerespHeader Connection: keep-alive
-- BerespHeader Keep-Alive: timeout=5
-- TTL RFC 120 10 0 1663833109 1663833109 1663833108 0 0 cacheable
-- VCL_call BACKEND_RESPONSE
-- VCL_return deliver
-- Timestamp Process: 1663833108.530118 0.005302 0.000046
-- Filters
-- Storage malloc s0
-- Fetch_Body 3 length stream
-- BackendClose 31 default recycle
-- Timestamp BerespBody: 1663833108.530294 0.005479 0.000176
-- Length 598
-- BereqAcct 195 0 195 214 598 812
-- End
The Hit 12 -1.375188 30.000000 0.000000 line from the logs shows that the object has a remaining lifetime of -1.375188 seconds which is still more than -30 which is the grace limit.
The Link bereq 65553 bgfetch log line proves that a background fetch is made to the backend to store the latest version of the content. The transaction id in the Link tag matches the BeReq transaction that is part of the logs.
So while the new object is being fetched in transaction 65553, the response that is returned is still the old one with Age: 26 to prove it.
Step 4: next the next request gets the fresh content
Because grace mode will trigger an async fetch if there's grace time left, the current user doesn't see the impact of that fetch. However, the next user will get fresh content. The following cURL output illustrates this:
➜ ~ curl -I localhost
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 598
ETag: W/"256-1rfBjX0LanGZOnmzdwpQfzIjU30"
Date: Thu, 22 Sep 2022 07:51:48 GMT
X-Varnish: 32790 65553
Age: 0
Via: 1.1 varnish (Varnish/6.6)
Accept-Ranges: bytes
Connection: keep-alive
Because Age: 0 is set, this is a brand new object. However as time progresses, the age counter will increase.
Conclusion
Soft purging with a non-zero grace value will not immediately remove an object from the cache. Instead it marks the object as expired and gives it grace time.
This ensures that the first visitor after the soft purge doesn't have to wait for the backend fetch to be completed.
It's a tradeoff between serving fresh content immediately and letting users benefit from the performance of Varnish will fetching content asynchronously and serving stale objects while that happens.
UPDATE: examining the your VCL & logs
After having examined your VCL & the VSL logs you provided, I'm assuming the creation of the caching hash differs from your GET call and your PURGE call.
Just for reference, the hash in vcl_hash is created using the URL value and the value of the Host header.
The logs indicate that you perform the GET call with the following Host header:
Host: test-prod.luminalpark.com
When you do the purge, you use the following Host header:
Host: 172.16.3.37
As this values differ, the hash will also differ. That's why the PURGE call results in a 404.
Conclusion: please use the right Host header to invalidate your content.
I'm trying to use Varnish to cache my site. But I hope Varnish bypass some requests with a named Cookies(RSS).
Varnish ver: 6.3
Here is a part of my default.vcl
sub vcl_recv {
set req.http.X-Varnish-PHP_SID = req.http.Cookie;
set req.http.X-Varnish-PHP_SID =
regsuball(req.http.X-Varnish-PHP_SID, ";? ?RSS=([a-zA-Z0-9]+)( |;| ;).*","\1");
set req.http.Cookie = req.http.X-Varnish-PHP_SID;
unset req.http.X-Varnish-PHP_SID;
}
Here is logs from Varnish
* << Request >> 452
- Begin req 411 rxreq
- Timestamp Start: 1591947637.399591 0.000000 0.000000
- Timestamp Req: 1591947637.400066 0.000475 0.000475
- VCL_use boot
- ReqStart 42.88.136.35 10072 a0
- ReqMethod POST
- ReqURL /~YeV.js
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.ttester.com:8000
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
- ReqHeader Xiaytvhcnb: liacddpfckbjzpovdxrvjhibipwqevuxivzelicudjgcngcgdvgddndfghzrtvoelxkblzkiwqewvjleubxycuitwojhdztogomqoqtyufkqqqecrfoytklnxztnjytcrfptiplkuidfrbtxgighjpoppdkrjmngknujsgeunfgdcghjejprujdcyqmgkmxwgfdamxofmzjneajjymvqrixfaypudahokxrydpoeuynewshpzq
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader Accept-Language: en-us
- ReqHeader Content-Type: application/x-www-form-urlencoded
- ReqHeader Origin: http://www.ttester.com:8000
- ReqHeader User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Referer: http://www.ttester.com:8000/~YeV.js
- ReqHeader Content-Length: 8
- ReqHeader Cookie:
- ReqHeader Cookie: BAIDU-STFS=6352be5e1748a73bfe4c898d9805215c56d6b8a9e6d30b2549168ae853c76078; RSS=1; Hm_lpvt_12e92bc53a305b423a5f20bd34b0a75e=1591947600; Hm_lvt_12e92bc53a305b423a5f20bd34b0a75e=1591936468; Hm_lpvt_12e92bc53a305b423a5f20bd34b0a75e=1591944261; Hm_l
- ReqHeader X-Forwarded-For: 42.88.136.35
- VCL_call RECV
- ReqHeader X-Varnish-PHP_SID:
- ReqUnset X-Varnish-PHP_SID:
- ReqHeader X-Varnish-PHP_SID:
- ReqUnset Cookie:
- ReqUnset Cookie: BAIDU-STFS=6352be5e1748a73bfe4c898d9805215c56d6b8a9e6d30b2549168ae853c76078; RSS=1; Hm_lpvt_12e92bc53a305b423a5f20bd34b0a75e=1591947600; Hm_lvt_12e92bc53a305b423a5f20bd34b0a75e=1591936468; Hm_lpvt_12e92bc53a305b423a5f20bd34b0a75e=1591944261; Hm_l
- ReqHeader Cookie:
- ReqUnset X-Varnish-PHP_SID:
- ReqUnset Host: www.ttester.com:8000
- ReqHeader host: www.ttester.com:8000
- VCL_return pass
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- Link bereq 453 pass
- Storage malloc Transient
- Timestamp ReqBody: 1591947637.400221 0.000629 0.000154
- Timestamp Fetch: 1591947637.407862 0.008270 0.007641
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Fri, 12 Jun 2020 07:40:37 GMT
- RespHeader Server: Apache/2.4.25 (Debian)
- RespHeader X-Powered-By: PHP/7.2.6
- RespHeader URL-Record-File: www.ttester.com/~YeV.js
- RespHeader URL-Cluster: cn5
- RespHeader cache-control: no-store, no-cache, must-revalidate
- RespHeader Set-Cookie: BAIDU-STFS=6352be5e1748a73bfe4c898d9805215c56d6b8a9e6d30b2549168ae853c76078; Expires=Friday, 12-Jun-2020 15:40:39 CST; Max-Age=2; path=/
- RespHeader Set-Cookie: RSS=1; Expires=Friday, 12-Jun-2020 15:40:39 CST; Max-Age=2; path=/
- RespHeader Vary: Accept-Encoding
- RespHeader Content-Encoding: gzip
- RespHeader Content-Length: 626
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader V-Cache-TTL: 0.000
- RespHeader V-Cache-Grace: 10.000
- RespHeader X-Varnish: 452
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.3)
- VCL_call DELIVER
- RespHeader V-Cache: MISS
- RespUnset Server: Apache/2.4.25 (Debian)
- RespHeader Server: MFPad
- RespUnset X-Powered-By: PHP/7.2.6
- RespUnset Via: 1.1 varnish (Varnish/6.3)
- RespUnset X-Varnish: 452
- VCL_return deliver
- Timestamp Process: 1591947637.407883 0.008292 0.000021
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1591947637.407919 0.008328 0.000036
- ReqAcct 2056 8 2064 633 626 1259
- End
I also tried another version of VCL as below and doesn't work as well.
sub vcl_recv {
if (req.http.Cookie && !(req.http.Cookie ~ "RSS=")) {
unset req.http.Cookie;
}
}
I just want Varnish can bypass a request with a RSS Cookie.
If I understand correctly, you want all pages to be cached, except the ones that contain the RSS cookie.
Here's the old school ugly way of doing it:
sub vcl_recv {
if (req.http.Cookie) {
set req.http.Cookie = ";" + req.http.Cookie;
set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
set req.http.Cookie = regsuball(req.http.Cookie, ";(RSS)=", "; \1=");
set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
if (req.http.cookie ~ "^\s*$") {
unset req.http.cookie;
}
}
}
However, if you were to upgrade to Varnish Cache 6.4, you'll have the vmod_cookie packaged with that release, which makes cookie manipulation a lot easier.
Here's how that would work using vmod_cookie:
import cookie;
sub vcl_recv {
cookie.parse(req.http.cookie);
cookie.keep("RSS");
set req.http.cookie = cookie.get_string();
}
Both implementations rely on the default Varnish behavior. Meaning that if after the substitution of the cookies, there is still a cookie left (in our case RSS), Varnish will not cache and perform a return(pass) in the vcl_recv hook.
Is this what you're looking for?
Something weird is going on with my varnishlog.
I am trying to filter all the requests in the varnishlog that have a ban, so I try this:
sudo varnishlog -g request -q 'ReqMethod eq "BAN"'
This gets all BAN and GET request, which is weird.
If I try:
sudo varnishlog -g request -q 'ReqMethod eq "POST"'
It works just fine and only shows the POST requests.
If I try:
sudo varnishlog -g request -q 'ReqMethod eq "ABSOLUTELYWRONG"'
It doesn't show anything at all, as expected.
So everything works fine except BAN.
I expect BAN to only show BAN requests.
Anyone knows what am I doing wrong?
EDIT here is an example of what I am getting using the BAN filter which should not be here because it's a GET:
*4* << Request >> 163935
-4- Begin req 163934 esi
-4- ReqURL someUrl
-4- ReqUnset Host: myhost.local.es
-4- ReqHeader Host: myhost.local.es
-4- ReqUnset Accept-Encoding: gzip
-4- ReqHeader Accept-Encoding: gzip
-4- Timestamp Start: 1572426799.985887 0.000000 0.000000
-4- ReqStart 127.0.0.1 47748
-4- ReqMethod GET
-4- ReqURL someUrl
-4- ReqProtocol HTTP/1.1
-4- ReqHeader X-Host: .*
-4- ReqHeader X-Url: .*
-4- ReqHeader X-Content-Type: .*
-4- ReqHeader X-Cache-Tags: (?<=,|^)(my-tag-9000
-4- ReqHeader User-Agent: Guzzle/3.9.3 curl/7.58.0 PHP/7.1.33-1+ubuntu18.04.1+deb.sury.org+1
-4- ReqHeader Host: myhost.local.com
-4- ReqHeader Accept-Encoding: gzip
-4- ReqHeader X-Forwarded-For: 127.0.0.1
-4- VCL_call RECV
-4- VCL_acl MATCH localhost "localhost"
-4- ReqHeader Surrogate-Capability: abc=ESI/1.0
-4- VCL_return hash
-4- VCL_call HASH
-4- VCL_return lookup
-4- Hit 32870 31533030.619874 86400.000000 0.000000
-4- VCL_call HIT
-4- VCL_return deliver
-4- RespProtocol HTTP/1.1
-4- RespStatus 200
-4- RespReason OK
-4- RespHeader Server: nginx/1.14.0 (Ubuntu)
-4- RespHeader Content-Type: text/html; charset=UTF-8
-4- RespHeader Cache-Control: public, s-maxage=31536000
-4- RespHeader X-Cache-Debug: 1
-4- RespHeader X-Cache-Tags: twig-randomnumber
-4- RespHeader X-Debug-Token: 9e5c89
-4- RespHeader X-Debug-Token-Link: mydebugtokenlink
-4- RespHeader Date: Wed, 30 Oct 2019 08:23:50 GMT
-4- RespHeader X-Url: /my/url/
-4- RespHeader X-Host: myhost.local.com
-4- RespHeader Content-Encoding: gzip
-4- RespHeader Vary: Accept-Encoding
-4- RespHeader X-Varnish: 163935 32870
-4- RespHeader Age: 2969
-4- RespHeader Via: 1.1 varnish (Varnish/5.2)
-4- VCL_call DELIVER
-4- RespUnset Via: 1.1 varnish (Varnish/5.2)
-4- RespUnset X-Cache-Tags: twig-randomnumber
-4- RespUnset X-Url: someUrl
-4- RespUnset X-Varnish: 163935 32870
-4- RespUnset Server: nginx/1.14.0 (Ubuntu)
-4- RespUnset Age: 2969
-4- RespUnset Cache-Control: public, s-maxage=31536000
-4- RespUnset Vary: Accept-Encoding
-4- RespHeader Vary: User-Agent, Accept-Encoding
-4- RespHeader Cache-Control: max-age=300, public
-4- VCL_return deliver
-4- Timestamp Process: 1572426799.985903 0.000016 0.000016
-4- RespHeader Accept-Ranges: bytes
-4- RespHeader Content-Length: 43
-4- Timestamp Resp: 1572426799.985922 0.000035 0.000020
-4- ReqAcct 0 0 0 0 28 28
-4- End
I am using varnish to speed up a customer's website load time. I have a problem with the images on a page. The Images on a page are not shown on the page. here is the chrome output headers when I hit Ctrl+f5:
Request URL:https://DOMAINNAME/wp-content/uploads/2017/12/telegram-768x255.png
Request Method:GET
Status Code:200 OK
Remote Address:IPADDRESS:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Accept-Ranges:bytes
Age:28
Connection:keep-alive
Content-Length:96169
Content-Type:image/png
Date:Sat, 30 Dec 2017 14:38:40 GMT
Last-Modified:Sat, 16 Dec 2017 11:06:23 GMT
Server:Litespeed
Strict-Transport-Security:max-age=31536000
X-Cache:HIT
X-Configured-By:ServerSetup.co
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.9
AlexaToolbar-ALX_NS_PH:AlexaToolbar/alx-4.0.1
Cache-Control:no-cache
Connection:keep-alive
Cookie:_ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806
Host:HOSTNAME
Pragma:no-cache
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
and here's the output when I hit Enter on the address bar:
Request URL:https://DOMAINNAME/wp-content/uploads/2017/12/telegram-768x255.png
Request Method:GET
Status Code:304 Not Modified
Remote Address:IPADDRESS:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Age:271
Connection:keep-alive
Content-Type:image/png
Date:Sat, 30 Dec 2017 14:42:43 GMT
Last-Modified:Sat, 16 Dec 2017 11:06:23 GMT
Server:Litespeed
Strict-Transport-Security:max-age=31536000
X-Cache:HIT
X-Configured-By:ServerSetup.co
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.9
AlexaToolbar-ALX_NS_PH:AlexaToolbar/alx-4.0.1
Cache-Control:max-age=0
Connection:keep-alive
Cookie:_ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806
Host:HOSTNAME
If-Modified-Since:Sat, 16 Dec 2017 11:06:23 GMT
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
and here's is the varnishlog for the image url:
varnishlog -g request -q "ReqUrl ~ 'wp-content/uploads/2017/12/telegram-768x255.png'"
* << Request >> 870337886
- Begin req 870337885 rxreq
- Timestamp Start: 1514645156.766974 0.000000 0.000000
- Timestamp Req: 1514645156.766974 0.000000 0.000000
- ReqStart 192.168.1.106 42860
- ReqMethod GET
- ReqURL /wp-content/uploads/2017/12/telegram-768x255.png
- ReqProtocol HTTP/1.0
- ReqHeader X-Real-IP: 192.168.1.104
- ReqHeader X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Nginx: on
- ReqHeader Host: HOSTNAME
- ReqHeader Connection: close
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader Accept: image/webp,image/apng,image/*,*/*;q=0.8
- ReqHeader Referer: https://DOMAINNAME/contactus/
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: en-US,en;q=0.9
- ReqHeader Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806; _gat=1
- ReqUnset X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- VCL_call RECV
- ReqUnset Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806; _gat=1
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- ReqUnset X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqUnset Accept-Language: en-US,en;q=0.9
- ReqUnset User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader cookie:
- ReqUnset cookie:
- ReqHeader cookie:
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 870337573
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Sat, 30 Dec 2017 14:37:35 GMT
- RespHeader Server: Apache/2.2.15 (CentOS)
- RespHeader Last-Modified: Sat, 16 Dec 2017 11:06:23 GMT
- RespHeader Content-Length: 96169
- RespHeader Content-Type: image/png
- RespHeader X-Varnish: 870337886 870337573
- RespHeader Age: 464
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespUnset X-Varnish: 870337886 870337573
- RespUnset Via: 1.1 varnish-v4
- RespHeader X-Configured-By: ServerSetup.co
- RespUnset Server: Apache/2.2.15 (CentOS)
- RespHeader Server: Apache
- VCL_return deliver
- Timestamp Process: 1514645156.767049 0.000075 0.000075
- Debug "RES_MODE 2"
- RespHeader Connection: close
- RespHeader Accept-Ranges: bytes
- Timestamp Resp: 1514645156.767130 0.000156 0.000081
- Debug "XXX REF 2"
- ReqAcct 631 0 631 267 96169 96436
- End
The problem is that the image is not shown on the page, but it is shown in preview section of the chrome developer panel. Moreover if I open the image in a new tab in browser it is shown properly.
Varnish version is 4.0.4. and the web server is Apache 2.2.
Edit: When I load the page through varnish I get the following errors on the console tab (chrome):
(index):1820 Uncaught ReferenceError: tinymce is not defined
at HTMLDocument.<anonymous> ((index):1820)
at HTMLDocument.dispatch (jquery.js:3)
at HTMLDocument.r.handle (jquery.js:3)
at Object.trigger (jquery.js:3)
at Object.a.event.trigger (jquery-migrate.min.js:2)
at HTMLDocument.<anonymous> (jquery.js:3)
at Function.each (jquery.js:2)
at a.fn.init.each (jquery.js:2)
at a.fn.init.trigger (jquery.js:3)
at HTMLDocument.<anonymous> ((index):1316)
But when I load the page directly from backend server, there is no errors and the images are shown properly!!
Edit:
varnish log for 304 Not Modified
* << Request >> 885566068
- Begin req 885566067 rxreq
- Timestamp Start: 1515309410.697993 0.000000 0.000000
- Timestamp Req: 1515309410.697993 0.000000 0.000000
- ReqStart 192.168.1.106 33782
- ReqMethod GET
- ReqURL /wp-content/uploads/2017/12/Untitled-1.png
- ReqProtocol HTTP/1.0
- ReqHeader X-Real-IP: 192.168.1.104
- ReqHeader X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Nginx: on
- ReqHeader Host: bigtheme.ir
- ReqHeader Connection: close
- ReqHeader Cache-Control: max-age=0
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: en-US,en;q=0.9
- ReqHeader Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.1237154839.1515307013
- ReqHeader If-None-Match: "9a2989-38271-560731bc13e20"
- ReqHeader If-Modified-Since: Sat, 16 Dec 2017 11:06:26 GMT
- ReqUnset X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- VCL_call RECV
- ReqUnset Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.1237154839.1515307013
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqUnset X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqUnset Accept-Language: en-US,en;q=0.9
- ReqUnset User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader cookie:
- ReqUnset cookie:
- ReqHeader cookie:
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 885027311
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Sun, 07 Jan 2018 07:15:40 GMT
- RespHeader Server: Apache/2.2.15 (CentOS)
- RespHeader Last-Modified: Sat, 16 Dec 2017 11:06:26 GMT
- RespHeader Content-Length: 230001
- RespHeader Content-Type: image/png
- RespHeader X-Varnish: 885566068 885027311
- RespHeader Age: 33
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespUnset X-Varnish: 885566068 885027311
- RespUnset Via: 1.1 varnish-v4
- RespHeader X-Configured-By: ServerSetup.co
- RespUnset Server: Apache/2.2.15 (CentOS)
- RespHeader Server: Litespeed
- VCL_return deliver
- Timestamp Process: 1515309410.698046 0.000053 0.000053
- RespProtocol HTTP/1.1
- RespStatus 304
- RespReason Not Modified
- RespReason Not Modified
- RespUnset Content-Length: 230001
- Debug "RES_MODE 0"
- RespHeader Connection: close
- Timestamp Resp: 1515309410.698061 0.000068 0.000015
- Debug "XXX REF 2"
- ReqAcct 731 0 731 231 0 231
- End
As your console tab shows, the problem is with a HTML document or Javascript (jquery), not with the image itself.
Moreover, why isn't your varnishlog showing the second request? What server is returning the "304 Not Modified"? There's something in between Chrome and Varnish. A Chrome plugin?
Our software uses a cookie "selected_language" when people change language from Spanish to anything else. By default the cookie is not set.
We need to cache our site taking into account the language, so our index.php is cached several times,one for each language.
Our hash function uses language cookie (We get more cookies but they are removed inside vcl_recv, just the language cookie (if any) gets to the hash routine:
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
if (req.http.Cookie) {
hash_data(req.http.Cookie);
}
}
Our problem comes when we want to purge our cache.
If we purge our index webpage, without cookies (spanish version), everything works ok:
curl -X PURGE http://www.arasaac.org/index.php
However if we try to purge our index webpage with cookies (english version), it does nothing.
I think purge uses hash so it should work if I send the related cookie.
Our purge code:
sub vcl_purge {
# Only handle actual PURGE HTTP methods, everything else is discarded
if (req.method != "PURGE") {
# restart request
set req.http.X-Purge = "Yes";
return(restart);
}
}
Inside vcl_recv:
# Allow purging
if (req.method == "PURGE") {
# if (!client.ip ~ purge) { # purge is the ACL defined at the begining
# Not from an allowed IP? Then die with an error.
# return (synth(405, "This IP is not allowed to send PURGE requests."));
# }
# If you got this stage (and didn't error out above), purge the cached result
return (purge);
}
These are all the logs in case they're needed...
I ask again for the page, that is cached:
* << Request >> 262149
- Begin req 262148 rxreq
- Timestamp Start: 1488455938.456980 0.000000 0.000000
- Timestamp Req: 1488455938.456980 0.000000 0.000000
- ReqStart 172.20.0.3 51672
- ReqMethod GET
- ReqURL /index.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.arasaac.org
- ReqHeader Connection: close
- ReqHeader X-Real-IP: 172.20.0.1
- ReqHeader X-Forwarded-For: 172.20.0.1
- ReqHeader X-Forwarded-Proto: http
- ReqHeader X-Forwarded-Ssl: off
- ReqHeader X-Forwarded-Port: 80
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Referer: http://www.arasaac.org/index.php
- ReqHeader Accept-Encoding: gzip, deflate, sdch
- ReqHeader Accept-Language: es-ES,es;q=0.8,en;q=0.6
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _ga=GA1.2.99795965.1488449144; _gat=1
- ReqUnset X-Forwarded-For: 172.20.0.1
- ReqHeader X-Forwarded-For: 172.20.0.1, 172.20.0.3
- VCL_call RECV
- ReqUnset Host: www.arasaac.org
- ReqHeader Host: www.arasaac.org
- ReqURL /
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _ga=GA1.2.99795965.1488449144; _gat=1
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _ga=GA1.2.99795965.1488449144; _gat=1
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _ga=GA1.2.99795965.1488449144; _gat=1
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _ga=GA1.2.99795965.1488449144; _gat=1
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _ga=GA1.2.99795965.1488449144; _gat=1
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate, sdch
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 3
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Thu, 02 Mar 2017 11:57:52 GMT
- RespHeader Server: Apache/2.4.10 (Debian)
- RespHeader X-Powered-By: PHP/5.6.30
- RespHeader Vary: Accept-Encoding
- RespHeader Content-Encoding: gzip
- RespHeader Content-Length: 5379
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader cache-control: public, max-age = 300
- RespHeader log: ha entrado aquí
- RespHeader X-CacheReason: varnishcache
- RespHeader X-Varnish: 262149 3
- RespHeader Age: 66
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespHeader X-Cache-Hits: 1
- RespUnset X-Powered-By: PHP/5.6.30
- RespUnset Server: Apache/2.4.10 (Debian)
- RespUnset X-Varnish: 262149 3
- RespUnset Via: 1.1 varnish-v4
- VCL_return deliver
- Timestamp Process: 1488455938.457118 0.000138 0.000138
- RespHeader Accept-Ranges: bytes
- Debug "RES_MODE 2"
- RespHeader Connection: close
- Timestamp Resp: 1488455938.457194 0.000215 0.000077
- ReqAcct 658 0 658 336 5379 5715
- End
I try to purge the page, so I send the cookie:
curl -X PURGE --cookie "selected_language=en" http://www.arasaac.org/index.php
Logs seem good for me:
* << Request >> 28
- Begin req 27 rxreq
- Timestamp Start: 1488456128.433721 0.000000 0.000000
- Timestamp Req: 1488456128.433721 0.000000 0.000000
- ReqStart 172.20.0.3 53286
- ReqMethod PURGE
- ReqURL /index.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.arasaac.org
- ReqHeader Connection: close
- ReqHeader X-Real-IP: 172.20.0.1
- ReqHeader X-Forwarded-For: 172.20.0.1
- ReqHeader X-Forwarded-Proto: http
- ReqHeader X-Forwarded-Ssl: off
- ReqHeader X-Forwarded-Port: 80
- ReqHeader User-Agent: curl/7.35.0
- ReqHeader Accept: */*
- ReqHeader Cookie: selected_language=en;
- ReqUnset X-Forwarded-For: 172.20.0.1
- ReqHeader X-Forwarded-For: 172.20.0.1, 172.20.0.3
- VCL_call RECV
- ReqUnset Host: www.arasaac.org
- ReqHeader Host: www.arasaac.org
- ReqURL /
- VCL_return purge
- VCL_call HASH
- VCL_return lookup
- VCL_call PURGE
- VCL_return synth
- Timestamp Process: 1488456128.433754 0.000033 0.000033
- RespHeader Date: Thu, 02 Mar 2017 12:02:08 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 28
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespReason Purged
- VCL_call SYNTH
- VCL_return deliver
- RespHeader Content-Length: 0
- Storage malloc Transient
- RespHeader Accept-Ranges: bytes
- Debug "RES_MODE 2"
- RespHeader Connection: close
- Timestamp Resp: 1488456128.433792 0.000070 0.000038
- ReqAcct 261 0 261 152 0 152
- End
However if I load again index.php in the browser, the cache is still there:
<< Request >> 30
- Begin req 29 rxreq
- Timestamp Start: 1488456171.661988 0.000000 0.000000
- Timestamp Req: 1488456171.661988 0.000000 0.000000
- ReqStart 172.20.0.3 53688
- ReqMethod GET
- ReqURL /index.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.arasaac.org
- ReqHeader Connection: close
- ReqHeader X-Real-IP: 172.20.0.1
- ReqHeader X-Forwarded-For: 172.20.0.1
- ReqHeader X-Forwarded-Proto: http
- ReqHeader X-Forwarded-Ssl: off
- ReqHeader X-Forwarded-Port: 80
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Referer: http://www.arasaac.org/index.php
- ReqHeader Accept-Encoding: gzip, deflate, sdch
- ReqHeader Accept-Language: es-ES,es;q=0.8,en;q=0.6
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqUnset X-Forwarded-For: 172.20.0.1
- ReqHeader X-Forwarded-For: 172.20.0.1, 172.20.0.3
- VCL_call RECV
- ReqUnset Host: www.arasaac.org
- ReqHeader Host: www.arasaac.org
- ReqURL /
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- ReqUnset Cookie: selected_language=en;
- ReqHeader Cookie: selected_language=en;
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate, sdch
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 3
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Thu, 02 Mar 2017 11:57:52 GMT
- RespHeader Server: Apache/2.4.10 (Debian)
- RespHeader X-Powered-By: PHP/5.6.30
- RespHeader Vary: Accept-Encoding
- RespHeader Content-Encoding: gzip
- RespHeader Content-Length: 5379
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader cache-control: public, max-age = 300
- RespHeader log: ha entrado aquí
- RespHeader X-CacheReason: varnishcache
- RespHeader X-Varnish: 30 3
- RespHeader Age: 299
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespHeader X-Cache-Hits: 2
- RespUnset X-Powered-By: PHP/5.6.30
- RespUnset Server: Apache/2.4.10 (Debian)
- RespUnset X-Varnish: 30 3
- RespUnset Via: 1.1 varnish-v4
- VCL_return deliver
- Timestamp Process: 1488456171.662050 0.000061 0.000061
- RespHeader Accept-Ranges: bytes
- Debug "RES_MODE 2"
- RespHeader Connection: close
- Timestamp Resp: 1488456171.662105 0.000117 0.000056
- ReqAcct 658 0 658 337 5379 5716
- End
By the way.. how is it that ReqHeader and Reqset lines in logs related to cookies are repeated many times?
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqUnset Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1; _ga=GA1.2.99795965.1488449144
- ReqHeader Cookie: PHPSESSID=242b73bc8a560b89308009e05af7eefd; selected_language=en; _gat=1;
...
My wild guess is that when you cleaning the cookies you end up with:
"selected_language=en; " (with trailing space)
Whereas in your purge request you're passing
"selected_language=en;"
Try to send purge like this:
curl -X PURGE --cookie "selected_language=en " http://www.arasaac.org/index.php
The repeated lines you see in varnishlog are for every time your VCL code tries to unset or manipulate cookie value otherwise (during cleanup stage).