Related
I'm trying to do a soft purge only for certain req.url values, all other invalidations are managed with a ban.
While ban is working, purge.soft(0s,30s) does not modify anythig in the cache, TTL remains the standard (7200s) and the cache remains active.
What am I doing wrong?
Full VCL code:
https://pastebin.com/QLmBh0hw
GET request log:
* << Request >> 524366
- Begin req 524315 rxreq
- Timestamp Start: 1664176259.698910 0.000000 0.000000
- Timestamp Req: 1664176259.698910 0.000000 0.000000
- ReqStart 172.16.1.194 23952 a0
- ReqMethod GET
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqProtocol HTTP/1.1
- ReqHeader X-Forwarded-For: 84.247.245.84, 130.176.221.197
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader Host: test-prod.luminalpark.com
- ReqHeader X-Amzn-Trace-Id: Root=1-63315083-50faab2c2fbacda82eba1f9c
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
- ReqHeader X-Amz-Cf-Id: ToerPgpur_0Y4SvPsU3hkSSzfK-PywWzlX-nlhnQasaji9IGXsFb5g==
- ReqHeader Via: 2.0 0f03c98743d9ffe79330c1f694241fc2.cloudfront.net (CloudFront)
- ReqHeader Cookie: _dc_gtm_UA-830149-18=1; _fbp=fb.1.1663062717375.1186628968; _ga=GA1.1.2073680112.1662976397; _ga_499EG6CXZD=GS1.1.1664175973.166.1.1664176257.0.0.0; _gcl_au=1.1.1770949614.1662976397; _gid=GA1.2.1336671677.1662976397; _hjSessionUser_21623=eyJpZCI
- ReqHeader Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
- ReqHeader Referer: https://test-prod.luminalpark.com/it-it/
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader cache-control: max-age=0
- ReqHeader sec-ch-ua: "Google Chrome";v="105", "Not)A;Brand";v="8", "Chromium";v="105"
- ReqHeader sec-ch-ua-mobile: ?0
- ReqHeader sec-ch-ua-platform: "Windows"
- ReqHeader dnt: 1
- ReqHeader upgrade-insecure-requests: 1
- ReqHeader sec-fetch-site: same-origin
- ReqHeader sec-fetch-mode: navigate
- ReqHeader sec-fetch-user: ?1
- ReqHeader sec-fetch-dest: document
- ReqHeader CloudFront-Viewer-HTTP-Version: 2.0
- ReqHeader CloudFront-Forwarded-Proto: https
- ReqHeader CloudFront-Viewer-Address: 84.247.245.84:61581
- ReqHeader CloudFront-Viewer-TLS: TLSv1.3:TLS_AES_128_GCM_SHA256:connectionReused
- ReqHeader X-Cloudfront-Origin: VC5ZNQ588QNE3S
- ReqUnset X-Forwarded-For: 84.247.245.84, 130.176.221.197
- ReqHeader X-Forwarded-For: 84.247.245.84, 130.176.221.197, 172.16.1.194
- VCL_call RECV
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 884747 7171.150022 60.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Server: nginx/1.18.0 (Ubuntu)
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader Cache-Control: must-revalidate, public, s-maxage=7200
- RespHeader Date: Mon, 26 Sep 2022 07:10:30 GMT
- RespHeader Strict-Transport-Security: max-age=31536000; includeSubDomains
- RespHeader X-Frame-Options: deny
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Referrer-Policy: strict-origin-when-cross-origin
- RespHeader sw-invalidation-states:
- RespHeader Set-Cookie: sw-states=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; secure; httponly; samesite=lax
- RespHeader Set-Cookie: sw-cache-hash=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; httponly
- RespHeader Content-Encoding: gzip
- RespHeader Vary: Accept-Encoding
- RespHeader x-url: /it-it/prodotti/catene/catene-di-luci/
- RespHeader X-Cacheable: YES
- RespHeader X-Varnish: 524366 884747
- RespHeader Age: 28
- RespHeader Via: 1.1 varnish (Varnish/6.0)
- VCL_call DELIVER
- RespUnset Cache-Control: must-revalidate, public, s-maxage=7200
- RespHeader Cache-Control: max-age=0, private
- RespUnset Set-Cookie: sw-states=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; secure; httponly; samesite=lax
- RespUnset Set-Cookie: sw-cache-hash=deleted; expires=Sun, 26-Sep-2021 07:10:29 GMT; Max-Age=0; path=/; httponly
- RespHeader X-Cache: HIT
- RespUnset sw-invalidation-states:
- RespHeader X-Cache-Hits: 2
- VCL_return deliver
- Timestamp Process: 1664176259.698985 0.000076 0.000076
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 110181
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664176259.699106 0.000196 0.000120
- ReqAcct 3158 0 3158 612 110181 110793
- End
and here's varnishlog during PURGE
curl -XPURGE http://<varnish-host>/it-it/prodotti/catene/catene-di-luci/
* << Request >> 622680
- Begin req 622679 rxreq
- Timestamp Start: 1664176453.712271 0.000000 0.000000
- Timestamp Req: 1664176453.712271 0.000000 0.000000
- ReqStart 172.16.2.136 47728 a0
- ReqMethod PURGE
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqProtocol HTTP/1.1
- ReqHeader Host: 172.16.3.37
- ReqHeader User-Agent: curl/7.68.0
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 172.16.2.136
- VCL_call RECV
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- ReqURL /it-it/prodotti/catene/catene-di-luci/
- VCL_acl MATCH purgers "ecommerce-node1-prod"
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- ReqHeader purged: 0
- VCL_return synth
- Timestamp Process: 1664176453.712326 0.000055 0.000055
- RespProtocol HTTP/1.1
- RespStatus 404
- RespReason Not Found
- RespReason Not Found
- RespHeader Date: Mon, 26 Sep 2022 07:14:13 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 622680
- VCL_call SYNTH
- RespHeader purged: 0
- VCL_return deliver
- RespHeader Content-Length: 0
- Storage malloc Transient
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664176453.712364 0.000093 0.000038
- ReqAcct 114 0 114 153 0 153
- End
I've tested it myself and I'm not experiencing any issues.
The VCL code
Here's the standard vmod_purge example VCL code I took from https://varnish-cache.org/docs/6.0/reference/vmod_generated.html#example and adapted:
I removed the ACL check
I converted purge.hard() to purge.soft(0s,30s)
sub vcl_recv {
if (req.method == "PURGE") {
return (hash);
}
}
sub my_purge {
set req.http.purged = purge.soft(0s,30s);
if (req.http.purged == "0") {
return (synth(404));
}
else {
return (synth(200));
}
}
sub vcl_hit {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_miss {
if (req.method == "PURGE") {
call my_purge;
}
}
sub vcl_synth {
if (req.method == "PURGE") {
if (req.http.purged) {
set resp.http.purged = req.http.purged;
}
return (deliver);
}
}
What soft purging does
The way you've configured soft purging will ensure the object is considered stale, however because you set the grace time to 30s, async revalidation may still happen until 30 seconds past the lifetime of the object.
Example HTTP calls
Here are some example calls that illustrate what happens when soft purging takes place.
Step 1: call the endpoint and receive a fresh object
➜ ~ curl -I localhost
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 595
ETag: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
Date: Thu, 22 Sep 2022 07:51:22 GMT
X-Varnish: 65546 12
Age: 18
Via: 1.1 varnish (Varnish/6.6)
Accept-Ranges: bytes
Connection: keep-alive
As you can see we're calling the http://localhost endpoint and get a result with an Age: 18 header. This means the object has been cached for 18 seconds.
Step 2: purge
➜ ~ curl -XPURGE -I localhost
HTTP/1.1 200 OK
Date: Thu, 22 Sep 2022 07:51:42 GMT
Server: Varnish
X-Varnish: 32784
purged: 1
Content-Length: 0
Accept-Ranges: bytes
Connection: keep-alive
In step 2 we're performing the soft purge. The purged: 1 header implies that 1 object was purged.
Step 3: grace mode kicks in
➜ ~ curl -I localhost
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 595
ETag: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
Date: Thu, 22 Sep 2022 07:51:22 GMT
X-Varnish: 65552 12
Age: 26
Via: 1.1 varnish (Varnish/6.6)
Accept-Ranges: bytes
Connection: keep-alive
After the purge the we're still seeing cached content being served because the Age: 26 header that implies the object has been cached for 26 seconds.
But the output from varnishlog -g request show that the stale content is served while an asynchronous fetch happens for the new content. This is a direct result of calling purge.soft(0s, 30s):
* << Request >> 65552
- Begin req 65551 rxreq
- Timestamp Start: 1663833108.524685 0.000000 0.000000
- Timestamp Req: 1663833108.524685 0.000000 0.000000
- VCL_use boot
- ReqStart 172.24.0.1 58300 http
- ReqMethod HEAD
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.79.1
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 172.24.0.1
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 12 -1.375188 30.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- Link bereq 65553 bgfetch
- Timestamp Fetch: 1663833108.524864 0.000179 0.000179
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Content-Type: application/json; charset=utf-8
- RespHeader Content-Length: 595
- RespHeader ETag: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
- RespHeader Date: Thu, 22 Sep 2022 07:51:22 GMT
- RespHeader X-Varnish: 65552 12
- RespHeader Age: 26
- RespHeader Via: 1.1 varnish (Varnish/6.6)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1663833108.524876 0.000191 0.000011
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1663833108.524937 0.000252 0.000061
- ReqAcct 74 0 74 275 0 275
** << BeReq >> 65553
-- Begin bereq 65552 bgfetch
-- VCL_use boot
-- Timestamp Start: 1663833108.524815 0.000000 0.000000
-- BereqMethod HEAD
-- BereqURL /
-- BereqProtocol HTTP/1.1
-- BereqHeader Host: localhost
-- BereqHeader User-Agent: curl/7.79.1
-- BereqHeader Accept: */*
-- BereqHeader X-Forwarded-For: 172.24.0.1
-- BereqMethod GET
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader If-None-Match: W/"253-BCiA67uc9pD4vCc07ppQaevMbj0"
-- BereqHeader X-Varnish: 65553
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- Timestamp Fetch: 1663833108.524843 0.000027 0.000027
-- Timestamp Connected: 1663833108.525006 0.000191 0.000163
-- BackendOpen 31 default 172.24.2.11 80 172.24.2.14 52026 connect
-- Timestamp Bereq: 1663833108.525047 0.000231 0.000040
-- Timestamp Beresp: 1663833108.530071 0.005256 0.005024
-- BerespProtocol HTTP/1.1
-- BerespStatus 200
-- BerespReason OK
-- BerespHeader Content-Type: application/json; charset=utf-8
-- BerespHeader Content-Length: 598
-- BerespHeader ETag: W/"256-1rfBjX0LanGZOnmzdwpQfzIjU30"
-- BerespHeader Date: Thu, 22 Sep 2022 07:51:48 GMT
-- BerespHeader Connection: keep-alive
-- BerespHeader Keep-Alive: timeout=5
-- TTL RFC 120 10 0 1663833109 1663833109 1663833108 0 0 cacheable
-- VCL_call BACKEND_RESPONSE
-- VCL_return deliver
-- Timestamp Process: 1663833108.530118 0.005302 0.000046
-- Filters
-- Storage malloc s0
-- Fetch_Body 3 length stream
-- BackendClose 31 default recycle
-- Timestamp BerespBody: 1663833108.530294 0.005479 0.000176
-- Length 598
-- BereqAcct 195 0 195 214 598 812
-- End
The Hit 12 -1.375188 30.000000 0.000000 line from the logs shows that the object has a remaining lifetime of -1.375188 seconds which is still more than -30 which is the grace limit.
The Link bereq 65553 bgfetch log line proves that a background fetch is made to the backend to store the latest version of the content. The transaction id in the Link tag matches the BeReq transaction that is part of the logs.
So while the new object is being fetched in transaction 65553, the response that is returned is still the old one with Age: 26 to prove it.
Step 4: next the next request gets the fresh content
Because grace mode will trigger an async fetch if there's grace time left, the current user doesn't see the impact of that fetch. However, the next user will get fresh content. The following cURL output illustrates this:
➜ ~ curl -I localhost
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 598
ETag: W/"256-1rfBjX0LanGZOnmzdwpQfzIjU30"
Date: Thu, 22 Sep 2022 07:51:48 GMT
X-Varnish: 32790 65553
Age: 0
Via: 1.1 varnish (Varnish/6.6)
Accept-Ranges: bytes
Connection: keep-alive
Because Age: 0 is set, this is a brand new object. However as time progresses, the age counter will increase.
Conclusion
Soft purging with a non-zero grace value will not immediately remove an object from the cache. Instead it marks the object as expired and gives it grace time.
This ensures that the first visitor after the soft purge doesn't have to wait for the backend fetch to be completed.
It's a tradeoff between serving fresh content immediately and letting users benefit from the performance of Varnish will fetching content asynchronously and serving stale objects while that happens.
UPDATE: examining the your VCL & logs
After having examined your VCL & the VSL logs you provided, I'm assuming the creation of the caching hash differs from your GET call and your PURGE call.
Just for reference, the hash in vcl_hash is created using the URL value and the value of the Host header.
The logs indicate that you perform the GET call with the following Host header:
Host: test-prod.luminalpark.com
When you do the purge, you use the following Host header:
Host: 172.16.3.37
As this values differ, the hash will also differ. That's why the PURGE call results in a 404.
Conclusion: please use the right Host header to invalidate your content.
I want to filter the access logs on:
User-agent contains "Googlebot"
Referer contains "google"
I use this varnishlog command:
varnishlog -q "ReqHeader ~ 'User-Agent.*Googlebot'"
this is my output:
* << Request >> 564834158
- Begin req 564834144 rxreq
- Timestamp Start: 1626180326.557796 0.000000 0.000000
- Timestamp Req: 1626180326.557796 0.000000 0.000000
- ReqStart xx.xxx.xx.xxx 45253
- ReqMethod GET
- ReqURL /xx/yy/xxx-yyy
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.yyyyyy.com
- ReqHeader AMP-Cache-Transform: google;v="1..7"
- ReqHeader Connection: keep-alive
- ReqHeader Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader From: googlebot(at)googlebot.com
- ReqHeader User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader If-Modified-Since: Mon, 12 Jul 2021 10:56:33 GMT
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: xx.xxx.xx.xxx
- VCL_call RECV
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqUnset Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader accept: application/vnd.fos.user-context-hash
- ReqHeader X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqURL /userContext.php
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 561267092 34.193790 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Tue, 13 Jul 2021 12:44:00 GMT
- RespHeader Server: Apache
- RespHeader Access-Control-Allow-Origin: https://www.yyyyyy.com
- RespHeader Access-Control-Allow-Credentials: true
- RespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT
- RespHeader Pragma: no-cache
- RespHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- RespHeader Cache-Control: max-age=120, s-max-age=120
- RespHeader Vary: Cookie
- RespHeader Set-Cookie: mainMenuId=1002323; expires=Thu, 12-Aug-2021 12:44:00 GMT; Max-Age=2592000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: PHPSESSID=qhn0l4m4f18g4g3cer3d8e5pg3; path=/; domain=.xxxx.yyyy; HttpOnly
- RespHeader Set-Cookie: connexion_id=1898574699; expires=Sun, 09-Jan-2022 12:44:00 GMT; Max-Age=15552000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: connexion_id=1898574699; expires=Sun, 09-Jan-2022 12:44:00 GMT; Max-Age=15552000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: connexion_id=1898574699; expires=Sun, 09-Jan-2022 12:44:00 GMT; Max-Age=15552000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: membre_statut_id=60; expires=Mon, 11-Oct-2021 12:44:00 GMT; Max-Age=7776000; path=/; domain=.xxxx.yyyy
- RespHeader Set-Cookie: statutSolde=pub; expires=Thu, 12-Aug-2021 12:44:00 GMT; Max-Age=2592000; path=/; domain=.xxxx.yyyy
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Content-Type: application/vnd.fos.user-context-hash
- RespHeader X-Varnish: 564834158 561267092
- RespHeader Age: 85
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespHeader X-Cache-Hits: 261
- ReqHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- VCL_return restart
- Timestamp Process: 1626180326.558006 0.000209 0.000209
- Timestamp Restart: 1626180326.558009 0.000213 0.000003
- Link req 564834159 restart
- End
* << Request >> 562977018
- Begin req 562977017 restart
- Timestamp Start: 1626180326.055452 0.000223 0.000000
- ReqStart xx.xxx.xx.xxx 47603
- ReqMethod GET
- ReqURL /userContext.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.yyyyyy.com
- ReqHeader Connection: keep-alive
- ReqHeader From: googlebot(at)googlebot.com
- ReqHeader User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: xx.xxx.xx.xxx
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader accept: application/vnd.fos.user-context-hash
- ReqHeader X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- VCL_call RECV
- ReqUnset Accept-Encoding: gzip
- ReqHeader Accept-Encoding: gzip
- ReqURL /be/fr/isseymiyake-sac-seau-lucent-rose-femme-4850263
- ReqUnset X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqUnset accept: application/vnd.fos.user-context-hash
- ReqHeader accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqUnset X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 562977019 fetch
- Timestamp Fetch: 1626180328.191066 2.135837 2.135614
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Tue, 13 Jul 2021 12:45:26 GMT
- RespHeader Access-Control-Allow-Origin: https://www.yyyyyy.com
- RespHeader Access-Control-Allow-Credentials: true
- RespHeader isVarnish: 1
- RespHeader Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader templateName: FICHE_PRODUIT_TPL_ID
- RespHeader Content-Encoding: gzip
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Content-Length: 41620
- RespHeader Content-Type: text/html; charset=ISO-8859-1
- RespHeader Cache-Control: max-age=4
- RespHeader X-Varnish: 562977018
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespHeader X-Cache: MISS
- RespUnset Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader Vary: ,Accept-Encoding
- RespUnset Vary: ,Accept-Encoding
- RespHeader Vary: Accept-Encoding
- VCL_return deliver
- Timestamp Process: 1626180328.191101 2.135873 0.000036
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1626180328.192502 2.137274 0.001400
- ReqAcct 407 0 407 499 41620 42119
- End
* << Request >> 564834159
- Begin req 564834158 restart
- Timestamp Start: 1626180326.558009 0.000213 0.000000
- ReqStart xx.xxx.xx.xxx 45253
- ReqMethod GET
- ReqURL /userContext.php
- ReqProtocol HTTP/1.1
- ReqHeader Host: www.yyyyyy.com
- ReqHeader AMP-Cache-Transform: google;v="1..7"
- ReqHeader Connection: keep-alive
- ReqHeader From: googlebot(at)googlebot.com
- ReqHeader User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader If-Modified-Since: Mon, 12 Jul 2021 10:56:33 GMT
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: xx.xxx.xx.xxx
- ReqHeader Accept-Encoding: gzip
- ReqHeader X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqHeader accept: application/vnd.fos.user-context-hash
- ReqHeader X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqHeader X-User-Context-Hash: dbd07ab4746551895276bf2342469e1dc3b0f86ca18d1bab2dec6b82c9698a8c
- VCL_call RECV
- ReqUnset Accept-Encoding: gzip
- ReqHeader Accept-Encoding: gzip
- ReqURL /xx/yy/xxx-yyy
- ReqUnset X-Fos-Original-Url: /xx/yy/xxx-yyy
- ReqUnset accept: application/vnd.fos.user-context-hash
- ReqHeader accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- ReqUnset X-Fos-Original-Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 564834160 fetch
- Timestamp Fetch: 1626180328.405520 1.847724 1.847511
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Tue, 13 Jul 2021 12:45:26 GMT
- RespHeader Access-Control-Allow-Origin: https://www.yyyyyy.com
- RespHeader Access-Control-Allow-Credentials: true
- RespHeader isVarnish: 1
- RespHeader Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader templateName: CARROUSEL_MARQUE_TPL_ID
- RespHeader Content-Encoding: gzip
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Content-Type: text/html; charset=ISO-8859-1
- RespHeader Cache-Control: max-age=4
- RespHeader X-Varnish: 564834159
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespHeader X-Cache: MISS
- RespUnset Vary: X-User-Context-Hash,Accept-Encoding
- RespHeader Vary: ,Accept-Encoding
- RespUnset Vary: ,Accept-Encoding
- RespHeader Vary: Accept-Encoding
- VCL_return deliver
- Timestamp Process: 1626180328.405548 1.847751 0.000027
- RespHeader Accept-Ranges: bytes
- RespHeader Transfer-Encoding: chunked
- RespHeader Connection: keep-alive
- Timestamp Resp: 1626180328.409407 1.851611 0.003860
- ReqAcct 586 0 586 507 63033 63540
- End
I want this Data Format:
Date / time or timestamp
Hostname
IP client
Referer
Path
User-agent
Status code
Bytes
LoadTime (in Milliseconds)
The scheme (http or htps)
Varnishlog
If you want to use varnishlog, this is the command you need:
varnishlog -c -g request -I Timestamp:Start -I ReqHeader:Host \
-i ReqStart -I ReqHeader:Referer -i ReqUrl -I ReqHeader:User-Agent \
-i RespStatus -i ReqAcct -I Timestamp:Resp -I ReqHeader:X-Forwarded-Proto \
-q "ReqHeader:User-Agent ~ 'Googlebot' and ReqHeader:Referer ~ 'google'"
Here's some potential output for this command:
* << Request >> 46
- Timestamp Start: 1626253862.684183 0.000000 0.000000
- ReqStart 172.17.0.1 58432 a0
- ReqURL /
- ReqHeader Host: localhost
- ReqHeader X-Forwarded-Proto: https
- ReqHeader Referer: google
- ReqHeader User-Agent: Googlebot
- RespStatus 200
- Timestamp Resp: 1626253862.684398 0.000215 0.000139
- ReqAcct 114 0 114 326 612 938
See https://varnish-cache.org/docs/trunk/reference/vsl.html for the meaning of every VSL field. See https://varnish-cache.org/docs/trunk/reference/vsl-query.html for more information on VSL queries.
The output is multiline and not so easy to parse. On the other hand, it's very verbose. As usual: it's a tradeoff.
Varnishncsa
It's also possible to use varnishncsa, which is the Apache-style logformat. The output is single-line and less verbose.
The following command can be used to retrieve the information you need:
varnishncsa -c -g request \
-q "ReqHeader:User-Agent ~ 'Googlebot' and ReqHeader:Referer ~ 'google'" \
-F "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %T"
The format you get is an extension of the standard format, which is the following:
%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
I just added %T to display the time it took to retrieve the content. This is expressed in seconds, and will be equal to 0 most of the time. If you want to have microsecond precision, you can use %{VSL:Timestamp:Resp[2]}x instead.
Here's some potential output for this command:
172.17.0.1 - - [14/Jul/2021:09:10:23 +0000] "GET http://localhost/ HTTP/1.1" 200 612 "google" "Googlebot" 0
However, if you want the exact order and fields you specified in the list, you'll end up with the following varnishncsa command:
varnishncsa -c -g request \
-q "ReqHeader:User-Agent ~ 'Googlebot' and ReqHeader:Referer ~ 'google'" \
-F "%t %{Host}i %h %{Referer}i %U %{User-agent}i %s %b %{VSL:Timestamp:Resp[2]}x %{X-Forwarded-Proto}i"
FYI: I'm retrieving the scheme via the X-Forwarded-Proto.
Here's some potential output:
[14/Jul/2021:09:09:19 +0000] localhost 172.17.0.1 google / Googlebot 200 612 0.001057 https
See https://varnish-cache.org/docs/trunk/reference/varnishncsa.html#display-varnish-logs-in-apache-ncsa-combined-log-format for more information.
I am using varnish to speed up a customer's website load time. I have a problem with the images on a page. The Images on a page are not shown on the page. here is the chrome output headers when I hit Ctrl+f5:
Request URL:https://DOMAINNAME/wp-content/uploads/2017/12/telegram-768x255.png
Request Method:GET
Status Code:200 OK
Remote Address:IPADDRESS:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Accept-Ranges:bytes
Age:28
Connection:keep-alive
Content-Length:96169
Content-Type:image/png
Date:Sat, 30 Dec 2017 14:38:40 GMT
Last-Modified:Sat, 16 Dec 2017 11:06:23 GMT
Server:Litespeed
Strict-Transport-Security:max-age=31536000
X-Cache:HIT
X-Configured-By:ServerSetup.co
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.9
AlexaToolbar-ALX_NS_PH:AlexaToolbar/alx-4.0.1
Cache-Control:no-cache
Connection:keep-alive
Cookie:_ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806
Host:HOSTNAME
Pragma:no-cache
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
and here's the output when I hit Enter on the address bar:
Request URL:https://DOMAINNAME/wp-content/uploads/2017/12/telegram-768x255.png
Request Method:GET
Status Code:304 Not Modified
Remote Address:IPADDRESS:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Age:271
Connection:keep-alive
Content-Type:image/png
Date:Sat, 30 Dec 2017 14:42:43 GMT
Last-Modified:Sat, 16 Dec 2017 11:06:23 GMT
Server:Litespeed
Strict-Transport-Security:max-age=31536000
X-Cache:HIT
X-Configured-By:ServerSetup.co
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.9
AlexaToolbar-ALX_NS_PH:AlexaToolbar/alx-4.0.1
Cache-Control:max-age=0
Connection:keep-alive
Cookie:_ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806
Host:HOSTNAME
If-Modified-Since:Sat, 16 Dec 2017 11:06:23 GMT
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
and here's is the varnishlog for the image url:
varnishlog -g request -q "ReqUrl ~ 'wp-content/uploads/2017/12/telegram-768x255.png'"
* << Request >> 870337886
- Begin req 870337885 rxreq
- Timestamp Start: 1514645156.766974 0.000000 0.000000
- Timestamp Req: 1514645156.766974 0.000000 0.000000
- ReqStart 192.168.1.106 42860
- ReqMethod GET
- ReqURL /wp-content/uploads/2017/12/telegram-768x255.png
- ReqProtocol HTTP/1.0
- ReqHeader X-Real-IP: 192.168.1.104
- ReqHeader X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Nginx: on
- ReqHeader Host: HOSTNAME
- ReqHeader Connection: close
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader Accept: image/webp,image/apng,image/*,*/*;q=0.8
- ReqHeader Referer: https://DOMAINNAME/contactus/
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: en-US,en;q=0.9
- ReqHeader Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806; _gat=1
- ReqUnset X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- VCL_call RECV
- ReqUnset Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.498856688.1514639806; _gat=1
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- ReqUnset X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqUnset Accept-Language: en-US,en;q=0.9
- ReqUnset User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader cookie:
- ReqUnset cookie:
- ReqHeader cookie:
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 870337573
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Sat, 30 Dec 2017 14:37:35 GMT
- RespHeader Server: Apache/2.2.15 (CentOS)
- RespHeader Last-Modified: Sat, 16 Dec 2017 11:06:23 GMT
- RespHeader Content-Length: 96169
- RespHeader Content-Type: image/png
- RespHeader X-Varnish: 870337886 870337573
- RespHeader Age: 464
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespUnset X-Varnish: 870337886 870337573
- RespUnset Via: 1.1 varnish-v4
- RespHeader X-Configured-By: ServerSetup.co
- RespUnset Server: Apache/2.2.15 (CentOS)
- RespHeader Server: Apache
- VCL_return deliver
- Timestamp Process: 1514645156.767049 0.000075 0.000075
- Debug "RES_MODE 2"
- RespHeader Connection: close
- RespHeader Accept-Ranges: bytes
- Timestamp Resp: 1514645156.767130 0.000156 0.000081
- Debug "XXX REF 2"
- ReqAcct 631 0 631 267 96169 96436
- End
The problem is that the image is not shown on the page, but it is shown in preview section of the chrome developer panel. Moreover if I open the image in a new tab in browser it is shown properly.
Varnish version is 4.0.4. and the web server is Apache 2.2.
Edit: When I load the page through varnish I get the following errors on the console tab (chrome):
(index):1820 Uncaught ReferenceError: tinymce is not defined
at HTMLDocument.<anonymous> ((index):1820)
at HTMLDocument.dispatch (jquery.js:3)
at HTMLDocument.r.handle (jquery.js:3)
at Object.trigger (jquery.js:3)
at Object.a.event.trigger (jquery-migrate.min.js:2)
at HTMLDocument.<anonymous> (jquery.js:3)
at Function.each (jquery.js:2)
at a.fn.init.each (jquery.js:2)
at a.fn.init.trigger (jquery.js:3)
at HTMLDocument.<anonymous> ((index):1316)
But when I load the page directly from backend server, there is no errors and the images are shown properly!!
Edit:
varnish log for 304 Not Modified
* << Request >> 885566068
- Begin req 885566067 rxreq
- Timestamp Start: 1515309410.697993 0.000000 0.000000
- Timestamp Req: 1515309410.697993 0.000000 0.000000
- ReqStart 192.168.1.106 33782
- ReqMethod GET
- ReqURL /wp-content/uploads/2017/12/Untitled-1.png
- ReqProtocol HTTP/1.0
- ReqHeader X-Real-IP: 192.168.1.104
- ReqHeader X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Nginx: on
- ReqHeader Host: bigtheme.ir
- ReqHeader Connection: close
- ReqHeader Cache-Control: max-age=0
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: en-US,en;q=0.9
- ReqHeader Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.1237154839.1515307013
- ReqHeader If-None-Match: "9a2989-38271-560731bc13e20"
- ReqHeader If-Modified-Since: Sat, 16 Dec 2017 11:06:26 GMT
- ReqUnset X-Forwarded-For: 46.225.112.57
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- VCL_call RECV
- ReqUnset Cookie: _ga=GA1.2.1062445401.1514382767; _gid=GA1.2.1237154839.1515307013
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqUnset X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqHeader X-Forwarded-For: 46.225.112.57, 192.168.1.106
- ReqUnset Accept-Language: en-US,en;q=0.9
- ReqUnset User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/63.0.3239.84 Chrome/63.0.3239.84 Safari/537.36
- ReqHeader cookie:
- ReqUnset cookie:
- ReqHeader cookie:
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 885027311
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Date: Sun, 07 Jan 2018 07:15:40 GMT
- RespHeader Server: Apache/2.2.15 (CentOS)
- RespHeader Last-Modified: Sat, 16 Dec 2017 11:06:26 GMT
- RespHeader Content-Length: 230001
- RespHeader Content-Type: image/png
- RespHeader X-Varnish: 885566068 885027311
- RespHeader Age: 33
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespHeader X-Cache: HIT
- RespUnset X-Varnish: 885566068 885027311
- RespUnset Via: 1.1 varnish-v4
- RespHeader X-Configured-By: ServerSetup.co
- RespUnset Server: Apache/2.2.15 (CentOS)
- RespHeader Server: Litespeed
- VCL_return deliver
- Timestamp Process: 1515309410.698046 0.000053 0.000053
- RespProtocol HTTP/1.1
- RespStatus 304
- RespReason Not Modified
- RespReason Not Modified
- RespUnset Content-Length: 230001
- Debug "RES_MODE 0"
- RespHeader Connection: close
- Timestamp Resp: 1515309410.698061 0.000068 0.000015
- Debug "XXX REF 2"
- ReqAcct 731 0 731 231 0 231
- End
As your console tab shows, the problem is with a HTML document or Javascript (jquery), not with the image itself.
Moreover, why isn't your varnishlog showing the second request? What server is returning the "304 Not Modified"? There's something in between Chrome and Varnish. A Chrome plugin?
I'm currently facing some random 503-errors on my Varnish and I can't figure out where they come from. The FetchError is the following: HTC Status -1, which means HTC_S_EOF see here
Somehow, the bereq instantly failes and there is not even a single request sent to the backend server.
Do you guys have an idea how to locate and fix this error?
I'm using the latest Varnish 5.2 Version available, based on Alpine.
Thank you in advance!
Here is a docker logs file of the corresponding bereq:
{"log":"- Begin bereq 5145026 pass\n","stream":"stdout","time":"2017-12-11T21:38:52.636374999Z"}
{"log":"- Timestamp Start: 1513028321.532661 0.000000 0.000000\n","stream":"stdout","time":"2017-12-11T21:38:52.636377235Z"}
{"log":"- BereqMethod POST\n","stream":"stdout","time":"2017-12-11T21:38:52.63637951Z"}
{"log":"- BereqURL /wp-admin/post.php\n","stream":"stdout","time":"2017-12-11T21:38:52.636384624Z"}
{"log":"- BereqProtocol HTTP/1.1\n","stream":"stdout","time":"2017-12-11T21:38:52.636387074Z"}
{"log":"- BereqHeader Host: www.example.com\n","stream":"stdout","time":"2017-12-11T21:38:52.636389318Z"}
{"log":"- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\n","stream":"stdout","time":"2017-12-11T21:38:52.636391598Z"}
{"log":"- BereqHeader Content-Length: 6225\n","stream":"stdout","time":"2017-12-11T21:38:52.636393963Z"}
{"log":"- BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\n","stream":"stdout","time":"2017-12-11T21:38:52.636396303Z"}
{"log":"- BereqHeader Accept-Encoding: gzip, deflate, br\n","stream":"stdout","time":"2017-12-11T21:38:52.636398632Z"}
{"log":"- BereqHeader Accept-Language: de\n","stream":"stdout","time":"2017-12-11T21:38:52.636400931Z"}
{"log":"- BereqHeader Content-Type: application/x-www-form-urlencoded\n","stream":"stdout","time":"2017-12-11T21:38:52.636403196Z"}
{"log":"- BereqHeader Cookie: scrollPositionX=undefined; scrollPositionY=undefined; wp-saving-post=45779-check; wordpress_sec_2bdc2b00e491d84e1c4be3840dfdb408=Joerg+Maire%7C1513151186%7CUhvUAmySXCqogpTK2a3VXswRI27TT1uZ4JyFuE
c2fdb1687cdfe93d1e69ee67280286479cbbec709af1\n","stream":"stdout","time":"2017-12-11T21:38:52.636406516Z"}
{"log":"- BereqHeader Dnt: 1\n","stream":"stdout","time":"2017-12-11T21:38:52.636409503Z"}
{"log":"- BereqHeader Referer: https://www.example.com/wp-admin/post.php?post=45779\u0026action=edit\n","stream":"stdout","time":"2017-12-11T21:38:52.636411823Z"}
{"log":"- BereqHeader Upgrade-Insecure-Requests: 1\n","stream":"stdout","time":"2017-12-11T21:38:52.636414276Z"}
{"log":"- BereqHeader X-Forwarded-Host: www.example.com\n","stream":"stdout","time":"2017-12-11T21:38:52.636416562Z"}
{"log":"- BereqHeader X-Forwarded-Port: 443\n","stream":"stdout","time":"2017-12-11T21:38:52.636431001Z"}
{"log":"- BereqHeader X-Forwarded-Proto: https\n","stream":"stdout","time":"2017-12-11T21:38:52.636433369Z"}
{"log":"- BereqHeader X-Forwarded-Server: lb-proxy-1\n","stream":"stdout","time":"2017-12-11T21:38:52.636435511Z"}
{"log":"- BereqHeader X-Real-Ip: XX.XX.XX.XXX\n","stream":"stdout","time":"2017-12-11T21:38:52.636437721Z"}
{"log":"- BereqHeader X-Forwarded-For: XX.XX.XX.XXX, 10.42.176.203\n","stream":"stdout","time":"2017-12-11T21:38:52.636439886Z"}
{"log":"- BereqHeader X-Varnish: 5145027\n","stream":"stdout","time":"2017-12-11T21:38:52.636442148Z"}
{"log":"- VCL_call BACKEND_FETCH\n","stream":"stdout","time":"2017-12-11T21:38:52.636444268Z"}
{"log":"- VCL_return fetch\n","stream":"stdout","time":"2017-12-11T21:38:52.636446498Z"}
{"log":"- BackendOpen 29 boot.default XX.XX.XX.XXX 80 10.42.160.172 55142\n","stream":"stdout","time":"2017-12-11T21:38:52.636448594Z"}
{"log":"- BackendStart XX.XX.XX.XXX 80\n","stream":"stdout","time":"2017-12-11T21:38:52.636450886Z"}
{"log":"- Timestamp Bereq: 1513028321.551239 0.018578 0.018578\n","stream":"stdout","time":"2017-12-11T21:38:52.636453022Z"}
{"log":"- FetchError HTC status -1\n","stream":"stdout","time":"2017-12-11T21:38:52.636455259Z"}
{"log":"- BackendClose 29 boot.default\n","stream":"stdout","time":"2017-12-11T21:38:52.636457381Z"}
{"log":"- Timestamp Beresp: 1513028332.632241 11.099580 11.081002\n","stream":"stdout","time":"2017-12-11T21:38:52.636459871Z"}
{"log":"- Timestamp Error: 1513028332.632249 11.099587 0.000008\n","stream":"stdout","time":"2017-12-11T21:38:52.636462044Z"}
{"log":"- BerespProtocol HTTP/1.1\n","stream":"stdout","time":"2017-12-11T21:38:52.636464306Z"}
{"log":"- BerespStatus 503\n","stream":"stdout","time":"2017-12-11T21:38:52.636466426Z"}
{"log":"- BerespReason Service Unavailable\n","stream":"stdout","time":"2017-12-11T21:38:52.636468611Z"}
{"log":"- BerespReason Backend fetch failed\n","stream":"stdout","time":"2017-12-11T21:38:52.636472903Z"}
{"log":"- BerespHeader Date: Mon, 11 Dec 2017 21:38:52 GMT\n","stream":"stdout","time":"2017-12-11T21:38:52.636475145Z"}
{"log":"- BerespHeader Server: Varnish\n","stream":"stdout","time":"2017-12-11T21:38:52.636477305Z"}
{"log":"- VCL_call BACKEND_ERROR\n","stream":"stdout","time":"2017-12-11T21:38:52.636479423Z"}
{"log":"- BerespHeader Content-Type: text/html; charset=utf-8\n","stream":"stdout","time":"2017-12-11T21:38:52.636481577Z"}
{"log":"- BerespHeader Retry-After: 5\n","stream":"stdout","time":"2017-12-11T21:38:52.636483718Z"}
{"log":"- VCL_return deliver\n","stream":"stdout","time":"2017-12-11T21:38:52.636485834Z"}
{"log":"- Storage malloc Transient\n","stream":"stdout","time":"2017-12-11T21:38:52.636487925Z"}
{"log":"- ObjProtocol HTTP/1.1\n","stream":"stdout","time":"2017-12-11T21:38:52.63649004Z"}
{"log":"- ObjStatus 503\n","stream":"stdout","time":"2017-12-11T21:38:52.63649216Z"}
{"log":"- ObjReason Backend fetch failed\n","stream":"stdout","time":"2017-12-11T21:38:52.636494252Z"}
{"log":"- ObjHeader Date: Mon, 11 Dec 2017 21:38:52 GMT\n","stream":"stdout","time":"2017-12-11T21:38:52.63649637Z"}
{"log":"- ObjHeader Server: Varnish\n","stream":"stdout","time":"2017-12-11T21:38:52.636498539Z"}
{"log":"- ObjHeader Content-Type: text/html; charset=utf-8\n","stream":"stdout","time":"2017-12-11T21:38:52.63650064Z"}
{"log":"- ObjHeader Retry-After: 5\n","stream":"stdout","time":"2017-12-11T21:38:52.63650281Z"}
{"log":"- Length 284\n","stream":"stdout","time":"2017-12-11T21:38:52.636525843Z"}
{"log":"- BereqAcct 1770 6225 7995 0 0 0\n","stream":"stdout","time":"2017-12-11T21:38:52.636528516Z"}
{"log":"- End \n","stream":"stdout","time":"2017-12-11T21:38:52.636530668Z"}
Unfortunately, that "problem" has not reoccured since I've moved away from my Rancher-Cattle-Cluster to Kubernetes.
I guess it was a kind of networking-issue.
Intermittent 503 errors in varnish cache server due to backend fetch failure
I am getting 503 backend fetch failed errors intermittently after many successful requests. They keep on occuring at random times.
this is the varnish log sample for error:
<< Request >> 28612478
Begin req 28612475 rxreq
Timestamp Start: 1469259438.392350 0.000000 0.000000
Timestamp Req: 1469259438.392350 0.000000 0.000000
ReqStart 10.201.1.11 49351
ReqMethod GET
ReqURL some url
ReqProtocol HTTP/1.1
ReqHeader Content-Type: application/json
ReqHeader Host: SomeHost
ReqHeader Connection: Keep-Alive
ReqHeader User-Agent: Apache-HttpClient/4.1 (java 1.5)
ReqHeader X-Forwarded-For: 10.201.1.11
VCL_call RECV
ReqURL some url
ReqURL some url
VCL_return hash
VCL_call HASH
VCL_return lookup
VCL_call MISS
VCL_return fetch
Link bereq 28612479 fetch
Timestamp Fetch: 1469259441.892771 3.500421 3.500421
RespProtocol HTTP/1.1
RespStatus 503
RespReason Backend fetch failed
RespHeader Date: Sat, 23 Jul 2016 07:37:21 GMT
RespHeader Server: Varnish
RespHeader Content-Type: text/html; charset=utf-8
RespHeader Retry-After: 5
RespHeader X-Varnish: 28612478
RespHeader Age: 0
RespHeader Via: 1.1 varnish-v4
VCL_call DELIVER
VCL_return deliver
Timestamp Process: 1469259441.892804 3.500454 0.000034
RespHeader Content-Length: 285
Debug "RES_MODE 2"
RespHeader Connection: keep-alive
Timestamp Resp: 1469259441.892848 3.500498 0.000043
ReqAcct 776 0 776 242 285 527
End
I have tried tuning so many parameters, but not able to get rid of these errors.
Thanks in advance.