I have a Varnish 4.0.3 server on Centos 7.2. Varnish has three backends configured. I am receiving intermittent 503's from Varnish. I have pulled a tcpdump during a 503 event, and I saw:
Consumer makes request to Varnish
Varnish opens socket to backend.
Backend responds in < 500ms
Varnish sends a ACK,FIN to the Backend.
Varnish sends a 503 to the consumer.
Backend sends a ACK,FIN to Varnish
The requests that fail do not fundementally appear different from requests that are succeeding. The failure rate is ~1 per 20k requests.
- Begin req 2795361 rxreq
- Timestamp Start: 1464106437.502383 0.000000 0.000000
- Timestamp Req: 1464106437.502383 0.000000 0.000000
- ReqStart 10.14.X.X 43190
- ReqMethod GET
- ReqURL /service/v2/service/parameter/parameter/parameter
- ReqProtocol HTTP/1.1
- ReqHeader Accept: application/json
- ReqHeader Content-Type: application/json
- ReqHeader Host: UpsteamLoadBalancer:6081
- ReqHeader Connection: Keep-Alive
- ReqHeader User-Agent: Apache-HttpClient/4.2.4 (java 1.5)
- ReqHeader X-Forwarded-For: 10.14.X.X
- VCL_call RECV
- ReqURL /service/v2/service/parameter/parameter/parameter
- ReqUnset X-Forwarded-For: 10.14.X.X
- ReqHeader X-Forwarded-For: 10.14.X.X
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Debug "XXXX MISS"
- VCL_call MISS
- VCL_return fetch
- Link bereq 2795368 fetch
- Timestamp Fetch: 1464106442.526296 5.023913 5.023913
- Timestamp Process: 1464106442.526311 5.023929 0.000015
- RespHeader Date: Tue, 24 May 2016 16:14:02 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 2795367
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Service Unavailable
- RespReason Service Unavailable
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- RespHeader Content-Length: 281
- Debug "RES_MODE 2"
- RespHeader Connection: keep-alive
- Timestamp Resp: 1464106442.526356 5.023974 0.000045
- ReqAcct 290 0 290 211 281 492
- End
Your client is using HTTP to communicate with Varnish.
The HTTP response 503 represents mean "The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay".
So this error is sent by the Varnish server indicating above reason.
Regards,
Sudhansu
Related
We're using a Varnish Cache Proxy (in a docker container) to serve both SSR as well as AJAX request. The goal is to have a shared cache (for logged out) users independent how they access the data. The options are Nuxt (SSR) or an AJAX call in the browser (through Axios). The browser makes requests to https://api.foobar.tld/levels, while the SSR makes it's requests to the internal docker container at http://api-foobar-cache-proxy/levels.
This mostly works, except for when it doesn't (case 4, in the log). The question is why?
Varnish uses the following (custom) hash function:
sub vcl_hash {
hash_data(req.url);
std.log("X-DEBUG-url:" + req.url);
if (req.http.Origin ~ "foobar") {
hash_data("HOSTfoobar");
std.log("X-DEBUG-host:1-HOSTfoobar");
} elseif (req.http.host ~ "foobar") {
hash_data("HOSTfoobar");
std.log("X-DEBUG-host:2-HOSTfoobar");
}
if (req.http.Locale) {
hash_data(req.http.Locale);
std.log("X-DEBUG-locale:1-" + req.http.Locale);
} else {
hash_data("de");
std.log("X-DEBUG-locale:2-de");
}
return (lookup);
}
Debug Log:
varnishncsa -F '"%r" %{Varnish:handling}x %{VCL_Log:X-DEBUG-url}x %{VCL_Log:X-DEBUG-host}x %{VCL_Log:X-DEBUG-locale}x' | grep "/levels" output:
1: "GET http://api-foobar-cache-proxy/levels HTTP/1.1" miss /levels 1-HOSTfoobar 1-de // SSR first call (miss, expected, cold cache)
2: "GET http://api-foobar-cache-proxy/levels HTTP/1.1" hit /levels 1-HOSTfoobar 1-de // SSR second call (hit, expected)
3: "OPTIONS https://api.foobar.tld/levels HTTP/1.1" pass /levels 1-HOSTfoobar 2-de // Browser OPTIONS call (miss, expected)
4: "GET https://api.foobar.tld/levels HTTP/1.1" miss /levels 1-HOSTfoobar 1-de // Browser first call (miss, not expected)
5: "OPTIONS https://api.foobar.tld/levels HTTP/1.1" pass /levels 1-HOSTfoobar 2-de // Browser OPTIONS call (miss, expected)
6: "GET https://api.foobar.tld/levels HTTP/1.1" hit /levels 1-HOSTfoobar 1-de // Browser second call (hit, expected)
Miss Expected:
1: Cold Cache
Pass Expected:
3, 5: OPTIONS are not cached, only GET requests
Hit Expected:
2, 6: From cache
Miss Unexpected:
4: Why? for all of them the 3 hash_data params where: /levels, HOSTfoobar and de. Furthermore the return (lookup) should prevent the built in vcl_hash, unless I got that completely wrong.
Some additional Debug Output from varnishlog for the requests:
Comparing the varnishlog for the requests. The full log can be found here https://0bin.net/paste/2UYa-YSr#ykSOg11iJxJXA-CsUiK4es1gGekUDN4VL3wqVA9Jqdv, the relevant portions(?):
Request 1:
[...]
- VCL_call HASH
- VCL_Log X-DEBUG-url:/levels
- VCL_Log X-DEBUG-host:1-HOSTfoobar
- VCL_Log X-DEBUG-locale:1-de
- VCL_return lookup
- VCL_call MISS
- ReqHeader x-cache: miss
- VCL_return fetch
- Link bereq 32771 fetch
[...]
Request 2:
[...]
- VCL_call HASH
- VCL_Log X-DEBUG-url:/levels
- VCL_Log X-DEBUG-host:1-HOSTfoobar
- VCL_Log X-DEBUG-locale:1-de
- VCL_return lookup
- Hit 32771 3585.968366 3600.000000 0.000000
- VCL_call HIT
[...]
Request 4:
[...]
- VCL_call HASH
- VCL_Log X-DEBUG-url:/levels
- VCL_Log X-DEBUG-host:1-HOSTfoobar
- VCL_Log X-DEBUG-locale:1-de
- VCL_return lookup
- VCL_call MISS
- ReqHeader x-cache: miss
- VCL_return fetch
- Link bereq 98318 fetch
[...]
Request 6:
[...]
- VCL_call HASH
- VCL_Log X-DEBUG-url:/levels
- VCL_Log X-DEBUG-host:1-HOSTfoobar
- VCL_Log X-DEBUG-locale:1-de
- VCL_return lookup
- Timestamp Waitinglist: 1636723569.802992 0.183061 0.183061
- Hit 98318 3600.182590 3600.000000 0.000000
- VCL_call HIT
[...]
So, turns out as usual there was nothing wrong with the computer, the user was the problem. On the Backend I use API Platform, which can respond in different content types. Ex. if you request /levels in the browser it will deliver the Swagger UI, however if you request it with Accept: application/json it will deliver the JSON version. Changing the Accept header from */* on the SSR (imitating cURL) requests to application/json, */* did the trick.
I'm noticing this behavior on Varnish 6.5 where it's not making backend calls per the max-age cache control origin response, if the request is not frequently requested by clients.
Below's the expected behavior I see for a cache requested every 1 second. It has 20 seconds max-age cache-control header from origin:
Request 1:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:02 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 1183681 1512819
age: 17
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
Request 2:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:04 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 891620 1512819
age: 19
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
Request 3:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:05 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 1183687 1512819
age: 20
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
Request 4:
HTTP/2 200
date: Tue, 20 Jul 2021 02:02:06 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 854039 1183688
age: 1
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
You can see the Request #4 above makes a new origin request with the cache request id being 1183688.
Now if I wait a long while and make that same request, the cache age is pretty old and varnish does not make an origin request to cache a fresh object:
Request 5 after a while:
HTTP/2 200
date: Tue, 20 Jul 2021 02:10:08 GMT
content-type: application/json
content-length: 33692
server: Apache/2.4.25 (Debian)
x-ua-compatible: IE=edge;chrome=1
pragma:
cache-control: public, max-age=20
x-varnish: 1512998 1183688
age: 482
via: 1.1 varnish (Varnish/6.5)
vary: Accept-Encoding
x-cache: HIT
accept-ranges: bytes
I suppose I could start adding the Expires header from origin, but looking for explanation why varnish behaves this way if the request is infrequent. Thanks.
TTL header precedence in Varnish
Varnish does check the max-age directive, but there might be other factors can cause the TTL to be an unexpected value.
Here's the TTL precedence:
The Cache-Control header's s-maxage directive is checked.
When there's no s-maxage, Varnish will look for max-age to set its TTL.
When there's no Cache-Control header being returned, Varnish will use the Expires header to set its TTL.
When none of the above apply, Varnish will use the default_ttl runtime parameter as the TTL value. Its default value is 120 seconds.
Only then will Varnish enter vcl_backend_response, letting you change the TTL.
Any TTL being set in VCL using set beresp.ttl will get the upper hand, regardless of any other value being set via response headers.
Your specific situation
The best way to figure out what's going on is by running varnishlog and adding a filter for the URL you want to track.
Here's an example for the homepage:
varnishlog -g request -q "ReqUrl eq '/'"
The output will be extremely verbose, but will contain all the info you need.
Tags that are of particular interest are:
TTL see https://varnish-cache.org/docs/6.5/reference/vsl.html#varnish-shared-memory-logging
BerespHeader (specifically the Cache-Control backend response header)
RespHeader (specifically the Cache-Control response header)
Please also have a look at your VCL and check whether or not the TTL is changed by set beresp.ttl =.
What do I need to help you
In summary, if you want further assistance, please provide your full VCL, as well as a varnishlog extract for the transactions that is giving you to unexpected behavior.
Based on that information, we'll have a pretty good idea what's going on.
I am seeing frequent errors Varnish FetchError "Timed out reusing backend connection". Checked couple of blogs, but not find any resolution. Could you please help?
BereqHeader Accept-Encoding: gzip
BereqHeader X-Varnish: 37849780
VCL_call BACKEND_FETCH
VCL_return fetch
BackendOpen 36 NODEJS_2 xx.xx.xx.xx 9000 yy.yy.yy.yy 43309
Timestamp Bereq: 1605444526.456709 0.000102 0.000102
FetchError Timed out reusing backend connection
BackendClose 36 NODEJS_2
Timestamp Beresp: 1605444571.456893 45.000285 45.000183
Timestamp Error: 1605444571.456900 45.000292 0.000006
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Backend fetch failed
BerespHeader Date: Sun, 15 Nov 2020 12:49:31 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
Length 285
BereqAcct 2940 185 3125 0 0 0
End
The Timestamp Beresp: 1605444571.456893 45.000285 45.000183 tag in your VSL output indicates that your backend took 45.000183 seconds to generate its response, which triggered the first_byte_timeout.
In reality, your backend probably needed more than 45 seconds to generate the output, but Varnish just gave up after it hit the timeout.
Here are your options:
Increase the first_byte_timeout runtime parameter to a better number
Examine why your backend it taking so long
Although option 1 is theoretically viable, you really want to go for option 2, and figure out why it is taking so long for the backend to respond.
I have a varnish 6 setup with 26 backends and after ram upgrade I have a problem where it throws 503 error after reload for about 15-30s and varnishlog says its - FetchError backend reload_20190417_131210_1488.server15: unhealthy
Full headers from varnishlog:
<< BeReq >> 106235039
Begin bereq 106235038 fetch
Timestamp Start: 1555506951.751066 0.000000 0.000000
BereqMethod GET
BereqURL /_files/b6/ee/59/4f/af/b6ee594fafd3f13556216d89452f3dd4_1.jpg
BereqProtocol HTTP/1.1
BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103
Safari/537.36
BereqHeader Accept: image/webp,image/apng,image/,/*;q=0.8
BereqHeader Referer: http://www.example.com/
BereqHeader Accept-Language: lv
BereqHeader x-range: bytes=1135466-1135466
BereqHeader grace: none
BereqHeader X-Forwarded-For: 84.237.232.159
BereqHeader host: www.example.com
BereqHeader Surrogate-Capability: key=ESI/1.0
BereqHeader Accept-Encoding: gzip
BereqHeader X-Varnish: 106235039
VCL_call BACKEND_FETCH
VCL_return fetch
FetchError backend reload_20190417_131210_1488.server15: unhealthy
Timestamp Beresp: 1555506951.751106 0.000040 0.000040
Timestamp Error: 1555506951.751111 0.000045 0.000005
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Service Unavailable
BerespReason Backend fetch failed
BerespHeader Date: Wed, 17 Apr 2019 13:15:51 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
Length 286
BereqAcct 0 0 0 0 0 0
End
We had 16 GB ram will malloc 8 GB and now it is 32 GB with 23 GB malloc. We are using varnish 6 with VSF so it is pretty complex setup but it worked just fine. It compiles just fine without any error but throws 503 backend fetch fail to some domains after reload.
The FetchError is pretty clear, the backend is sick. Check it using varnishadm backend.health, it should tell you what is failing.
Showing your backend definition would help too.
We are currently loadtesting a Sharepoint environment hosted in IIS8.5 (On Windows 2012R2), using Loadrunner 11.52. We are using the WinINET based replay mechanism as there were SSL issues when trying to use the LR sockets implementation.
The authentication for the site is set to allow Windows Authentication. The users are all unique Active Directory users.
We have an issue where after starting 50 users, the users start failing due to them being unable to authenticate.
We have captured both successful (first 50 users) authentication and unsuccessful (users after the first 50) auth using Fiddler.
On initially loading the web page, the server returns a 401 with authentication headers as expected:
Request:
GET https://myserver/Pages/default.aspx HTTP/1.1
Cookie: WSS_FullScreenMode=false
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)
Accept-Encoding: gzip, deflate
Accept: */*
Host: myserver
Connection: Keep-Alive
Cache-Control: no-cache
Response:
HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/8.5
SPRequestGuid: ad71f69c-0b10-d049-46c6-1f0b1f7bd574
request-id: ad71f69c-0b10-d049-46c6-1f0b1f7bd574
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 2
SPIisLatency: 0
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4667
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Thu, 26 Mar 2015 07:13:44 GMT
Content-Length: 16
Proxy-Support: Session-Based-Authentication
401 UNAUTHORIZED
WinINET then handles this appropriately, returning a Kerberos auth token, which the server accepts:
GET https://myserver/Pages/default.aspx HTTP/1.1
Cookie: WSS_FullScreenMode=false
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)
Accept-Encoding: gzip, deflate
Accept: */*
Host: myserver
Connection: Keep-Alive
Cache-Control: no-cache
Authorization: Negotiate YIISMQYGKwYBBQUCoIISJTCCEiGgMDAuB...<long-auth-token-string>...=
Response:
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Mar 2015 07:13:44 GMT
Last-Modified: Thu, 26 Mar 2015 07:13:44 GMT
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
X-SharePointHealthScore: 0
X-AspNet-Version: 4.0.30319
SPRequestGuid: ad71f69c-db17-d049-46c6-15c51828a26e
request-id: ad71f69c-db17-d049-46c6-15c51828a26e
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 40
SPIisLatency: 0
WWW-Authenticate: Negotiate oYGzMIGwoAMKAQChCw...<long auth token>...=
Persistent-Auth: true
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4667
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Thu, 26 Mar 2015 07:13:44 GMT
Content-Length: 80492
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html dir="ltr" lang="en-US">
<head><meta name="GENERATOR" content="Microsoft SharePoint" /><
...
Some googling has suggested that the "YIIS" at the beginning of the auth token indicates that it has chosen Kerberos in response to the Negotiate auth option. This works fine for the first 50 users we start (each with unique AD user credentials).
However, for the 51st user, and each subsequent user, WinINET seems to start attempting to use NTLM authentication instead, which the server rejects.
Request (same as previously successful user):
GET https://myserver/Pages/default.aspx HTTP/1.1
Cookie: WSS_FullScreenMode=false
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)
Accept-Encoding: gzip, deflate
Accept: */*
Host: myserver
Connection: Keep-Alive
Cache-Control: no-cache
Response (same as previously successful user):
HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/8.5
SPRequestGuid: 6a71f69c-6b4f-d049-46c6-1e90257415f1
request-id: 6a71f69c-6b4f-d049-46c6-1e90257415f1
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 1
SPIisLatency: 0
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4667
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Thu, 26 Mar 2015 07:09:10 GMT
Content-Length: 16
Proxy-Support: Session-Based-Authentication
401 UNAUTHORIZED
Client then sends the request for auth:
GET https://myserver/Pages/default.aspx HTTP/1.1
Cookie: WSS_FullScreenMode=false
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)
Accept-Encoding: gzip, deflate
Accept: */*
Host: myserver
Connection: Keep-Alive
Cache-Control: no-cache
Authorization: Negotiate TlRMTVNTUAABAAA...<short NTLM auth token>...==
Some googling suggests that the T1R at the start of the auth token means it has chosen to use NTLM this time, instead of Kerberos which was used for all the users up till this one.
The server responds with another 401 to this:
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate TlRMTVNTUAACAAAACAAIADgAAAAVgonicMgf76hzy7QAAAAAAAAAAL4AvgBAAA...<long NTLM auth token>=
SPRequestGuid: 6a71f69c-cb6b-d049-46c6-14cbc16d1ea9
request-id: 6a71f69c-cb6b-d049-46c6-14cbc16d1ea9
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 1
SPIisLatency: 0
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4667
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Thu, 26 Mar 2015 07:09:10 GMT
Content-Length: 0
Proxy-Support: Session-Based-Authentication
Wininet then sends another auth request as follows:
GET https://myserver/Pages/default.aspx HTTP/1.1
Cookie: WSS_FullScreenMode=false
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)
Accept-Encoding: gzip, deflate
Accept: */*
Host: myserver
Connection: Keep-Alive
Cache-Control: no-cache
Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAKIAAACEAYQBu...<much longer NTLM token>...=
Which results in another 401:
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/8.5
SPRequestGuid: 6a71f69c-cb6b-d049-46c6-18ce53703ae9
request-id: 6a71f69c-cb6b-d049-46c6-18ce53703ae9
X-FRAME-OPTIONS: SAMEORIGIN
SPRequestDuration: 236
SPIisLatency: 0
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4667
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Thu, 26 Mar 2015 07:09:10 GMT
Content-Length: 0
Proxy-Support: Session-Based-Authentication
At this point WinINET seems to give up, and passes the 401 on to LoadRunner which then fails the user.
The first 50 users keep working as normal. We can also load another user manually through IE while those 50 are still running. We can also wait some time and eventually we can start more users.
So, there seems to be two issues here:
Why is WinINET changing to use NTLM auth instead of Kerberos after 50 users have been started?
Why is the server rejecting the NTLM auth.
Any ideas on what might be causing this, and/or how I can investigate further, would be very appreciated. My suspicion is that it may be some sort of Active Directory limit that blocks Kerberos from working for a period after 50 users have been authenticated, however I do not know how to prove/disprove that.
Thanks.