I have a varnish 6 setup with 26 backends and after ram upgrade I have a problem where it throws 503 error after reload for about 15-30s and varnishlog says its - FetchError backend reload_20190417_131210_1488.server15: unhealthy
Full headers from varnishlog:
<< BeReq >> 106235039
Begin bereq 106235038 fetch
Timestamp Start: 1555506951.751066 0.000000 0.000000
BereqMethod GET
BereqURL /_files/b6/ee/59/4f/af/b6ee594fafd3f13556216d89452f3dd4_1.jpg
BereqProtocol HTTP/1.1
BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103
Safari/537.36
BereqHeader Accept: image/webp,image/apng,image/,/*;q=0.8
BereqHeader Referer: http://www.example.com/
BereqHeader Accept-Language: lv
BereqHeader x-range: bytes=1135466-1135466
BereqHeader grace: none
BereqHeader X-Forwarded-For: 84.237.232.159
BereqHeader host: www.example.com
BereqHeader Surrogate-Capability: key=ESI/1.0
BereqHeader Accept-Encoding: gzip
BereqHeader X-Varnish: 106235039
VCL_call BACKEND_FETCH
VCL_return fetch
FetchError backend reload_20190417_131210_1488.server15: unhealthy
Timestamp Beresp: 1555506951.751106 0.000040 0.000040
Timestamp Error: 1555506951.751111 0.000045 0.000005
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Service Unavailable
BerespReason Backend fetch failed
BerespHeader Date: Wed, 17 Apr 2019 13:15:51 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
Length 286
BereqAcct 0 0 0 0 0 0
End
We had 16 GB ram will malloc 8 GB and now it is 32 GB with 23 GB malloc. We are using varnish 6 with VSF so it is pretty complex setup but it worked just fine. It compiles just fine without any error but throws 503 backend fetch fail to some domains after reload.
The FetchError is pretty clear, the backend is sick. Check it using varnishadm backend.health, it should tell you what is failing.
Showing your backend definition would help too.
Related
I am seeing frequent errors Varnish FetchError "Timed out reusing backend connection". Checked couple of blogs, but not find any resolution. Could you please help?
BereqHeader Accept-Encoding: gzip
BereqHeader X-Varnish: 37849780
VCL_call BACKEND_FETCH
VCL_return fetch
BackendOpen 36 NODEJS_2 xx.xx.xx.xx 9000 yy.yy.yy.yy 43309
Timestamp Bereq: 1605444526.456709 0.000102 0.000102
FetchError Timed out reusing backend connection
BackendClose 36 NODEJS_2
Timestamp Beresp: 1605444571.456893 45.000285 45.000183
Timestamp Error: 1605444571.456900 45.000292 0.000006
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Backend fetch failed
BerespHeader Date: Sun, 15 Nov 2020 12:49:31 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
Length 285
BereqAcct 2940 185 3125 0 0 0
End
The Timestamp Beresp: 1605444571.456893 45.000285 45.000183 tag in your VSL output indicates that your backend took 45.000183 seconds to generate its response, which triggered the first_byte_timeout.
In reality, your backend probably needed more than 45 seconds to generate the output, but Varnish just gave up after it hit the timeout.
Here are your options:
Increase the first_byte_timeout runtime parameter to a better number
Examine why your backend it taking so long
Although option 1 is theoretically viable, you really want to go for option 2, and figure out why it is taking so long for the backend to respond.
I'm trying to run a web application that I've built in an iframe on another domain. I'm able to load the page within the iframe, but any ajax requests on the page result in a 403 error as per below:
Request URL: https://test.mydomain.com/get_extra_services/
Request Method: POST
Status Code: 403 Forbidden
Remote Address: 207.38.86.14:443
Referrer Policy: no-referrer-when-downgrade
Connection: keep-alive
Content-Length: 1382
Content-Type: text/html
Date: Thu, 18 Jun 2020 22:57:41 GMT
Server: nginx
X-Content-Type-Options: nosniff
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 22
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: _ga=GA1.2.2146753382.1592180975; _gid=GA1.2.1219012919.1592286837
DNT: 1
Host: test.mydomain.com
Origin: https://test.mydomain.com
Referer: https://test.mydomain.com/order/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
X-CSRFToken: null
X-Requested-With: XMLHttpRequest
serviceid: 18
checked: 1
Any thoughts on what is causing this error?
Thanks!
You need to setup CORS to make requests from a different domain.
https://enable-cors.org/
I've set up my simple website with valid Let's Encrypt SSL certificate (from certbot). My nginx config is very short and trivial.
Website shows up correctly in latest Firefox. It shows 404 page, which is OK to me and should work as expected: 404 page.
If I try Google Chrome, i get an error:
The webpage at https://example.org/ might be temporarily down or it
may have moved permanently to a new web address.
ERR_INVALID_SIGNED_EXCHANGE
I assume that the application/signed-exchange header may cause this.
What is this header and should i remove it from response?
Request
GET / HTTP/1.1
Host: example.org
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,ru;q=0.8
DNT: 1
example.org example.org
Response
HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 29 Mar 2019 12:05:49 GMT
Content-Type: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Content-Length: 345
Connection: keep-alive
What to fix?
The Content-Type in the response is incorrect. It should be a single type, as Steffen Ullrich said. For a 404 page, I suspect you want Content-Type: text/html.
This may be something particular to your nginx config. On my server, 404 pages have Content-Type: text/html.
I have a Server 2012 R2 box running IIS. I've tried enabling compression for several sites running on that box, but I can't figure out why it won't work. My request headers all show accept-encoding, but the response headers are always Transfer-Encoding:chunked and Vary:Accept-Encoding. The following steps have been performed to try to get gzip compression working:
Dynamic and Static compression have been enabled on each site and at the machine level
Both compression methods are installed from Server Manager
Httpcompression and urlcompression nodes have been manually added to web.configs
Mime types are defined for compression
frequentHitThreshold has been set to 1, so all content should be compressed after the first attempt to access it
A trace has been done to see why compression isn't occurring. The only information I have is the code DYNAMIC_COMPRESSION_NOT_SUCCESS with a reason of 1.
Here are the headers:
GET http://redactedservername:8082/ HTTP/1.1
Host: redactedservername:8082
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: ASP.NET_SessionId=gnqovt55ggt22lycufudc0ns
`
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Vary: Accept-Encoding
Date: Wed, 22 Jun 2016 14:00:57 GMT
Transfer-Encoding: chunked
What other steps can be performed to get compression to work?
Compression was working, but ESET Antivirus was doing its job of monitoring web traffic. This modified the response and I didn't get gzip content encoding as expected. Disabling ESET and testing again showed that compression was functioning.
I have a Varnish 4.0.3 server on Centos 7.2. Varnish has three backends configured. I am receiving intermittent 503's from Varnish. I have pulled a tcpdump during a 503 event, and I saw:
Consumer makes request to Varnish
Varnish opens socket to backend.
Backend responds in < 500ms
Varnish sends a ACK,FIN to the Backend.
Varnish sends a 503 to the consumer.
Backend sends a ACK,FIN to Varnish
The requests that fail do not fundementally appear different from requests that are succeeding. The failure rate is ~1 per 20k requests.
- Begin req 2795361 rxreq
- Timestamp Start: 1464106437.502383 0.000000 0.000000
- Timestamp Req: 1464106437.502383 0.000000 0.000000
- ReqStart 10.14.X.X 43190
- ReqMethod GET
- ReqURL /service/v2/service/parameter/parameter/parameter
- ReqProtocol HTTP/1.1
- ReqHeader Accept: application/json
- ReqHeader Content-Type: application/json
- ReqHeader Host: UpsteamLoadBalancer:6081
- ReqHeader Connection: Keep-Alive
- ReqHeader User-Agent: Apache-HttpClient/4.2.4 (java 1.5)
- ReqHeader X-Forwarded-For: 10.14.X.X
- VCL_call RECV
- ReqURL /service/v2/service/parameter/parameter/parameter
- ReqUnset X-Forwarded-For: 10.14.X.X
- ReqHeader X-Forwarded-For: 10.14.X.X
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Debug "XXXX MISS"
- VCL_call MISS
- VCL_return fetch
- Link bereq 2795368 fetch
- Timestamp Fetch: 1464106442.526296 5.023913 5.023913
- Timestamp Process: 1464106442.526311 5.023929 0.000015
- RespHeader Date: Tue, 24 May 2016 16:14:02 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 2795367
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Service Unavailable
- RespReason Service Unavailable
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- RespHeader Content-Length: 281
- Debug "RES_MODE 2"
- RespHeader Connection: keep-alive
- Timestamp Resp: 1464106442.526356 5.023974 0.000045
- ReqAcct 290 0 290 211 281 492
- End
Your client is using HTTP to communicate with Varnish.
The HTTP response 503 represents mean "The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay".
So this error is sent by the Varnish server indicating above reason.
Regards,
Sudhansu