Varnish Cache v4: Incorrect Backend Health Check Response - varnish

Question

I've setup Varnish Cache (4) in front of my CMS to help cache requests. In the event that my CMS goes down, I'd like to deliver cached items for a set grace period. I've followed many examples provided online but am running into an issue where Varnish doesn't recognize that my backend is down. When I manually shutdown the CMS the std.health(req.backend_hint)) continues to return true and attempts to retrieve an item from the backend, which then returns a 503 response.
Question: Am I incorrectly assuming std.health(req.backend_hint)) will recognize that my CMS is down?
Here is the VCL script I've been using to test:
sub vcl_recv {
# Initial State
set req.http.grace = "none";
set req.http.x-host = req.http.host;
set req.http.x-url = req.url;
return(hash);
}
sub vcl_backend_response {
set beresp.ttl = 10s;
set beresp.grace = 1h;
}
sub vcl_deliver {
# Copy grace to resp so we can tell from client
set resp.http.grace = req.http.grace;
# Add debugging headers to cache requests
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
}
else {
set resp.http.X-Cache = "MISS";
}
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# normal hit
return (deliver);
}
# We have no fresh content, lets look at the cache
elsif (std.healthy(req.backend_hint)) {
# Backend is healthy. Limit age to 10s.
if (obj.ttl + 10s > 0s) {
set req.http.grace = "normal(limited)";
return (deliver);
} else {
# No candidate for grace. Fetch a fresh object.
return(fetch);
}
} else {
# backend is sick - use full grace
if (obj.ttl + obj.grace > 0s) {
set req.http.grace = "full";
return (deliver);
} else {
# no graced object.
return (fetch);
}
}
}
Again, when I shutdown the CMS the std.healthy(req.backend_hint)) still reports the backend as healthy and never jumps to the final else statement.
Thanks for taking a look.

Answer 1

To properly use std.healthy you of course need to configure backend probes. So at the top of your VCL file you would first configure a probe:
probe site_probe {
.request =
"HEAD / HTTP/1.1"
"Host: example.com"
"Connection: close";
.interval = 5s; # check the health of each backend every 5 seconds
.timeout = 3s; # timing out after 1 second by default.
.window = 5; # If 3 out of the last 5 polls succeeded the backend is considered healthy, otherwise it will be marked as sick
.threshold = 3;
}
Make sure to replace example.com with your main website domain. It is important to put (or omit) the www. prefix so that the probe will not get redirect and marked as failed.
And of course your backend definition should be configured to use the defined probe:
backend default {
.host = "127.0.0.1";
.port = "8080";
.probe = site_probe;
}