Varnish - Seeing backend_busy without max_connection - varnish

Question

Using Varnish 3.0.4, we're seeing a lot of backend_busy, but we do not have set max_connection setting in any of our backends.
I'm trying to find out why because it seems some clients requests get aborted.
what could cause a backend_busy ?
thanks for any tips
here's my varnishstat -1
client_conn 0 0.00 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 0 0.00 Client requests received
cache_hit 0 0.00 Cache hits
cache_hitpass 0 0.00 Cache hits for pass
cache_miss 0 0.00 Cache misses
backend_conn 0 0.00 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 386013 8.38 Backend conn. too many
backend_fail 0 0.00 Backend conn. failures
backend_reuse 1140751 24.77 Backend conn. reuses
backend_toolate 292966 6.36 Backend conn. was closed
backend_recycle 31329 0.68 Backend conn. recycles
backend_retry 364845 7.92 Backend conn. retry
fetch_head 93857 2.04 Fetch head
fetch_length 0 0.00 Fetch with Length
fetch_chunked 0 0.00 Fetch chunked
fetch_eof 1007 0.02 Fetch EOF
fetch_bad 766163 16.63 Fetch had bad headers
fetch_close 16152 0.35 Fetch wanted close
fetch_oldhttp 783263 17.00 Fetch pre HTTP/1.1 closed
fetch_zero 14372 0.31 Fetch zero len
fetch_failed 83 0.00 Fetch failed
fetch_1xx 792868 17.21 Fetch no body (1xx)
fetch_204 28600 0.62 Fetch no body (204)
fetch_304 0 0.00 Fetch no body (304)
n_sess_mem 0 . N struct sess_mem
n_sess 849 . N struct sess
n_object 0 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 0 . N struct objectcore
n_objecthead 0 . N struct objecthead
n_waitinglist 0 . N struct waitinglist
n_vbc 674 . N struct vbc
n_wrk 1819 . N worker threads
n_wrk_create 155 0.00 N worker threads created
n_wrk_failed 237296 5.15 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 237394 5.15 work request queue length
n_wrk_queued 237937 5.17 N queued work requests
n_wrk_drop 128 0.00 N dropped work requests
n_backend 96 . N backends
n_expired 135 . N expired objects
n_lru_nuked 7710 . N LRU nuked objects
n_lru_moved 0 . N LRU moved objects
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 49182 1.07 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 25 0.00 Total Sessions
s_req 104479 2.27 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 205233 4.46 Total pass
s_fetch 0 0.00 Total fetch
s_hdrbytes 0 0.00 Total header bytes
s_bodybytes 1093243 23.73 Total body bytes
sess_closed 0 0.00 Session Closed
sess_pipeline 385979 8.38 Session Pipeline
sess_readahead 1140751 24.77 Session Read Ahead
sess_linger 758 0.02 Session Linger
sess_herd 482049 10.46 Session herd
shm_records 823074 17.87 SHM records
shm_writes 470696378 10218.54 SHM writes
shm_flushes 10834209993 235204.18 SHM flushes due to overflow
shm_cont 167868 3.64 SHM MTX contention
shm_cycles 19 0.00 SHM cycles through buffer
sms_nreq 18 0.00 SMS allocator requests
sms_nobj 1092005 . SMS outstanding allocations
sms_nbytes 635568 . SMS outstanding bytes
sms_balloc 85592445 . SMS bytes allocated
sms_bfree 5323477 . SMS bytes freed
backend_req 42 0.00 Backend requests made
n_vcl 44231 0.96 N vcl total
n_vcl_avail 37 0.00 N vcl available
n_vcl_discard 23820 0.52 N vcl discarded
n_ban 0 . N total active bans
n_ban_add 0 0.00 N new bans added
n_ban_retire 2136286212 46377.49 N old bans deleted
n_ban_obj_test 2136286212 46377.49 N objects tested
n_ban_re_test 857240 18.61 N regexps tested against
n_ban_dups 1 0.00 N duplicate bans removed
hcb_nolock 1 0.00 HCB Lookups without lock
hcb_lock 0 0.00 HCB Lookups with lock
hcb_insert 28435 0.62 HCB Inserts
esi_errors 14884 0.32 ESI parse errors (unlock)
esi_warnings 246931 5.36 ESI parse warnings (unlock)
accept_fail 218496 4.74 Accept failures
client_drop_late 5603745 121.65 Connection dropped late
uptime 29071691616 631128.92 Client uptime
dir_dns_lookups 46063 1.00 DNS director lookups
dir_dns_failed 685875 14.89 DNS director failed lookups
dir_dns_hit 356122 7.73 DNS director cached lookups hit
dir_dns_cache_full 355962 7.73 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 0 0.00 Gunzip operations
LCK.sms.creat 0 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 0 0.00 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 0 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 0 0.00 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 0 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 0 0.00 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 0 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 0 0.00 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 0 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 0 0.00 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 0 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 0 0.00 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 0 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 0 0.00 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 0 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 0 0.00 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 0 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 0 0.00 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 0 0.00 Created locks
LCK.objhdr.destroy 0 0.00 Destroyed locks
LCK.objhdr.locks 0 0.00 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 0 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 0 0.00 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 0 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 0 0.00 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 0 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 0 0.00 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 0 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 0 0.00 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 0 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 0 0.00 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 0 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 0 0.00 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 0 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 0 0.00 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 0 0.00 Allocator requests
SMA.s0.c_fail 0 0.00 Allocator failures
SMA.s0.c_bytes 0 0.00 Bytes allocated
SMA.s0.c_freed 0 0.00 Bytes freed
SMA.s0.g_alloc 0 . Allocations outstanding
SMA.s0.g_bytes 0 . Bytes outstanding
SMA.s0.g_space 0 . Bytes available
SMA.Transient.c_req 0 0.00 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 0 0.00 Bytes allocated
SMA.Transient.c_freed 0 0.00 Bytes freed
SMA.Transient.g_alloc 0 . Allocations outstanding
SMA.Transient.g_bytes 0 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.default(127.0.0.1,,8080).vcls 0 . VCL references
VBE.default(127.0.0.1,,8080).happy 0 . Happy health probes
VBE.wiki(127.0.0.1,,8081).vcls 0 . VCL references
VBE.wiki(127.0.0.1,,8081).happy 0 . Happy health probes
VBE.tiles(127.0.0.1,,9090).vcls 0 . VCL references
VBE.tiles(127.0.0.1,,9090).happy 0 . Happy health probes
VBE.geoserver(127.0.0.1,,43074).vcls 0 . VCL references
VBE.geoserver(127.0.0.1,,43074).happy 0 . Happy health probes
VBE.iconserver(127.0.0.1,,43077).vcls 0 . VCL references
VBE.iconserver(127.0.0.1,,43077).happy 0 . Happy health probes
VBE.uploadserver(127.0.0.1,,43079).vcls 0 . VCL references
VBE.uploadserver(127.0.0.1,,43079).happy 0 . Happy health probes
VBE.graphserver(127.0.0.1,,43080).vcls 0 . VCL references
VBE.graphserver(127.0.0.1,,43080).happy 0 . Happy health probes
VBE.kmlserver(127.0.0.1,,43082).vcls 0 . VCL references
VBE.kmlserver(127.0.0.1,,43082).happy 0 . Happy health probes
VBE.feedbackserver(127.0.0.1,,43085).vcls 0 . VCL references
VBE.feedbackserver(127.0.0.1,,43085).happy 0 . Happy health probes
VBE.weatherserver(127.0.0.1,,43086).vcls 0 . VCL references
VBE.weatherserver(127.0.0.1,,43086).happy 0 . Happy health probes
VBE.analyticsserver(127.0.0.1,,43087).vcls 0 . VCL references
VBE.analyticsserver(127.0.0.1,,43087).happy 0 . Happy health probes
VBE.chartserver(127.0.0.1,,43088).vcls 0 . VCL references
VBE.chartserver(127.0.0.1,,43088).happy 0 . Happy health probes
VBE.watcher(127.0.0.1,,8090).vcls 0 . VCL references
VBE.watcher(127.0.0.1,,8090).happy 0 . Happy health probes
VBE.render(127.0.0.1,,8040).vcls 0 . VCL references
VBE.render(127.0.0.1,,8040).happy 0 . Happy health probes
VBE.route(127.0.0.1,,43073).vcls 0 . VCL references
VBE.route(127.0.0.1,,43073).happy 0 . Happy health probes
VBE.osmsync(127.0.0.1,,9010).vcls 0 . VCL references
VBE.osmsync(127.0.0.1,,9010).happy 0 . Happy health probes
VBE.search(127.0.0.1,,8100).vcls 0 . VCL references
VBE.search(127.0.0.1,,8100).happy 0 . Happy health probes
VBE.indexserver(127.0.0.1,,8101).vcls 0 . VCL references
VBE.indexserver(127.0.0.1,,8101).happy 0 . Happy health probes
VBE.rewriteproxy(127.0.0.1,,9030).vcls 0 . VCL references
VBE.rewriteproxy(127.0.0.1,,9030).happy 0 . Happy health probes
VBE.pomplo(127.0.0.1,,9081).vcls 0 . VCL references
VBE.pomplo(127.0.0.1,,9081).happy 0 . Happy health probes
VBE.admin(127.0.0.1,,9000).vcls 0 . VCL references
VBE.admin(127.0.0.1,,9000).happy 0 . Happy health probes
VBE.graphite(127.0.0.1,,7070).vcls 0 . VCL references
VBE.graphite(127.0.0.1,,7070).happy 0 . Happy health probes
VBE.notificationserver(127.0.0.1,,8070).vcls 0 . VCL references
VBE.notificationserver(127.0.0.1,,8070).happy 0 . Happy health probes
VBE.orient(127.0.0.1,,2480).vcls 0 . VCL references
VBE.orient(127.0.0.1,,2480).happy 0 . Happy health probes
VBE.geoip(127.0.0.1,,43930).vcls 0 . VCL references
VBE.geoip(127.0.0.1,,43930).happy 0 . Happy health probes
here's my vcl (included in default.vcl)
import std;
acl internal {
"localhost";
"192.168.0.0"/16;
"10.0.0.0"/8;
}
probe status {
.url = "/status.json";
.timeout = 100 ms;
.interval = 1s;
.threshold = 1;
}
probe rootCheck {
.url = "/";
.timeout = 100 ms;
.interval = 1s;
.threshold = 1;
}
backend tiles {
.host = "127.0.0.1";
.port = "9090";
.probe = {
.url = "/serverInfo.json";
.timeout = 100 ms;
.interval = 1s;
.threshold = 1;
}
}
backend geoserver {
.host = "127.0.0.1";
.port = "43074";
.probe = rootCheck;
}
backend iconserver {
.host = "127.0.0.1";
.port = "43077";
.probe = rootCheck;
}
backend uploadserver {
.host = "127.0.0.1";
.port = "43079";
.probe = rootCheck;
}
backend graphserver {
.host = "127.0.0.1";
.port = "43080";
.probe = rootCheck;
}
backend kmlserver {
.host = "127.0.0.1";
.port = "43082";
.probe = rootCheck;
}
backend weatherserver {
.host = "127.0.0.1";
.port = "43086";
.probe = {
.url = "/status";
.timeout = 100 ms;
.interval = 1s;
.threshold = 1;
}
}
backend analyticsserver {
.host = "127.0.0.1";
.port = "43087";
.probe = status;
}
backend chartserver {
.host = "127.0.0.1";
.port = "43088";
.probe = status;
}
backend watcher {
.host = "127.0.0.1";
.port = "8090";
.probe = {
.url = "/test";
.timeout = 100 ms;
.interval = 1s;
.threshold = 1;
}
}
backend route {
.host = "127.0.0.1";
.port = "43073";
.probe = status;
}
backend osmsync {
.host = "127.0.0.1";
.port = "9010";
.probe = {
.url = "/state.json";
.timeout = 100 ms;
.interval = 1s;
.threshold = 1;
}
}
backend search {
.host = "127.0.0.1";
.port = "8100";
.probe = status;
}
backend indexserver {
.host = "127.0.0.1";
.port = "8101";
.probe = rootCheck;
}
backend rewriteproxy {
.host = "127.0.0.1";
.port = "9030";
.probe = rootCheck;
}
backend webserver {
.host = "127.0.0.1";
.port = "9081";
.probe = rootCheck;
}
backend admin {
.host = "127.0.0.1";
.port = "9000";
.probe = rootCheck;
}
backend graphite {
.host = "127.0.0.1";
.port = "7070";
.probe = rootCheck;
}
backend notificationserver {
.host = "127.0.0.1";
.port = "8070";
.probe = status;
}
backend orient {
.host = "127.0.0.1";
.port = "2480";
}
backend geoip {
.host = "127.0.0.1";
.port = "43930";
.probe = status;
}
sub allow_only_internal {
if (!client.ip ~ internal) {
error 405 "Not allowed.";
}
}
sub vcl_recv {
if (req.url ~ "\.(png|gif|jpg|swf|css|js|bis|apk|plist|ipa|woff|svg|eot|ttf|json)(\?.*|)$") {
unset req.http.cookie;
}
if (req.url ~ "^/admin") {
call allow_only_internal;
}
if( req.http.host ~ "^tile" ) {
unset req.http.cookie;
set req.http.host = "tiles";
set req.backend = tiles;
} else if( req.http.host ~ "^data" ) {
set req.backend = geoserver;
} else if( req.http.host ~ "^icon" ) {
unset req.http.cookie;
set req.backend = iconserver;
} else if( req.http.host ~ "^upload" ) {
set req.backend = uploadserver;
} else if( req.http.host ~ "^graphite" ) { // must be before graph
if( req.url !~ "^/render" ) { //allow render API anywhere
call allow_only_internal;
}
set req.backend = graphite;
} else if( req.http.host ~ "^graph" ) {
set req.backend = graphserver;
} else if( req.http.host ~ "^kml" ) {
set req.backend = kmlserver;
} else if( req.http.host ~ "^notification" ) {
set req.backend = notificationserver;
} else if( req.http.host ~ "^osmsync" ) {
set req.backend = osmsync;
} else if( req.http.host ~ "^watcher" ) {
set req.backend = watcher;
} else if( req.http.host ~ "^route" ) {
set req.backend = route;
} else if( req.http.host ~ "^search" ) {
set req.backend = search;
} else if( req.http.host ~ "^index" ) {
call allow_only_internal;
set req.backend = indexserver;
} else if( req.http.host ~ "^weather" ) {
set req.backend = weatherserver;
} else if( req.http.host ~ "^analytics" ) {
set req.backend = analyticsserver;
} else if( req.http.host ~ "^chart" ) {
set req.backend = chartserver;
} else if( req.http.host ~ "^geoip" ) {
set req.backend = geoip;
} else if( req.http.host ~ "^rewriteproxy" || req.http.host ~ "^showroom" ) {
set req.backend = rewriteproxy;
} else if( req.http.host ~ "^admin" ) {
call allow_only_internal;
set req.backend = admin;
} else if( req.http.host ~ "^orient" ) {
call allow_only_internal;
set req.backend = orient;
}
//url matching is at the bottom to avoid confusion
else if( req.url ~ "^/query" ) {
set req.backend = geoserver;
} else if( req.url ~ "^/data/" ) {
set req.url = regsub( req.url, "^/data(/.*)", "\1" );
set req.backend = geoserver;
} else if( req.url ~ "^/icon/" ) {
set req.url = regsub( req.url, "^/icon(/.*)", "\1" );
set req.backend = iconserver;
} else if( req.url ~ "^/graph/" ) {
set req.url = regsub( req.url, "^/graph(/.*)", "\1" );
set req.backend = graphserver;
} else if( req.http.host ~ "^(www\.|static|map)" ) {
set req.backend = webserver;
} else {
// todo: handle HTTPS with req.http.X-Forwarded-Proto
// redirect dev and demo to avoid dead links for old urls
error 750 regsub(req.http.host, "^(w+\.|dev\.|demo\.)?", "http://www.") + req.url;
}
}
sub vcl_deliver {
if (resp.http.Location ~ ".*:8080.*") {
set resp.http.Location = regsub(resp.http.Location, ":[0-9]+", "");
}
if (req.backend == graphite) {
set resp.http.Access-Control-Allow-Origin = "*";
set resp.http.Access-Control-Allow-Methods = "GET, OPTIONS";
set resp.http.Access-Control-Allow-Headers = "origin, authorization, accept";
}
unset resp.http.x-url;
unset resp.http.x-host;
return (deliver);
}
sub vcl_fetch {
if (req.url ~ "\.(png|gif|jpg|swf|css|js|bis|apk|plist|ipa|woff|svg|eot|ttf|json)(\?.*|)$") {
unset beresp.http.set-cookie;
}
if( beresp.http.content-type ~ "application/json"
|| beresp.http.content-type ~ "application/javascript"
|| (beresp.http.content-type ~ "application/octet-stream" && req.url ~ "\.bis$")
|| beresp.http.content-type ~ "text/javascript"
|| beresp.http.content-type ~ "text/plain"
|| beresp.http.content-type ~ "text/html"
|| beresp.http.content-type ~ "text/css" ) {
set beresp.do_gzip = true;
}
//allow ban lurker to work. https://www.varnish-software.com/static/book/Cache_invalidation.html#smart-bans
set beresp.http.x-url = req.url;
set beresp.http.x-host = req.http.host;
}
sub vcl_pipe {
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
}
}
sub vcl_recv {
if (req.http.Upgrade ~ "(?i)websocket") {
return (pipe);
}
}
sub vcl_error {
if (obj.status == 750) {
set obj.http.Location = obj.response;
set obj.status = 302;
return (deliver);
}
set obj.http.Content-Type = "text/html; charset=utf-8";
set obj.http.Retry-After = "3";
if( req.http.host ~ "^map" ) {
synthetic regsuball(regsuball(std.fileread("/etc/varnish/error_map.html"), "\$obj\.status\$", obj.status), "\$obj\.response\$", obj.response);
} else {
synthetic regsuball(regsuball(std.fileread("/etc/varnish/error.html"), "\$obj\.status\$", obj.status), "\$obj\.response\$", obj.response);
}
return (deliver);
}

Answer 1

That would normally be because none of your backends in the requested director are healthy at the time.
None of your servers seems to be used in your VCL nor have any health probes. So it's hard to know why exactly this happens. You should also really be using the backends you define and do health checks on them.
When you see something more like the following for all backends your varnishstat is least giving you some useful information:
VBE.wiki(127.0.0.1,,8081).vcls 1
VBE.wiki(127.0.0.1,,8081).happy 18446744073709551615
You can also check current status and short history by "debug.health" in your varnishadm console.

Answer 2

Try updating your libvarnishapi lib — this helped me (ubuntu):
sudo apt-get install libvarnishapi1
and reboot.
If you are using a wrong version (mine was libvarnishapi-dev), you don't get varnishstat working properly.

Answer 3

By default when backend_busy is incremented, that means the client receives a 5xx error response. However by using VCL (Varnish Configuration Language), it seems that you can apparently configure Varnish to recover from a "busy backend" by using a different backend, or by serving an outdated or synthetic response.
(Source)
Note: I think backend_busy is different from backend_unhealthy.
Actually Varnish periodically pings the backend to make sure it is still up and responsive. If it doesn’t receive a 200 response quickly enough, the backend is marked as unhealthy and every new request to it increments backend_unhealthy until the backend recovers and sends a timely 200 response.