Troubleshooting 500 Error Due to Cookie Size - varnish

Question

Visitors to a website get 500 Internal Server error after browsing for a bit due to a tracking cookie that pushes the overall cookie size for our domain to over 4kb (it's a page view cookie so it appends the page name each time you visit a new page).
I can reproduce the issue using curl with a very large cookie payload. In doing this I've been able to verify where exactly the 500 is coming from (we go from Cloudflare to Varnish to the backend webserver). I've verified that the requests that fail don't make it to the webserver, so I believe Varnish is the one serving up the 500s. I have also watched the varnishlog and seen the 500s come through.
This is an example response from the varnishlog
-- VCL_return hash
-- VCL_call HASH
-- VCL_return lookup
-- Hit 57254162
-- VCL_call HIT
-- VCL_return deliver
-- RespProtocol HTTP/1.1
-- RespStatus 200
-- RespReason OK
-- RespHeader X-Powered-By: Express
-- RespHeader Date: Thu, 01 Aug 2019 23:05:52 GMT
-- RespHeader Content-Type: application/json; charset=utf-8
-- RespHeader Content-Length: 1174
-- RespHeader X-Varnish: 57156196 57519178
-- RespHeader Age: 86
-- RespHeader Via: 1.1 varnish-v4
-- VCL_call DELIVER
-- RespHeader X-Cache: HIT
-- RespUnset X-Powered-By: Express
-- VCL_return deliver
-- Timestamp Process: 1564700838.564547 0.000354 0.000354
-- RespHeader Accept-Ranges: bytes
-- Debug "RES_MODE 2"
-- RespHeader Connection: keep-alive
-- Error workspace_client overflow
-- RespProtocol HTTP/1.1
-- RespStatus 500
-- RespReason Internal Server Error
-- Timestamp Resp: 1564700838.564580 0.000387 0.000033
-- ReqAcct 10063 0 10063 0 0 0
-- End
Here is what I'd added to the vcl_recv section to remove the offending cookie
set req.http.Cookie = regsuball(req.http.Cookie, "_loc_[^;]+(; )?", "");
I don't understand what the significance is of two RespStatus entries here. Why is it 200, and then 500? I've also noticed that if I use curl, which is using HTTP/1.1 I get the 500, but if I use HTTPie, which uses HTTP/2, I get a 200. Is that expected? Would Varnish handle the cookie size differently depending on the http version?
*Edited: I think I've figured out that the difference in the two response statuses are that one is the delivery of the content to varnish, and the second is the delivery of the content to the client.

Answer 1

As the log says, the workspace is too small to accommodate the transaction (headers, notably), try increasing it:
varnishadm param.set workspace_client 128k
For a long explanation: varnish uses a "worspace" for each transaction. This is a chunk of memory used to allocate data, and the whole chunk is wiped out at the end of the transaction. The headers notably are copied into the workspace, and everytime to add or modify a header, it goes there too.
Ths issue here is that you don't have enough space. Earlier version would just panic, but it's now smarter and just produces a synthetic response with a 500 status. The trick is that it realizes the lack of workspace after the initial response has been copied, so you see both responses in the log.