Varnish Cache is not sending Content-Length header to backend server - varnish

i am trying to cache POST request by using Varnish cache. I am using vcl file as
vcl 4.0;
import std;
import bodyaccess;
backend default {
.host = "**.***.**.***";
.port = "**";
}
sub vcl_recv {
unset req.http.x-cache;
unset req.http.X-Body-Len;
if (req.method == "POST") {
std.log("Will cache POST for: " + req.http.host + req.url);
std.cache_req_body(3000KB);
set req.http.X-Body-Len = bodyaccess.len_req_body();
if (req.http.X-Body-Len == "-1") {
return(synth(400, "The request body size exceeds the limit"));
}
return (hash);
}
}
sub vcl_hash {
# To cache POST and PUT requests
if (req.http.X-Body-Len) {
bodyaccess.hash_req_body();
} else {
hash_data("");
}
}
sub vcl_backend_fetch {
if (bereq.http.X-Body-Len) {
set bereq.method = "POST";
}
}
sub vcl_backend_response {
set beresp.ttl = 86400s;
unset beresp.http.Cache-Control;
set beresp.http.Cache-Control = "public, max-age=86400";
}
sub vcl_hit {
set req.http.x-cache = "hit";
}
sub vcl_miss {
set req.http.x-cache = "miss";
}
sub vcl_pass {
set req.http.x-cache = "pass";
}
sub vcl_pipe {
set req.http.x-cache = "pipe uncacheable";
}
sub vcl_synth {
set resp.http.x-cache = "synth synth";
}
sub vcl_deliver {
if (obj.uncacheable) {
set req.http.x-cache = req.http.x-cache + " uncacheable" ;
} else {
set req.http.x-cache = req.http.x-cache + " cached" ;
}
set resp.http.x-cache = req.http.x-cache;
}
When i am doing post request directly it is working fine but as soon as I go through Cache Server, backend Server is returning "HTTP Error 411. The request must be chunked or have a content length."
On debugging furthet
What I found from varnishlog is
3 BereqMethod b POST
3 BereqURL b /someurl
3 BereqProtocol b HTTP/1.1
3 BereqHeader b Content-Type: application/json
3 BereqHeader b Accept-Language: en-US
3 BereqHeader b User-Agent: PostmanRuntime/7.24.0
3 BereqHeader b Accept: */*
3 BereqHeader b Postman-Token: 6fe71b81-841d-400f-957a-45b6518bfca6
3 BereqHeader b Host: *.*.*.*
3 BereqHeader b X-Forwarded-For: *.*.*.*
3 BereqHeader b X-Body-Len: 161
3 BereqHeader b Accept-Encoding: gzip
3 BereqHeader b x-cache: miss
3 BereqMethod b GET
3 BereqHeader b X-Varnish: 3
Varnish is not sending Content-Length where backend server needs it
How can I set this content-length

You can set it as any other header, i.e. in vcl_recv:
set req.http.Content-Length = "value"

Related

Varnish add more slash in URL

My varnish cache is accelerator mode and it cause adding more slash after URL.
I check with gtmetrix:
Avoid landing page redirects for the following chain of redirected URLs.
https://www.example2.com/
https://www.example2.com//
https://www.example2.com/
and with google pagespeed test:
Reduce server response time
In our test, your server responded in 1.6 seconds. There are many factors that can slow down your server response time.
Here is my config:
vcl 4.0;
backend default {
.host = "localhost";
.port = "8080";
}
acl purger {
"localhost";
"XXX.XXX.XXX.XXX";
}
sub vcl_recv {
if (client.ip != "127.0.0.1" && req.http.host ~ "XXX.com") {
set req.http.x-redir = "https://www.example.com/" + req.url;
return(synth(850, ""));
}
if (client.ip != "127.0.0.1" && req.http.host ~ "YYY.com") {
set req.http.x-redir = "https://www.example2.com/" + req.url;
return(synth(850, ""));
}
if (req.method == "PURGE") {
if (!client.ip ~ purger) {
return(synth(405, "This IP is not allowed to send PURGE requests."));
}
return (purge);
}
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.http.Authorization || req.method == "POST") {
return (pass);
}
if (req.url ~ "/feed") {
return (pass);
}
set req.http.cookie = regsuball(req.http.cookie, "wp-settings-\d+=[^;]+(; )?", "");
set req.http.cookie = regsuball(req.http.cookie, "wp-settings-time-\d+=[^;]+(; )?", "");
if (req.http.cookie == "") {
unset req.http.cookie;
}
}
sub vcl_synth {
if (resp.status == 850) {
set resp.http.Location = req.http.x-redir;
set resp.status = 302;
return (deliver);
}
}
sub vcl_purge {
set req.method = "GET";
set req.http.X-Purger = "Purged";
return (restart);
}
sub vcl_backend_response {
set beresp.ttl = 24h;
set beresp.grace = 1h;
}
sub vcl_deliver {
if (req.http.X-Purger) {
set resp.http.X-Purger = req.http.X-Purger;
}
}
Somebody know the reason why varnish do that ?
req.url always includes a / at the beginning.
so correct would be:
set req.http.x-redir = "https://www.example2.com" + req.url;
your question does not correspond to the example1/example2 as it includes XXX and YYY
if you can post a less redacted VCL - i may can help you a bit more
you may need a != "/" on req.url before redirecting. as this would result in a //

Varnish configuration (always MISS)

I had this working at one point, but now it is broke again (may be because of some code changes in php), but default.vcl hasn't changed.
varnish version is
varnishd (varnish-3.0.7 revision f544cd8)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2014 Varnish Software AS
here are some rules that i want varnish to follow
I need varnish to cache all pages and remove php session or any other cookies except when there is a special cookie "sh_loggedin" present
/social-signup should be pass because it creates above cookie when user logs in
/content should be pass because that is admin area
add HIT / MISS or counter related things to header so i know if varnish is working
ignore all cache control or age headers, and make sure varnish HITS everything except when "sh_loggedin" cookie present
js,css,images, etc should be always served by varnish regardless
allow google analytics tracking to work
Here is what it looks like
backend default {
.host = "127.0.0.1";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}
acl purge {
"localhost";
"127.0.0.1";
}
sub vcl_recv {
if(req.url ~ "/social-signup") {
return (pass);
}
if(req.url ~ "/scripts") {
return (pass);
}
if(req.url ~ "/content") {
return (pass);
}
if(req.url ~ "/api") {
return (pass);
}
// Remove has_js and Google Analytics __* cookies.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
// Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
if (!req.backend.healthy) {
unset req.http.Cookie;
}
if (req.request == "GET" && req.url ~ "^/varnishcheck$") {
error 200 "Varnish is Ready";
}
if(req.url ~ "/blog") {
return (pass);
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "PURGE" &&
req.request != "DELETE") {
# Non-RFC2616 or CONNECT which is weird.
return (pipe);
}
# We only deal with GET, PURGE and HEAD by default.
if (req.request != "GET" && req.request != "HEAD" && req.request != "PURGE") {
return (pass);
}
# --- PURGE ---
if (req.request == "PURGE") {
# Check if the ip coresponds with the acl purge
if (!client.ip ~ purge) {
# Return error code 405 (Forbidden) when not
error 405 "Not allowed.";
}
return (lookup);
}
# --- PASSTHROUGH ---
# Always cache things with these extensions.
if (req.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
unset req.http.cookie;
return (lookup);
}
if(req.url ~ "/scripts") {
return (pass);
}
if(req.url ~ "/api") {
return (pass);
}
# Skip the Varnish cache for install, update, and cron.
if (req.url ~ "install\.php|update\.php|cron\.php") {
return (pass);
}
# Pass server-status.
if (req.url ~ ".*/server-status$") {
return (pass);
}
# Support for Pressflow Cookie-Cache Bypass.
if (req.http.cookie ~ "NO_CACHE") {
return (pass);
}
# Force lookup if the request is a no-cache request from the client.
if (req.http.Cache-Control ~ "no-cache") {
return (pass);
}
# Don't check cache if Drupal SESSION is set.
if (req.http.cookie ~ "SESS") {
return (pass);
}
# We "hide" the non-session cookies.
if (req.http.cookie) {
set req.http.X-Varnish-Cookie = req.http.cookie;
unset req.http.cookie;
}
# --- MISC ---
# Normalize the Accept-Encoding header
# as per: http://varnish-cache.org/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
# No point in compressing these.
unset req.http.Accept-Encoding;
}
else if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
}
else if (req.http.Accept-Encoding ~ "deflate") {
# Next, try deflate if it is supported.
set req.http.Accept-Encoding = "deflate";
}
else {
# Unknown or deflate algorithm.
unset req.http.Accept-Encoding;
}
}
# Let's have a little grace.
set req.grace = 5m;
return (lookup);
}
sub vcl_hash {
if (req.http.cookie) {
hash_data(req.http.cookie);
}
}
# Strip any cookies before an image/js/css is inserted into cache.
sub vcl_fetch {
remove beresp.http.Cache-Control;
remove beresp.http.Age;
set beresp.http.Age = "10";
set beresp.http.Cache-Control = "public";
set beresp.grace = 5m;
# These status codes should always pass through and never cache.
if (beresp.status == 503 || beresp.status == 500) {
set beresp.http.X-Cacheable = "NO: obj.status";
set beresp.http.X-Cacheable-status = beresp.status;
return (hit_for_pass);
}
if (req.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)(\?[a-z0-9]+)?$") {
unset beresp.http.set-cookie;
}
else if (beresp.http.Cache-Control) {
unset beresp.http.Expires;
}
if(req.url !~ "/content") {
unset beresp.http.Expires;
}
if (bereq.http.Cookie !~ "__sh_loggedin__") {
unset bereq.http.Cookie;
unset beresp.http.Set-Cookie;
}
if (beresp.status == 301) {
set beresp.ttl = 1h;
return(deliver);
}
# All tests passed, therefore item is cacheable
set beresp.http.X-Cacheable = "YES";
}
# Set a header to track a cache HIT/MISS.
sub vcl_deliver {
set resp.http.cache-control = "max-age = 3600";
set resp.http.Age = "10";
if (obj.hits > 0) {
set resp.http.X-Varnish-Cache = "HIT";
set resp.http.X-Varnish-Hits = obj.hits;
}
else {
set resp.http.X-Varnish-Cache = "MISS";
}
# Set a header to track the webhead.
set resp.http.X-Varnish-IP = server.ip;
}
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.http.X-Varnish-Cookie) {
set bereq.http.cookie = req.http.X-Varnish-Cookie;
unset bereq.http.X-Varnish-Cookie;
}
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}
sub vcl_error {
set obj.http.Content-Type = "text/html; charset=utf-8";
if (obj.status == 401) {
# Prompt for password.
set obj.http.WWW-Authenticate = "Basic realm=Secured";
}
synthetic {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>"} + obj.status + " " + obj.response + {"</title>
</head>
<body>
<div id="page">
<h1>Page Could Not Be Loaded</h1>
<p>We're very sorry, but the page could not be loaded properly. This should be fixed very soon, and we apologize for any inconvenience.</p>
<hr />
<h4>Debug Info:</h4>
<pre>Status: "} + obj.status + {"
Response: "} + obj.response + {"
XID: "} + req.xid + {"</pre>
</div>
</body>
</html>
"};
return (deliver);
}
http://www.isvarnishworking.com/ says, my site is properly served by varnish, but i know it is not, because HIT counter does not show up and logs don't say so too.
here is the response I am getting
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 10
cache-control: max-age = 3600
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Thu, 18 Aug 2016 23:29:25 GMT
Server: Apache/2.4.23 (Amazon) OpenSSL/1.0.1k-fips PHP/5.6.24
Vary: Accept-Encoding,User-Agent
Via: 1.1 varnish
X-Cacheable: YES
X-Content-Type-Options: nosniff
X-Frame-Options: GOFORIT
X-Varnish: 1595154742
X-Varnish-Cache: MISS
X-Varnish-IP: 172.31.41.246
X-XSS-Protection: 1; mode=block
Connection: keep-alive
The age value in response header is greater than zero that implies you are getting cached response. Otherwise it would be zero.
As i think you have defined the Age header in the VCL configuration, that's why we are getting it in the response header.
Please correct me if i am wrong

Can't see X-Varnish in the respond header output

I have installed varnish for my webserver magento in local. Seems to varnish does not working! i can't see X-Varnish in the respond header output,
$ curl -I 1900.loc
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Set-Cookie: frontend=hkmilsm5e32o51vf0g9kb55f12; expires=Sat, 23-Aug-2014 02:56:15 GMT; path=/; domain=1900.loc; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
Date: Sat, 23 Aug 2014 01:56:16 GMT
Age: 0
Connection: keep-alive
I think file default.vcl corrupted.
default.vcl
C{
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <pthread.h>
static pthread_mutex_t lrand_mutex = PTHREAD_MUTEX_INITIALIZER;
void generate_uuid(char* buf) {
pthread_mutex_lock(&lrand_mutex);
long a = lrand48();
long b = lrand48();
long c = lrand48();
long d = lrand48();
pthread_mutex_unlock(&lrand_mutex);
sprintf(buf, "frontend=%08lx-%04lx-%04lx-%04lx-%04lx%08lx",
a,
b & 0xffff,
(b & ((long)0x0fff0000) >> 16) | 0x4000,
(c & 0x0fff) | 0x8000,
(c & (long)0xffff0000) >> 16,
d
);
return;
}
}C
import std;
backend default {
.host = "192.168.1.100";
.port = "8000";
.connect_timeout = 500s;
.first_byte_timeout = 500s;
.between_bytes_timeout = 500s;
}
backend admin {
.host = "192.168.1.100";
.port = "8000";
.connect_timeout = 6000s;
.first_byte_timeout = 21600s;
.between_bytes_timeout = 21600s;
}
acl crawler_acl {
"127.0.0.1";
}
acl debug_acl {
}
sub remove_cache_headers {
unset beresp.http.Cache-Control;
unset beresp.http.Expires;
unset beresp.http.Pragma;
unset beresp.http.Cache;
unset beresp.http.Age;
}
sub remove_double_slashes {
set req.url = regsub(req.url, "(.*)//+(.*)", "\1/\2");
}
sub generate_session {
if (req.url ~ ".*[&?]SID=([^&]+).*") {
set req.http.X-Varnish-Faked-Session = regsub(
req.url, ".*[&?]SID=([^&]+).*", "frontend=\1");
} else {
C{
char uuid_buf [50];
generate_uuid(uuid_buf);
VRT_SetHdr(sp, HDR_REQ,
"\030X-Varnish-Faked-Session:",
uuid_buf,
vrt_magic_string_end
);
}C
}
if (req.http.Cookie) {
std.collect(req.http.Cookie);
set req.http.Cookie = req.http.X-Varnish-Faked-Session +
"; " + req.http.Cookie;
} else {
set req.http.Cookie = req.http.X-Varnish-Faked-Session;
}
}
sub generate_session_expires {
C{
time_t now = time(NULL);
struct tm now_tm = *gmtime(&now);
now_tm.tm_sec += 86400;
mktime(&now_tm);
char date_buf [50];
strftime(date_buf, sizeof(date_buf)-1, "%a, %d-%b-%Y %H:%M:%S %Z", &now_tm);
VRT_SetHdr(sp, HDR_RESP,
"\031X-Varnish-Cookie-Expires:",
date_buf,
vrt_magic_string_end
);
}C
}
sub vcl_recv {
call normalize_req_url;
#return(pass);
#if (false || client.ip ~ debug_acl) {
#error 403 "External ESI requests are not allowed";
#}
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (!true || req.http.Authorization ||
req.request !~ "^(GET|HEAD)$" ||
req.http.Cookie ~ "varnish_bypass=1") {
return (pipe);
}
call remove_double_slashes;
if (req.http.Accept-Encoding) {
if (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} else if (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding;
}
}
if (req.url ~ "^(/shop/)(?:(?:index|litespeed)\.php/)?") {
set req.http.X-Turpentine-Secret-Handshake = "1";
if (req.url ~ "^(/shop/)(?:(?:index|litespeed)\.php/)?livesuperfoods_admin") {
set req.backend = admin;
return (pipe);
}
if (req.http.Cookie ~ "\bcurrency=") {
set req.http.X-Varnish-Currency = regsub(
req.http.Cookie, ".*\bcurrency=([^;]*).*", "\1");
}
if (req.http.Cookie ~ "\bstore=") {
set req.http.X-Varnish-Store = regsub(
req.http.Cookie, ".*\bstore=([^;]*).*", "\1");
}
if (req.url ~ "/turpentine/esi/getBlock/") {
set req.http.X-Varnish-Esi-Method = regsub(
req.url, ".*/method/(\w+)/.*", "\1");
set req.http.X-Varnish-Esi-Access = regsub(
req.url, ".*/access/(\w+)/.*", "\1");
if (req.http.X-Varnish-Esi-Method == "esi" && req.esi_level == 0 &&
!(false || client.ip ~ debug_acl)) {
error 403 "External ESI requests are not allowed";
}
}
if (req.http.Cookie !~ "frontend=") {
if (client.ip ~ crawler_acl ||
req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
set req.http.Cookie = "frontend=crawler-session";
} else {
call generate_session;
}
}
if (true &&
req.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
unset req.http.Cookie;
unset req.http.X-Varnish-Faked-Session;
return (lookup);
}
if (req.url ~ "^(/shop/)(?:(?:index|litespeed)\.php/)?(?:livesuperfoods_admin|api|cron\.php)" ||
req.url ~ "\?.*__from_store=") {
return (pipe);
}
if (true &&
req.url ~ "(?:[?&](?:__SID|XDEBUG_PROFILE)(?=[&=]|$))") {
return (pass);
}
}
}
sub vcl_pipe {
set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
set bereq.http.X-Forwarded-For = regsub(bereq.http.X-Forwarded-For, "$", ", ");
set bereq.http.X-Forwarded-For = regsub(bereq.http.X-Forwarded-For, "$", client.ip);
}
sub vcl_hash {
hash_data(req.url);
if (req.http.Host) {
hash_data(req.http.Host);
} else {
hash_data(server.ip);
}
hash_data(req.http.Ssl-Offloaded);
if (req.http.X-Normalized-User-Agent) {
hash_data(req.http.X-Normalized-User-Agent);
}
if (req.http.Accept-Encoding) {
hash_data(req.http.Accept-Encoding);
}
if (req.http.X-Varnish-Store || req.http.X-Varnish-Currency) {
hash_data("s=" + req.http.X-Varnish-Store + "&c=" + req.http.X-Varnish-Currency);
}
if (req.http.X-Varnish-Esi-Access == "private" &&
req.http.Cookie ~ "frontend=") {
hash_data(regsub(req.http.Cookie, "^.*?frontend=([^;]*);*.*$", "\1"));
}
return (hash);
}
sub vcl_hit {
}
sub vcl_fetch {
set req.grace = 15s;
if (req.url ~ "^(/shop/)(?:(?:index|litespeed)\.php/)?") {
unset beresp.http.Vary;
set beresp.do_gzip = true;
if (beresp.status != 200 && beresp.status != 404) {
set beresp.ttl = 15s;
return (hit_for_pass);
} else {
if (beresp.http.Set-Cookie) {
set beresp.http.X-Varnish-Set-Cookie = beresp.http.Set-Cookie;
unset beresp.http.Set-Cookie;
}
call remove_cache_headers;
if (beresp.http.X-Turpentine-Esi == "1") {
set beresp.do_esi = true;
}
if (beresp.http.X-Turpentine-Cache == "0") {
set beresp.ttl = 15s;
return (hit_for_pass);
} else {
if (true &&
bereq.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
set beresp.ttl = 28800s;
set beresp.http.Cache-Control = "max-age=28800";
} elseif (req.http.X-Varnish-Esi-Method) {
if (req.http.X-Varnish-Esi-Access == "private" &&
req.http.Cookie ~ "frontend=") {
set beresp.http.X-Varnish-Session = regsub(req.http.Cookie,
"^.*?frontend=([^;]*);*.*$", "\1");
}
if (req.http.X-Varnish-Esi-Method == "ajax" &&
req.http.X-Varnish-Esi-Access == "public") {
set beresp.http.Cache-Control = "max-age=" + regsub(
req.url, ".*/ttl/(\d+)/.*", "\1");
}
set beresp.ttl = std.duration(
regsub(
req.url, ".*/ttl/(\d+)/.*", "\16s"),
300s);
if (beresp.ttl == 15s) {
set beresp.ttl = 15s;
return (hit_for_pass);
}
} else {
set beresp.ttl = 3600s;
}
}
}
return (deliver);
}
}
sub vcl_deliver {
if (req.http.X-Varnish-Faked-Session) {
call generate_session_expires;
set resp.http.Set-Cookie = req.http.X-Varnish-Faked-Session +
"; expires=" + resp.http.X-Varnish-Cookie-Expires + "; path=/";
if (req.http.Host) {
set resp.http.Set-Cookie = resp.http.Set-Cookie +
"; domain=" + regsub(req.http.Host, ":\d+$", "");
}
set resp.http.Set-Cookie = resp.http.Set-Cookie + "; httponly";
unset resp.http.X-Varnish-Cookie-Expires;
}
if (false || client.ip ~ debug_acl) {
set resp.http.X-Varnish-Hits = obj.hits;
set resp.http.X-Varnish-Esi-Method = req.http.X-Varnish-Esi-Method;
set resp.http.X-Varnish-Esi-Access = req.http.X-Varnish-Esi-Access;
set resp.http.X-Varnish-Currency = req.http.X-Varnish-Currency;
set resp.http.X-Varnish-Store = req.http.X-Varnish-Store;
} else {
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Turpentine-Cache;
unset resp.http.X-Turpentine-Esi;
unset resp.http.X-Turpentine-Flush-Events;
unset resp.http.X-Turpentine-Block;
unset resp.http.X-Varnish-Session;
unset resp.http.X-Varnish-Set-Cookie;
}
}
I've take a lot days to find a fix !
I get this error trying to compile this VCL:
# varnishd -C -f default.vcl
Message from VCC-compiler:
Undefined sub normalize_req_url, first reference:
('input' Line 98 Pos 6)
call normalize_req_url;
-----#################-
Running VCC-compiler failed, exit 1
VCL compilation failed
Ignoring that problem, there is another block in vcl_deliver that performs an unset on the header in question:
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Turpentine-Cache;
unset resp.http.X-Turpentine-Esi;
unset resp.http.X-Turpentine-Flush-Events;
unset resp.http.X-Turpentine-Block;
unset resp.http.X-Varnish-Session;
unset resp.http.X-Varnish-Set-Cookie;

Varnish Probe works again After “Service Scan” by Domain Dossier

My Varnish Cache works fine except there is annoying issue that every few hours, the backend probe will report sick for 3 of my 4 backends. So site can't open, report 503 obviously.
The strange thing is everytime this happens, I went to Domain Dossier (http://centralops.net/co/DomainDossier.aspx) and did a "service scan". The scan will show 503 error, but do have all the headers shown, just X-Varnish has one number, obviously cache is not working....
BUT right after I've done with the "service scan", I went to do a "varnishadm backend.list", I will 100% of the time see the number counts start to go from "sick 0/10" to finally "healthy 10/10". Like I said usually I have 3 domains in sick, so I do "service scan" again for another 2 sick domains, everyone of them gets successful probe ping and back to work....
Any one had the same issues?
Apache: Listen domain.ip1:8080 Listen domain.ip2:8080 Listen domain.ip4:8080 (domain2 use same ip as domain3)
Here is default.vcl:
backend domain1 {
.host = "domain.ip1";
.port = "8080";
.connect_timeout = 600s;
.saintmode_threshold = 0;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.probe = {
.url = "/favicon.ico";
.interval = 10s;
.timeout = 3s;
.window = 10;
.threshold = 3;
# .initial = 3;
.expected_response = 301;
}
}
backend domain2 {
.host = "domain.ip2";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.probe = {
.url = "/favicon.ico";
.interval = 10s;
.timeout = 3s;
.window = 10;
.threshold = 3;
# .initial = 3;
.expected_response = 200;
}
}
backend domain3 {
.host = "domain.ip3";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.probe = {
.url = "/favicon.ico";
.interval = 10s;
.timeout = 3s;
.window = 10;
.threshold = 3;
# .initial = 3;
.expected_response = 200;
}
}
backend domain4 {
.host = "domain.ip4";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.probe = {
.url = "/favicon.ico";
.interval = 10s;
.timeout = 3s;
.window = 10;
.threshold = 3;
# .initial = 3;
.expected_response = 200;
}
}
sub vcl_recv {
if (req.http.host ~ "^www.domain1.com") {
set req.http.host = "www.domain1.com";
set req.backend = domain1;
return (lookup);
}
if (req.http.host ~ "^www.domain2.com") {
set req.http.host = "www.domain2.com";
set req.backend = domain2;
return (lookup);
}
if (req.http.host ~ "^www.domain3.com") {
set req.http.host = "www.domain3.com";
set req.backend = domain3;
return (lookup);
}
if (req.http.host ~ "^www.domain4.com") {
set req.http.host = "www.domain4.com";
set req.backend = domain4;
return (lookup);
}
}
When backend is healthy, probe 10/10, the header is like:
HTTP/1.1 301 Moved Permanently
Server: Apache
Location: http://www.domain1.com/
Cache-Control: max-age=900
Expires: Tue, 24 Jun 2014 07:30:25 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 304
Accept-Ranges: bytes
Date: Tue, 24 Jun 2014 07:15:25 GMT
X-Varnish: 1997501007
Age: 0
Via: 1.1 varnish
Connection: close
OR
HTTP/1.1 301 Moved Permanently
Server: Apache
Location: http://www.domain1.com/
Cache-Control: max-age=900
Expires: Tue, 24 Jun 2014 07:30:25 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 304
Accept-Ranges: bytes
Date: Tue, 24 Jun 2014 07:18:53 GMT
X-Varnish: 1997501047 1997501007
Age: 208
Via: 1.1 varnish
Connection: close
When backend is sick, I use Domain Dossier to get a header, it will show like this:
HTTP/1.1 503
Server: Apache
Location: http://www.domain1.com/
Cache-Control: max-age=900
Expires: Tue, 24 Jun 2014 07:30:25 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 304
Accept-Ranges: bytes
Date: Tue, 24 Jun 2014 07:15:25 GMT
X-Varnish: 1997501007
Age: 0
Via: 1.1 varnish
Connection: close
/etc/sysconfig/varnish file:
DAEMON_OPTS="-a :80\
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,512m \
-p ping_interval=10 \
-p thread_pools=2 \
-p thread_pool_max=8000 \
-p thread_pool_min=100 \
-p thread_pool_add_delay=2 \
-p sess_timeout=60 \
-p cli_timeout=120 \
"

Varnish frontend server, Litespeed backend server

I wonder if someone could advise what I'm doing wrong.
I have Varnish set up on a frontend server, and this is working. I can put the IP address for several public websites into this (including CNN) into the configuration, and they are cached by Varnish.
When I put one of my site IP address' into Varnish, I get a 404 error back from the Litespeed server - this shows it's connecting to the server, but obviously not picking up the correct site to show Varnish. I have several domains on the Litespeed server, each with it's own IP address. The same happens with the direct URL.
This is I guess a problem with the Litespeed configuration - but any assistance would be appreciated.
For reference, I'm using the VCL configuration file for PageCache, which is a Magento module for Varnish.
Justin
# default backend definition. Set this to point to your content server.
backend default {
.host = "xxx.xxx.xxx.xxx";
.port = "80";
}
# admin backend with longer timeout.
backend admin {
.host = "xxx.xxx.xxx.xxx";
.port = "80";
.first_byte_timeout = 18000s;
.between_bytes_timeout = 18000s;
}
# add your Magento server IP to allow purges from the backend
acl purge {
"localhost";
"127.0.0.1";
}
sub vcl_recv {
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE" &&
req.request != "PURGE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# purge request
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
ban("obj.http.X-Purge-Host ~ " + req.http.X-Purge-Host + " && obj.http.X-Purge-URL ~ " + req.http.X-Purge-Regex + " && obj.http.Content-Type ~ " + req.http.X-Purge-Content-Type);
error 200 "Purged.";
}
# switch to admin backend configuration
if (req.http.cookie ~ "adminhtml=") {
set req.backend = admin;
}
# we only deal with GET and HEAD by default
if (req.request != "GET" && req.request != "HEAD") {
return (pass);
}
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://[^/]+", "");
# static files are always cacheable. remove SSL flag and cookie
if (req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$") {
unset req.http.Https;
unset req.http.Cookie;
}
# not cacheable by default
if (req.http.Authorization || req.http.Https) {
return (pass);
}
# do not cache any page from
# - index files
# - ...
if (req.url ~ "^/(index)") {
return (pass);
}
# as soon as we have a NO_CACHE cookie pass request
if (req.http.cookie ~ "NO_CACHE=") {
return (pass);
}
# normalize Aceept-Encoding header
# http://varnish.projects.linpro.no/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
# remove Google gclid parameters
set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"
return (lookup);
}
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
if (!(req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$")) {
call design_exception;
}
return (hash);
}
sub vcl_fetch {
if (beresp.status == 500) {
set beresp.saintmode = 10s;
return (restart);
}
set beresp.grace = 5m;
# add ban-lurker tags to object
set beresp.http.X-Purge-URL = req.url;
set beresp.http.X-Purge-Host = req.http.host;
if (beresp.status == 200 || beresp.status == 301 || beresp.status == 404) {
if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type ~ "text/xml") {
if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
set beresp.ttl = 0s;
return (hit_for_pass);
}
# marker for vcl_deliver to reset Age:
set beresp.http.magicmarker = "1";
# Don't cache cookies
unset beresp.http.set-cookie;
} else {
# set default TTL value for static content
set beresp.ttl = 4h;
}
return (deliver);
}
return (hit_for_pass);
}
sub vcl_deliver {
# debug info
if (resp.http.X-Cache-Debug) {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = "MISS";
}
set resp.http.X-Cache-Expires = resp.http.Expires;
} else {
# remove Varnish/proxy header
remove resp.http.X-Varnish;
remove resp.http.Via;
remove resp.http.Age;
remove resp.http.X-Purge-URL;
remove resp.http.X-Purge-Host;
}
if (resp.http.magicmarker) {
# Remove the magic marker
unset resp.http.magicmarker;
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, post-check=0, pre-check=0";
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "Mon, 31 Mar 2008 10:00:00 GMT";
set resp.http.Age = "0";
}
}
sub design_exception {
}
And...
START=yes
NFILES=131072
MEMLOCK=82000
INSTANCE=$(uname -n)
## Alternative 2, Configuration with VCL
#
# Listen on port 6080, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request. Use a 1GB
# fixed-size cache file.
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
To what port is your Varnish server bound? And what port do you use for Litespeed?
It would be useful if you could post your config files here to give you further advices.
This is what I needed! Configure multiple sites with Varnish
I needed to put the virtual host names into the configuration file. Followed this guide and had it work in moments!

Resources