How can I customize the message shown to a user who reaches a Guru Meditation error when making a request thru my Varnish server? - varnish

I am trying to customize what appears on the screen for a user when he reaches a Guru Meditation error screen, upon making an erring request to my backend, which has a Varnish reverse proxy in front. I have tried placing log.info(client.ip) in the different subroutines of default.vcl, only to run into compilation errors when trying to start the varnish service. I am using a Linux virtual machine.

You certainly can. Have a look at the following tutorial I created: https://www.varnish-software.com/developers/tutorials/vcl-synthetic-output-template-file/.
Modifying the synthetic output templates
Here's the VCL code you need to extend the regular vcl_synth subroutine in case you call return(synth()) from your VCL code, as well as the code you need to extend vcl_backend_response in case of a backend fetch error:
vcl 4.1;
import std;
sub vcl_synth {
set resp.http.Content-Type = "text/html; charset=utf-8";
set resp.http.Retry-After = "5";
set resp.body = regsuball(std.fileread("/etc/varnish/synth.html"),"<<REASON>>",resp.reason);
return (deliver);
}
sub vcl_backend_error {
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
set beresp.body = regsuball(std.fileread("/etc/varnish/synth.html"),"<<REASON>>",beresp.reason);
return (deliver);
}
As explained in the tutorial, you can store the HTML code you want to display in an HTML and load this into your VCL output.
The trick is to put a <<REASON>> placeholder in your HTML where the actual error message gets parsed into.
Adding custom logging to your VCL
If you want to add custom logging that gets sent to VSL, you can use the std.log() function that is part of vmod_std.
Here's some example VCL code that uses this function:
vcl 4.1;
import std;
sub vcl_recv {
std.log("Client IP: " + client.ip);
}
The log will be displayed through a VCL_Log tag in your VSL output.
If you want to filter out VCL_Log tags, you can use the following command:
varnishlog -g request -i VCL_Log
This is the output you may receive:
* << Request >> 32770
- VCL_Log Client IP: 127.0.0.1
** << BeReq >> 32771
If you're not filtering the VCL_Log tag, you'll see it appear in your VSL output if your run varnishlog -g request.
Tip: if you want to see the full log transaction but only for a specific URL, just run varnishlog -g request -q "ReqUrl eq '/'". This will only display the logs for the homepage.
Update: displaying the client IP in the synthetic output
The VCL code below injects the X-Forwarded-For header into the output by concatenating to the reason phrase:
vcl 4.1;
import std;
sub vcl_synth {
set resp.http.Content-Type = "text/html; charset=utf-8";
set resp.http.Retry-After = "5";
set resp.body = regsuball(std.fileread("/etc/varnish/synth.html"),"<<REASON>>",resp.reason + " (" + req.http.X-Forwarded-For + ")");
return (deliver);
}
sub vcl_backend_error {
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
set beresp.body = regsuball(std.fileread("/etc/varnish/synth.html"),"<<REASON>>",beresp.reason + " (" + bereq.http.X-Forwarded-For + ")");
return (deliver);
}
It's also possible to provide a second placeholder in the template and perform an extra regsuball() call. But for the sake of simplicity, the X-Forwarded-For header is just attached to the reason string.

Related

Ban or Purge the varnish cache

Q1:i am caching the content for mobile and desktop. I want to purge or ban cache only for mobile or only for desktop. So how to purge and ban cache for mobile and desktop.
Q2: i want to bypass the cache for Desktop user agent.I want to cache only mobile user agent firstly.Please help.This is my VCL code for cache mobile and desktop user agent.
Regular purging in Varnish is done based on the URL and removes all variations. If you only want to remove specific objects for one of the cache variations (mobile vs desktop), you'll need to use banning.
Here's an official banning tutorial: https://www.varnish-software.com/developers/tutorials/ban/
The VCL code
If we use your VCL code as the basis, here's the complete VCL including the banning logic:
vcl 4.1;
backend default {
.port = "8000";
}
acl purge {
"localhost";
"192.168.55.0"/24;
}
include "devicedetect.vcl";
sub vcl_recv {
call devicedetect;
if(req.http.X-UA-Device ~ "^(mobile|tablet)\-.+$") {
set req.http.X-UA-Device = "mobile";
} else {
set req.http.X-UA-Device = "desktop";
}
}
sub vcl_recv {
if (req.method == "BAN") {
if (!client.ip ~ purge) {
return (synth(405));
}
if (!req.http.x-invalidate-pattern) {
if(!req.http.x-invalidate-ua-device) {
return (purge);
}
ban("obj.http.x-url == " + req.url
+ " && obj.http.x-host == " + req.http.host
+ " && obj.http.x-ua-device == " + req.http.x-invalidate-ua-device);
return (synth(200,"Ban added"));
}
if(!req.http.x-invalidate-ua-device) {
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host);
return (synth(200,"Ban added"));
}
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host
+ " && obj.http.x-ua-device == " + req.http.x-invalidate-ua-device);
return (synth(200,"Ban added"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
set beresp.http.x-ua-device = bereq.http.X-UA-Device;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
unset resp.http.x-ua-device;
}
sub vcl_hash {
hash_data(req.http.X-UA-Device);
}
How to run
Here are a couple of examples of how to execute the bans.
1. Invalidate a page for both mobile and desktop
The following command will remove the /my-page page from the cache for the domain.ext domain. This will remove both the mobile and the desktop version:
curl -XBAN http://domain.ext/my-page
2. Invalidate a page for the mobile version of the website
The following command will remove the /my-page page from the cache for the domain.ext domain, but only for the mobile version:
curl -XBAN -H "x-invalidate-ua-device: mobile" http://domain.ext/my-page
3. Invalidate a page for the mobile version of the website
The following command will remove the /my-page page from the cache for the domain.ext domain, but only for the desktop version:
curl -XBAN -H "x-invalidate-ua-device: desktop" http://domain.ext/my-page
4. Invalidate multiple pages for both the mobile and the desktop version
The following command will remove all pages from the cache that start with /my-* for the domain.ext domain. Both for the mobile and desktop version of the website
curl -XBAN -H "x-invalidate-pattern: /my-" http://domain.ext/my-page
5. Invalidate multiple pages for the mobile website
The following command will remove all pages from the cache that start with /my-* for the domain.ext domain, but only for the mobile version of the website:
curl -XBAN -H "x-invalidate-pattern: /my-" -H "x-invalidate-ua-device: mobile" http://domain.ext/my-page
6. Invalidate multiple pages for the desktop website
The following command will remove all pages from the cache that start with /my-* for the domain.ext domain, but only for the desktop version of the website:
curl -XBAN -H "x-invalidate-pattern: /my-" -H "x-invalidate-ua-device: desktop" http://domain.ext/my-page
Further customizations
The VCL code assumes that the 192.168.55.0/24 IP range will be used to invalidate the cache remotely. Please make sure the right IP addresses, hostnames and CIDRs are part of the purge ACL.
The ban executions were done using the domain.ext domain name. Please use the right hostname to invalidate your cache.
If the hostname you're using to invalidate (e.g. "localhost") is not that hostname with which the objects are stored in the cache, please assign an explicit Host header to your invalidation calls.
Here's an example where the ban call is done locally, but the Host header to match is domain.ext:
curl -XBAN -H "Host: domain.ext" -H "x-invalidate-pattern: /my-" -H "x-invalidate-ua-device: desktop" http://localhost/my-page
Bypassing the cache for desktop users
To answer your second question, here's how you bypass the cache for the desktop website:
sub vcl_recv {
if(req.http.X-UA-Device == "desktop") {
return(pass);
}
}
This small snippet of VCL code can be added to your existing one. In one of the earlier vcl_recv definitions the X-UA-Device header is set, which can be reused here.

Varnish with MPEG-TS Traffic

We are trying use Varnish as proxy/cache for our media server. Our streams are MPEG-TS (h264/h265) over http. There is 1000 live streams on this media server and each stream getting multiple connection. We tried to configure Varnish shown as below but we have these problems.
Streams get close after a short period of time
Sometimes cant able to connect to streams, stuck at connecting...
Got these errors on varnislog;
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Resource temporarily unavailable
- FetchError eof socket fail
- FetchError Resource temporarily unavailable
- FetchError eof socket fail
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Could not get storage
- FetchError Resource temporarily unavailable
- FetchError eof socket fail
- FetchError Resource temporarily unavailable
- FetchError eof socket fail
- FetchError Resource temporarily unavailable
- FetchError eof socket fail
- FetchError Could not get storage
- FetchError Could not get storage
My config;
vcl 4.0;
import directors;
backend s6855 {
.host = "127.0.0.1";
.port = "6855";
.first_byte_timeout = 10s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 5s; # How long to wait for a backend connection?
.between_bytes_timeout = 30s; # How long to wait between bytes received from our backend?
}
backend s6866 {
.host = "127.0.0.1";
.port = "6866";
.first_byte_timeout = 10s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 5s; # How long to wait for a backend connection?
.between_bytes_timeout = 30s; # How long to wait between bytes received from our backend?
}
backend s6877 {
.host = "127.0.0.1";
.port = "6877";
.first_byte_timeout = 10s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 5s; # How long to wait for a backend connection?
.between_bytes_timeout = 30s; # How long to wait between bytes received from our backend?
}
backend s6888 {
.host = "127.0.0.1";
.port = "6888";
.first_byte_timeout = 10s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 5s; # How long to wait for a backend connection?
.between_bytes_timeout = 30s; # How long to wait between bytes received from our backend?
}
backend s6899 {
.host = "127.0.0.1";
.port = "6899";
.first_byte_timeout = 10s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 5s; # How long to wait for a backend connection?
.between_bytes_timeout = 30s; # How long to wait between bytes received from our backend?
}
sub vcl_init {
new fb = directors.round_robin();
fb.add_backend(s6855);
fb.add_backend(s6866);
fb.add_backend(s6877);
fb.add_backend(s6888);
fb.add_backend(s6899);
}
sub vcl_recv {
set req.grace = 120s;
set req.backend_hint = fb.backend();
if (req.url ~ "(\.ts)" ) {
unset req.http.Range;
}
if (req.http.cookie) {
unset req.http.cookie;
}
if (req.method != "GET" && req.method != "HEAD") {
return (pipe);
}
if (req.method == "GET" && req.url ~ "(\.ts)" ) {
unset req.http.Accept-Encoding;
return(hash);
}
return(hash);
}
sub vcl_hash {
hash_data(req.url);
return(lookup);
}
sub vcl_backend_response {
set beresp.grace = 2m;
set beresp.ttl = 120s;
set beresp.do_gunzip = false;
set beresp.do_gzip = false;
if (bereq.url ~ "(\.ts)") {
set beresp.ttl = 60s;
set beresp.http.X-Cacheable = "YES";
}
else {
set beresp.ttl = 10m;
set beresp.http.X-Cacheable = "NO";
}
if ( beresp.status == 404 ) {
set beresp.ttl = 5m;
}
return(deliver);
}
sub vcl_hit {
if (obj.ttl == 0s) {
return(pass);
}
return(deliver);
}
sub vcl_miss {
}
sub vcl_deliver {
set resp.http.X-Served-By = "For Test";
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = "MISS";
}
if(resp.http.magicmarker) {
unset resp.http.magicmarker;
set resp.http.Age="0";
}
unset resp.http.Via;
unset resp.http.X-Varnish;
}
Varnish Usage
Since pretty new to Varnish not sure how to debug the problem, your help will be appreciated.
Thanks
The problem you're experiencing is not just a lack of object storage, but the fact that your biggest HTTP response is larger than the total size of the object storage.
This means Varnish cannot LRU evict the required space to fit the object in cache.
Could not get storage is an error that is typically returned when this happens.
Check the sizes
It is important to figure out how big your cache is, and what the size of the object is that fails on you.
Your varnishd runtime settings will tell you how big your object storage is. The -s malloc,<size> contains this value.
You can also use varnishstat to check the size & usage of your memory cache and the transient storage:
varnishstat -f SMA.*.g* -f MAIN.n_lru_nuked
The MAIN.n_lru_nuked counter that is also included in this command, will indicate how many objects that Varnish is forcefully removing from the cache to clear up space for new objects.
Fixing the issue
The easiest way to fix the issue, is to assign more memory to Varnish via -s malloc,<size>. Don't forget to restart Varnish after you have changed these settings.
After that, the following command will help you figure out if there's enough storage, and if Varnish still needs to forcefully remove objects from cache the free up space:
varnishstat -f SMA.*.g* -f MAIN.n_lru_nuked
A more sustainable plan
Another plan is to rely on the Massive Storage Engine (MSE). This is a storage engine that is part of Varnish Enterprise.
It combines memory and disk storage, and is optimized to handle large volumes of data. It avoids fragmentation, and is architected to not suffer from the typical latency of disk access.
There are official machine images for AWS, Azure & Google Cloud that allow you to experiment with this storage engine, without having to buy a license upfront.
A killer MSE feature is the memory governor. This is a mechanism that dynamically sizes the memory storage of your caches based on the needs of requests & responses.
If you run short of memory, and there isn't a lot of memory needed for thread handling, the memory governor will automatically assign more memory to the storage engine.
If you use the persistence layer of MSE, you can host terrabytes of data on a single machine, without running into these issues.
At Varnish Software, the company that builds Varnish Enterprise, we see MSE as the primary feature that OTT video streaming companies use to accelerate video delivery.
What if my assessment is completely wrong
Although the Could not get storage error usually appears when Varnish is trying to store huge objects in cache when the size of the cache is too small, I could also be wrong.
In that case, I would advise you to run varnishlog and see the full trace of what's going on in that specific transaction:
varnishlog -g request -q "ReqUrl eq '/my-url'"
This examples gets all the details of requests for /my-url. Please change this to the URL you're trying to monitor.
The output will usually give you a better understanding of how Varnish is behaving. This can help us figure out how to fix the issue, if my initial assessment was wrong.

Docusign Listener Argument Null (dev Sandbox)

I have created a webmethod in a C# webservice that listens for Docusign to call when an Envelope status changes:
[WebMethod]
public void DocuSignConnectUpdate(DocuSignEnvelopeInformation DocuSignEnvelopeInformation)
{
//Check if null
if (DocuSignEnvelopeInformation == null)
{
File.WriteAllText("C:\\websites\\DataAPI\\datalog.txt", "Data: " + "Data is null");
}
else
{
string envelopeId = "";
try
{
//Write a line in a file
File.WriteAllText("C:\\websites\\DataAPI\\datalog.txt", "Data: " + DocuSignEnvelopeInformation.ToString());
//Get some data out
envelopeId = DocuSignEnvelopeInformation.EnvelopeStatus.EnvelopeID;
//Write Data to a file
File.WriteAllText("C:\\websites\\DataAPI\\innerdatalog.txt", "Data: " + DocuSignEnvelopeInformation.ToString());
}
catch (Exception ex)
{
// could not serialize
File.WriteAllText("C:\\websites\\DataAPI\\errorlog.txt", "Exception: " + ex.Message);
throw new SoapException(ex.Message, SoapException.ClientFaultCode);
}
}
The issue I am having is that DocuSignEnvelopeInformation argument is not being set when called, so the code keeps terminating at the if==null statement. When I run the envelope data to the API using SoapUI everything works correctly. Any ideas what I'm missing would be appreciated.
EDIT: I wanted to Add the Interface here too since I forgot it originally
[ServiceContract(ConfigurationName = "IOperations", Namespace = "https://www.docusign.net/API/3.0")]
public interface IOperations
{
[OperationContract(Action = "DocuSignConnectListener/Operations/DocuSignConnectUpdate")]
[XmlSerializerFormat]
string DocuSignConnectUpdate(DocuSignEnvelopeInformation DocuSignEnvelopeInformation);
}
When a DocuSign webhook is set to use SOAP mode, the notification is sent as a SOAP request to your server (your listener).
If SOAP mode is off, then the notification is sent as a regular POST request with an XML body.
In your question, you say that
When I run the envelope data to the API using SoapUI everything works correctly
So it sounds like everything is worked as designed.
Ok I finally figured this out, turns out it just wasn't pretty enough so I added a decoration specifically:
[SoapDocumentMethod("http://tempuri.org/DocuSignConnectUpdate",
RequestNamespace = "http://tempuri.org",
ResponseNamespace = "http://tempuri.org",
Use = System.Web.Services.Description.SoapBindingUse.Literal,
ParameterStyle = System.Web.Services.Protocols.SoapParameterStyle.Wrapped)]
on the method and now everything works like its supposed too. Now that I look at it, it makes a lot more sense.

Varnish Cache v4: Incorrect Backend Health Check Response

I've setup Varnish Cache (4) in front of my CMS to help cache requests. In the event that my CMS goes down, I'd like to deliver cached items for a set grace period. I've followed many examples provided online but am running into an issue where Varnish doesn't recognize that my backend is down. When I manually shutdown the CMS the std.health(req.backend_hint)) continues to return true and attempts to retrieve an item from the backend, which then returns a 503 response.
Question: Am I incorrectly assuming std.health(req.backend_hint)) will recognize that my CMS is down?
Here is the VCL script I've been using to test:
sub vcl_recv {
# Initial State
set req.http.grace = "none";
set req.http.x-host = req.http.host;
set req.http.x-url = req.url;
return(hash);
}
sub vcl_backend_response {
set beresp.ttl = 10s;
set beresp.grace = 1h;
}
sub vcl_deliver {
# Copy grace to resp so we can tell from client
set resp.http.grace = req.http.grace;
# Add debugging headers to cache requests
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
}
else {
set resp.http.X-Cache = "MISS";
}
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# normal hit
return (deliver);
}
# We have no fresh content, lets look at the cache
elsif (std.healthy(req.backend_hint)) {
# Backend is healthy. Limit age to 10s.
if (obj.ttl + 10s > 0s) {
set req.http.grace = "normal(limited)";
return (deliver);
} else {
# No candidate for grace. Fetch a fresh object.
return(fetch);
}
} else {
# backend is sick - use full grace
if (obj.ttl + obj.grace > 0s) {
set req.http.grace = "full";
return (deliver);
} else {
# no graced object.
return (fetch);
}
}
}
Again, when I shutdown the CMS the std.healthy(req.backend_hint)) still reports the backend as healthy and never jumps to the final else statement.
Thanks for taking a look.
To properly use std.healthy you of course need to configure backend probes. So at the top of your VCL file you would first configure a probe:
probe site_probe {
.request =
"HEAD / HTTP/1.1"
"Host: example.com"
"Connection: close";
.interval = 5s; # check the health of each backend every 5 seconds
.timeout = 3s; # timing out after 1 second by default.
.window = 5; # If 3 out of the last 5 polls succeeded the backend is considered healthy, otherwise it will be marked as sick
.threshold = 3;
}
Make sure to replace example.com with your main website domain. It is important to put (or omit) the www. prefix so that the probe will not get redirect and marked as failed.
And of course your backend definition should be configured to use the defined probe:
backend default {
.host = "127.0.0.1";
.port = "8080";
.probe = site_probe;
}

Amazon CloudFront invalidation API

The images used in our application are rendered from Amazon CloudFront.
When an existing image is modified, it does not reflect the image change immediately since CloudFront take around 24 hours to update.
As a workaround, I'm planning to call CreateInvalidation to reflect the file change immediately.
Is it possible to use this invalidation call without SDK?
Using ColdFusion programming language and does not seem to have SDK for this.
You can simply make POST request. Example on PHP from Steve Jenkins
<?php
/**
* Super-simple AWS CloudFront Invalidation Script
* Modified by Steve Jenkins <steve stevejenkins com> to invalidate a single file via URL.
*
* Steps:
* 1. Set your AWS Access Key
* 2. Set your AWS Secret Key
* 3. Set your CloudFront Distribution ID (or pass one via the URL with &dist)
* 4. Put cf-invalidate.php in a web accessible and password protected directory
* 5. Run it via: http://example.com/protected_dir/cf-invalidate.php?filename=FILENAME
* or http://example.com/cf-invalidate.php?filename=FILENAME&dist=DISTRIBUTION_ID
*
* The author disclaims copyright to this source code.
*
* Details on what's happening here are in the CloudFront docs:
* http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html
*
*/
$onefile = $_GET['filename']; // You must include ?filename=FILENAME in your URL or this won't work
if (!isset($_GET['dist'])) {
$distribution = 'DISTRIBUTION_ID'; // Your CloudFront Distribution ID, or pass one via &dist=
} else {
$distribution = $_GET['dist'];
}
$access_key = 'AWS_ACCESS_KEY'; // Your AWS Access Key goes here
$secret_key = 'AWS_SECRET_KEY'; // Your AWS Secret Key goes here
$epoch = date('U');
$xml = <<<EOD
<InvalidationBatch>
<Path>{$onefile}</Path>
<CallerReference>{$distribution}{$epoch}</CallerReference>
</InvalidationBatch>
EOD;
/**
* You probably don't need to change anything below here.
*/
$len = strlen($xml);
$date = gmdate('D, d M Y G:i:s T');
$sig = base64_encode(
hash_hmac('sha1', $date, $secret_key, true)
);
$msg = "POST /2010-11-01/distribution/{$distribution}/invalidation HTTP/1.0\r\n";
$msg .= "Host: cloudfront.amazonaws.com\r\n";
$msg .= "Date: {$date}\r\n";
$msg .= "Content-Type: text/xml; charset=UTF-8\r\n";
$msg .= "Authorization: AWS {$access_key}:{$sig}\r\n";
$msg .= "Content-Length: {$len}\r\n\r\n";
$msg .= $xml;
$fp = fsockopen('ssl://cloudfront.amazonaws.com', 443,
$errno, $errstr, 30
);
if (!$fp) {
die("Connection failed: {$errno} {$errstr}\n");
}
fwrite($fp, $msg);
$resp = '';
while(! feof($fp)) {
$resp .= fgets($fp, 1024);
}
fclose($fp);
print '<pre>'.$resp.'</pre>'; // Make the output more readable in your browser
Some alternatives to invalidating an object are:
Updating Existing Objects Using Versioned Object Names, such as image_1.jpg changing to image_2.jpg
Configuring CloudFront to Cache Based on Query String Parameters such as configuring CloudFront to recognize parameters (eg ?version=1) as part of the filename, therefore your app can reference a new version by using ?version=2 and that forces CloudFront to treat it as a different object
For frequent modifications, I think the best approach is to append a query string to the image url (time stamp or hash value of the object) and configure Cloudfront to forward query strings, which will always return the latest image when query string differs.
For infrequent modifications, apart from SDK, you can use AWS CLI which will also allows to invalidate the cache upon builds, integrating with your CI/CD tools.
E.g
aws cloudfront create-invalidation --distribution-id S11A16G5KZMEQD \
--paths /index.html /error.html

Resources