How to route requests in varnish on the basis of request parameters (path, query), onto a changing list of backend servers - varnish

Question

How to route requests in varnish on the basis of request parameters (path, query), onto a changing list of backend servers.
Where each request can be handled by a particular backend server.

Answer 1

A simple example
Here's a simple example where you'll inspect req.url and look for the right path and querystring. Based on the value a backend will be selected:
vcl 4.1;
backend backend1 {
.host = "backend1.example.com";
.port = "80";
}
backend backend2 {
.host = "backend2.example.com";
.port = "80";
}
sub vcl_recv {
if(req.url ~ "/\?action=foo") {
set req.backend_hint = backend1;
} else {
set req.backend_hint = backend2;
}
}
You can make the logic as complex as you want and add is many if-statements as you want.
Limitations
There are 3 very specific limitations to the example above:
The backends are static and should be explicitly defined in VCL
DNS hostnames are resolved at compile time and DNS changes aren't noticed at runtime
You cannot use request or runtime information to compose the hostname of your backends
I noticed the term a changing list of backends in your question. I'm not sure to what extent static backends will affect you. But if they do, the only real solution is to use dynamic backends in Varnish Enterprise.
Dynamic backends in Varnish Enterprise
Varnish Enterprise, the commercial version of Varnish offers a dynamic backends module that circumvents these limitations.
Backends can be defined on-the-fly and DNS changes are spotted when the DNS TTL expires.
Here's an example:
vcl 4.1;
import goto;
backend default none;
sub vcl_backend_fetch {
if(bereq.url ~ "^/(images|videos|static)/\?country=(us|uk|fr|jp)") {
}
set bereq.backend = goto.dns_backend(regsub(bereq.url,"^/(images|videos|static)/\?country=(us|uk|fr|jp)","\1.\2.backend.example.com"));
}
When the request URL is /images/?country=uk, the backend would become images.uk.backend.example.com.
This is just a hypothetical example. The point I'm trying to make is that you can define backends on the fly and that DNS changes are spotted at runtime.
See https://docs.varnish-software.com/varnish-cache-plus/vmods/goto/ if you want to learn more about dynamic backends in Varnish Enterprise.
Varnish Enterprise disclaimer
This post is not a Varnish Enterprise advertisement. If you can solve your issues with open source Varnish, that's absolutely fine.
If Varnish Cache would limit you in your use case, Varnish Enterprise is there to solve that problem.
Although Varnish Enterprise is commercial software that requires purchasing a license, there are cheap ways to get started.
If you're running or testing your application on Cloud platforms such as, AWS, Azure or Google Cloud, you can use one of our Varnish Enterprise machine images without having to purchase a license up front.
License fees are billed per hour by the Cloud platform and there are free trials available. Some platforms even have a cheaper "developer edition" that allow you to test Varnish Enterprise on smaller Cloud instances.