Varnish as a cache in front of apache virtual hosts - varnish

I would like to configure varnish as a cache for one of my websites for a temporary heavy load.
I set up several virtual machines with varnish that should cache my main website.
As my main server hosts several websites the apache server is configured to be a virtual host server.
So I defined the main website domainname in the default.vcl instead of an IP address, hoping varnish would propagate the requests properly to the right apache virtualhost.
But it seems it just uses the IP address, and is caching the apache default page that appears when sending a http request to my main website ip address.
Is there a way to configure varnish so it calls my backend using the right url, and not an IP:port ?
my varnish config looks like :
backend default{
.host = "www.myvhost.com";
.port = "80";
}
instead of :
backend default{
.host = "my.ip";
.port = "80";
}
because I need the varnish instances : http://www1.myvhost.com, http://www2.myvhost.com ... to cache the main server http://www.myvhost.com
but not http://myip:80/
Thanks for your help

You probably ean the varnish is the reverse proxy not the apache
As my main server hosts several websites the apache server is
configured to be a reverse proxy.
You can configure your hostname in the backend definition of your default.vcl
https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
All your traffic is going to point on the varnish for all your vhosts, excepts if you have several IP on your server.
I think what you are looking for is flitering queries for a single domain. You can achieve this by filtering on the host.
sub vcl_recv {
if (req.http.host ~ "(www\.)?yourdomain\.com") {
return(pass);
}
}
All traffic not on your domain will be directly pass to the backend without lookup

Related

How might one set up a reverse proxy that cannot decrypt traffic?

I'd like to have a reverse HTTPS proxy that CANNOT decrypt proxied traffic (ie an HTTPS passthrough/tunnel). The idea is to run this proxy on a VPS and point a domain to it, allowing for the IP address of the origin server to remain unexposed while maintaining end-to-end encryption.
Is this possible? I could probably proxy requests without difficulty since the destination address in that direction is fixed, but proxying responses seems problematic given that the proxy would be unable to read the client IP within an encrypted response.
A potential solution is to have the origin server package the encrypted response and destination address in a request made to the proxy, but I am unsure as to how I might generate the encrypted request without sending it (using node.js, which is the application running on the origin server).
From your question, I got that you want to listen to requests from your VPC server and pass the request to your other server which has to remain unexposed.
This can be configured with the web server which you are using for proxy ( considering AWS allows port forwarding from a VPN server to non-VPN server ).
I prefer doing this with Nginx as it is easy, open-source with less code and more functionality.
There is a concept of load balancing which does the same as you mentioned above.
steps :
Install Nginx and keep it active.
Create a new configuration file in /etc/nginx/sites-enabled
write the below code with modifications:
http {
upstream myapp1 {
server srv1.example.com;
server srv2.example.com;
server srv3.example.com;
}
server {
listen 80;
location / {
proxy_pass http://myapp1;
}
}
}
and at the place of srv1.example.com and srv2.example.com add the domain to which you want to redirect requests
Save the file and restart the Nginx
Boom!! it should redirect all incoming requests to your application.

How to route requests in varnish on the basis of request parameters (path, query), onto a changing list of backend servers

How to route requests in varnish on the basis of request parameters (path, query), onto a changing list of backend servers.
Where each request can be handled by a particular backend server.
A simple example
Here's a simple example where you'll inspect req.url and look for the right path and querystring. Based on the value a backend will be selected:
vcl 4.1;
backend backend1 {
.host = "backend1.example.com";
.port = "80";
}
backend backend2 {
.host = "backend2.example.com";
.port = "80";
}
sub vcl_recv {
if(req.url ~ "/\?action=foo") {
set req.backend_hint = backend1;
} else {
set req.backend_hint = backend2;
}
}
You can make the logic as complex as you want and add is many if-statements as you want.
Limitations
There are 3 very specific limitations to the example above:
The backends are static and should be explicitly defined in VCL
DNS hostnames are resolved at compile time and DNS changes aren't noticed at runtime
You cannot use request or runtime information to compose the hostname of your backends
I noticed the term a changing list of backends in your question. I'm not sure to what extent static backends will affect you. But if they do, the only real solution is to use dynamic backends in Varnish Enterprise.
Dynamic backends in Varnish Enterprise
Varnish Enterprise, the commercial version of Varnish offers a dynamic backends module that circumvents these limitations.
Backends can be defined on-the-fly and DNS changes are spotted when the DNS TTL expires.
Here's an example:
vcl 4.1;
import goto;
backend default none;
sub vcl_backend_fetch {
if(bereq.url ~ "^/(images|videos|static)/\?country=(us|uk|fr|jp)") {
}
set bereq.backend = goto.dns_backend(regsub(bereq.url,"^/(images|videos|static)/\?country=(us|uk|fr|jp)","\1.\2.backend.example.com"));
}
When the request URL is /images/?country=uk, the backend would become images.uk.backend.example.com.
This is just a hypothetical example. The point I'm trying to make is that you can define backends on the fly and that DNS changes are spotted at runtime.
See https://docs.varnish-software.com/varnish-cache-plus/vmods/goto/ if you want to learn more about dynamic backends in Varnish Enterprise.
Varnish Enterprise disclaimer
This post is not a Varnish Enterprise advertisement. If you can solve your issues with open source Varnish, that's absolutely fine.
If Varnish Cache would limit you in your use case, Varnish Enterprise is there to solve that problem.
Although Varnish Enterprise is commercial software that requires purchasing a license, there are cheap ways to get started.
If you're running or testing your application on Cloud platforms such as, AWS, Azure or Google Cloud, you can use one of our Varnish Enterprise machine images without having to purchase a license up front.
License fees are billed per hour by the Cloud platform and there are free trials available. Some platforms even have a cheaper "developer edition" that allow you to test Varnish Enterprise on smaller Cloud instances.

Nginx and Dedicated Varnish

I’m new in varnish, is it possible configure varnish on dedicated varnish server? I have separate nginx lb in front of kubernetes cluster. My goal is caching a lot of static files like .js .css and images or even static page, so every request cache related to services in kubernetes cluster will serve in varnish server, is it possible to do that? I attach varnish configuration, please check
10.10.10.27: nginx-lb-01, 10.10.10.28: nginx-lb-02, 10.10.10.29: Varnish
I already try to configure but I think it failed because when Im check using varnishstat, there’s no traffic average statistic. In every nginx vhost I already config default port site 8080 & redirect to 443
How can I solve this ?
Thank you
varnish config screenshot

Setting up my first Varnish Cache server

I am attempting to set up my first Varnish Cache server and I have a couple questions for any person(s) experienced.
1.) I am running Varnish as a stand alone server. Do I need Apache also installed on the same server. Ultimately the actual site that will be behind Varnish is not on this server.
2.) Do I point the domain to Varnish and then set the config to point to the ip address of the server that is hosting the site? If so, how do you point it to the right site?
3.) If Varnish is standalone and I have an Apache content server, can they both be port 80 and just change the ip address in the default.vcl
backend default {
.host = "198.221.134.235";
.port = "80";
}
Sorry for the basic questions. I have been on Google all weekend and I found plenty of information on how to install and config Varnish but it seems like the site you want to Cache is on the same server since all of them are changing the port Apache listens to and that seems like it would mean the site is living on the same server.
And if you have any good sites with information, please feel free to share them! Thanks again!
No, Varnish and Apache (or any other HTTP/webserver) can run on a separate server.
Indeed, point the domain to the IP of Varnish and setup a backend as described in the documentation: https://www.varnish-cache.org/docs/3.0/tutorial/backend_servers.html. The IP
of your webserver will be the IP of the backend.
Correct, as long as Apache and Varnish are on separate servers they both can listen on port 80
If I am not mistaken you will have the following setup:
DNS example.com => 1.1.1.1
IP 1.1.1.1:80: Varnish (backend: 1.1.1.2:80)
IP 1.1.1.2:80: Apache

Varnish removes Public IP from X-Forwarded-for

I am facing an issue where varnish is not sending Intermediary proxy IP or Public IP in a particular case.
Scenario is as below :
Some Hotel / Company has squid proxy configured and all traffic for Internet is routed via Squid.
User accessing my company's site first hits the Load Balancer then Varnish & then Apache
Apache is configured with mod_geoip. The code on my site does the Country redirection based on the IP address.
Problem :
When an user (Behind that squid proxy) accessing my company's website behind Load Balancer -> VARNISH -> Apache - Here apache gets only Internal IP (His Private IP) & Load Balancer Internal IP as X-forwarded-for and Hence the IP based redirection FAILS!
In Apache logs (Configured to log X-Forwarded-IP) I see that Users Private IP & then My Load Balancers Private IP.
172.10.5.10, LoadBalancerIP - - [.......]
The same user when accesses another site which does NOT have Varnish, hits Load Balancer -> Apache - Here apache gets Users Private IP & Users Public IP as X-forwarded-for and IP based country redirection works fine.
In Apache logs (Configured to log X-Forwarded-IP) I see that Users Private IP and then his Public IP is also logged.
172.10.5.10, PublicIP - - [.......]
My Varnish Config is as below.
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
PS: I've already Google'd all links I could find and every link gives the following Varnish Config
For now to get this issue resolved, I had to bypass varnish and now website traffic is directly hitting Apache but I need to get Varnish back in place to server content from Cache and Speedy delivery.
Will appreciate if someone can guide me to how resolve this issue.
Thanks!
The above mentioned issue has been resolved. Credit goes to Mithrandir # Varnish IRC Channel. Thanks!
Below mentioned changes were required to resolve the issue.
At the start of the default.vcl add :
import std;
Below " if (req.http.x-forwarded-for) " add :
std.collect(req.http.x-forwarded-for);
Do varnish configtest & reload. This should start showing the Public IP.
Below is the explanation from the documentation of vmod_std :
collect
Prototype
collect(HEADER header)
Return value
Void
Description
Collapses the header, joining the headers into one.
Example
std.collect(req.http.cookie); This will collapse several Cookie:
headers into one, long cookie header.
In Varnish 4 this is not needed anymore, see upgrade info here

Resources