Varnish removes Public IP from X-Forwarded-for - varnish

I am facing an issue where varnish is not sending Intermediary proxy IP or Public IP in a particular case.
Scenario is as below :
Some Hotel / Company has squid proxy configured and all traffic for Internet is routed via Squid.
User accessing my company's site first hits the Load Balancer then Varnish & then Apache
Apache is configured with mod_geoip. The code on my site does the Country redirection based on the IP address.
Problem :
When an user (Behind that squid proxy) accessing my company's website behind Load Balancer -> VARNISH -> Apache - Here apache gets only Internal IP (His Private IP) & Load Balancer Internal IP as X-forwarded-for and Hence the IP based redirection FAILS!
In Apache logs (Configured to log X-Forwarded-IP) I see that Users Private IP & then My Load Balancers Private IP.
172.10.5.10, LoadBalancerIP - - [.......]
The same user when accesses another site which does NOT have Varnish, hits Load Balancer -> Apache - Here apache gets Users Private IP & Users Public IP as X-forwarded-for and IP based country redirection works fine.
In Apache logs (Configured to log X-Forwarded-IP) I see that Users Private IP and then his Public IP is also logged.
172.10.5.10, PublicIP - - [.......]
My Varnish Config is as below.
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
PS: I've already Google'd all links I could find and every link gives the following Varnish Config
For now to get this issue resolved, I had to bypass varnish and now website traffic is directly hitting Apache but I need to get Varnish back in place to server content from Cache and Speedy delivery.
Will appreciate if someone can guide me to how resolve this issue.
Thanks!

The above mentioned issue has been resolved. Credit goes to Mithrandir # Varnish IRC Channel. Thanks!
Below mentioned changes were required to resolve the issue.
At the start of the default.vcl add :
import std;
Below " if (req.http.x-forwarded-for) " add :
std.collect(req.http.x-forwarded-for);
Do varnish configtest & reload. This should start showing the Public IP.
Below is the explanation from the documentation of vmod_std :
collect
Prototype
collect(HEADER header)
Return value
Void
Description
Collapses the header, joining the headers into one.
Example
std.collect(req.http.cookie); This will collapse several Cookie:
headers into one, long cookie header.

In Varnish 4 this is not needed anymore, see upgrade info here

Related

404 error when connecting to IP address but not in domain name

I ran a website with Nginx and NodeJS on Lightsail.
When I go to my mywebsite.com, my website is accessible and is secure. However, when I try to enter the public IP for that website, Nginx returns a 404 error page. The website is certified by LetsEncrypt.
I do manage to access the IP address on the browser by adding http:// before the IP address. However, when I normally go to my IP address, Nginx returns a 404 error. Why is this so?
I'm fine with the IP being not accessible. But my CDN is not working because of it. I think it needs to be accessible and not return a 404 page?
Why do I go about diagnosing and solving this problem?
Thanks in advance.
The Host: header identifies the hostname/domain-name used by the browser to access a website when the webserver is serving multiple websites on the same TCP binding (IP address + port number + protocol). The webserver then uses that to determine what content to serve.
Web-browsers generate the Host: header from the URI in the address bar.
When you access http://1.2.3.4/foobar your browser sends this request:
GET /foobar
Host: 1.2.3.4
When you access http://example.com/foobar your browser sends this request:
GET /foobar
Host: example.com
When you access http://www.example.com/foobar your browser sends this request:
GET /foobar
Host: www.example.com
When you access https://example.com/foobar or https://www.example.com/foobar the requests are the same, but the underlying HTTP connection is wrapped in a TLS (formerly SSL) connection. Things get complicated with HTTP/2 (SPDY) and HTTP/3 (QUIC) so I won't explain how those work.
In your case, you need to edit your nginx configuration to add another binding to your website that maps http://<IP-address> to your website, as well as https://<IP-address>. You need to check your web-application's code to ensure that it can handle requests using unexpected Host: headers - you can also configure nginx to modify the Host header for incoming requests to your usual header.
If you're using HTTPS then you'll need to add your IP address as a SAN (Subject Alternate Name) to your TLS certificate - which is generally a bad idea because IP addresses change and LetsEncrypt doesn't provide certificates for IP addresses anyway so this is moot.
For dev purposes and when you're running nginx locally (i.e. http://localhost) then you no-longer need a TLS certificate for localhost because since early 2020 Chromium-based browsers now recognize http://localhost as being a "secure" website even though it's over http:// and not https://:
https://www.chromestatus.com/feature/6269417340010496
https://bugs.chromium.org/p/chromium/issues/detail?id=589141
https://bugs.chromium.org/p/chromium/issues/detail?id=691930
For an immediate workaround, you can use a browser-extension to override the Host: header when you're accessing a site by IP address.
e.g.:
https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj?hl=en
https://chrome.google.com/webstore/detail/modify-header-value-http/cbdibdfhahmknbkkojljfncpnhmacdek?hl=en
https://chrome.google.com/webstore/detail/requestly-redirect-url-mo/mdnleldcmiljblolnjhpnblkcekpdkpa?hl=en

Varnish as a cache in front of apache virtual hosts

I would like to configure varnish as a cache for one of my websites for a temporary heavy load.
I set up several virtual machines with varnish that should cache my main website.
As my main server hosts several websites the apache server is configured to be a virtual host server.
So I defined the main website domainname in the default.vcl instead of an IP address, hoping varnish would propagate the requests properly to the right apache virtualhost.
But it seems it just uses the IP address, and is caching the apache default page that appears when sending a http request to my main website ip address.
Is there a way to configure varnish so it calls my backend using the right url, and not an IP:port ?
my varnish config looks like :
backend default{
.host = "www.myvhost.com";
.port = "80";
}
instead of :
backend default{
.host = "my.ip";
.port = "80";
}
because I need the varnish instances : http://www1.myvhost.com, http://www2.myvhost.com ... to cache the main server http://www.myvhost.com
but not http://myip:80/
Thanks for your help
You probably ean the varnish is the reverse proxy not the apache
As my main server hosts several websites the apache server is
configured to be a reverse proxy.
You can configure your hostname in the backend definition of your default.vcl
https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
All your traffic is going to point on the varnish for all your vhosts, excepts if you have several IP on your server.
I think what you are looking for is flitering queries for a single domain. You can achieve this by filtering on the host.
sub vcl_recv {
if (req.http.host ~ "(www\.)?yourdomain\.com") {
return(pass);
}
}
All traffic not on your domain will be directly pass to the backend without lookup

Getting Client Ip on NODE.JS Through Cloudflare

I am currently using
data["ip"] = socket.client.conn.remoteAddress;;
But this will only show me the cloudflare IPs and I am not a fan of leaking my server's ip so how can I get my client's ip through cloudflare?
After me having to research for a couple of good hours I found out.
socket.client.request.headers['cf-connecting-ip']
This thread says that in socket.io 1+, you should be using:
socket.request.connection.remoteAddress
or:
socket.handshake.address
But, if there are cloudfare proxy servers in the path (which there presumably are), then this will only report the true client IP address if the cloudfare proxy servers set the appropriate header to the actual client IP address when they proxy the initial HTTP connection that initiates the socket.io connection.

Setting up my first Varnish Cache server

I am attempting to set up my first Varnish Cache server and I have a couple questions for any person(s) experienced.
1.) I am running Varnish as a stand alone server. Do I need Apache also installed on the same server. Ultimately the actual site that will be behind Varnish is not on this server.
2.) Do I point the domain to Varnish and then set the config to point to the ip address of the server that is hosting the site? If so, how do you point it to the right site?
3.) If Varnish is standalone and I have an Apache content server, can they both be port 80 and just change the ip address in the default.vcl
backend default {
.host = "198.221.134.235";
.port = "80";
}
Sorry for the basic questions. I have been on Google all weekend and I found plenty of information on how to install and config Varnish but it seems like the site you want to Cache is on the same server since all of them are changing the port Apache listens to and that seems like it would mean the site is living on the same server.
And if you have any good sites with information, please feel free to share them! Thanks again!
No, Varnish and Apache (or any other HTTP/webserver) can run on a separate server.
Indeed, point the domain to the IP of Varnish and setup a backend as described in the documentation: https://www.varnish-cache.org/docs/3.0/tutorial/backend_servers.html. The IP
of your webserver will be the IP of the backend.
Correct, as long as Apache and Varnish are on separate servers they both can listen on port 80
If I am not mistaken you will have the following setup:
DNS example.com => 1.1.1.1
IP 1.1.1.1:80: Varnish (backend: 1.1.1.2:80)
IP 1.1.1.2:80: Apache

Grails get Client IP when deploying to Cloud?

I have a Grails App and I am looking to lock it down by ip so that only a number of IP ranges can access the application. I have used the Spring Security command within Config to achieve this:
grails.plugins.springsecurity.ipRestrictions
Then when running the App in the cloud (Jelastic) even though I am on one of the IP's I listed it doesn’t let me access the areas I want. I then put some code in the app shown below to pull back the address of the Client and it shows the address of maybe the cloud proxy server instead of the Client using the App:
request.getRemoteAddr()
I think that it wont let me access the areas I want as its reading my IP as the IP of the cloud proxy, I have also tried running the commands below to see if any of them return my actual IP, however they were all null :S
request.getHeader("X-Forwarded-For");
request.getHeader("Proxy-Client-IP");
request.getHeader("WL-Proxy-Client-IP")
request.getHeader("HTTP_CLIENT_IP")
request.getHeader("HTTP_X_FORWARDED_FOR")
I just need to know if there is some way of restricting this application down in the cloud by Client IP instead of it using the IP of the Cloud Proxy? Thanks in advance
All requests to Jelastic instances are coming through infrastructure global Resolver.
So, you are right, request.getRemoteAddr() returns IP of Resolver and it's not recognized by your allowed list.
Workaround for this is purchasing external IP for your app server in Jelastic. In this case all requests will come directly to your instance.
I also recommend you get on board the dedicated Jelastic Community in order to share your experience and get help from others.
Have you configured a nginx instance in front of your tomcat? I am not sure if there are any jelastic specifics, but you have to configure nginx so that it passes the ip to the proxied service, see http://wiki.nginx.org/HttpRealIpModule
You could e.g. set a custom header, if you don't want to overwrite the defaults:
proxy_set_header X-Real-IP $remote_addr;
Have you tried dumping the headers you actually get in your app?
On Cloudfoundry.com I can see that I get 'x-forwarded-for'
class HeaderController {
def headerTest = {
def headerNames = request.headerNames.collect{ it }
headerNames.each {
render "$it : ${request.getHeader(it)}\n"
}
render "Remote addr : ${request.getRemoteAddr()}\n"
render "Forward addr : ${request.getHeader('x-forwarded-for' )}\n"
}
}
In Jelastic cloud 'x-forwarded-for' displays your IP as well.
As a follow up I suggest you to check the related topic on Jelastic Community
So supposedly you'd have to set your IP restriction config in a way that it checks the value of 'x-forwarded-for'.

Resources