My current setup has 2 HAProxies configured with keepalived for High Availability, the 2 proxies serve as a Reverse Proxy and Load Balancer for virtual webservices. I know that HAProxy can check the health of its backend (I've already configured this) but my question is something else.
At my company there's a F5 Big-IP Load Balancer which serves as the first line of defense, it will redirect requests to my HAProxies when needed.
I need to know if there is a way to let my F5 Big-IP check the health of the HAProxies frontend, so when the proxies are booting no requests will be lost.
Thanks
There used to be a mode health option but in recent versions the easiest way is to use a monitor-uri on a given port:
listen health_check_http_url
bind :8888
mode http
monitor-uri /healthz
option dontlognull
You can use the monitor-uri in a frontend and select it with an ACL too but the port version is much clear and straightforward.
https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4.2-mode
https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4.2-monitor-uri
From the HAProxy Reference Manual:
Health-checking mode
--------------------
This mode provides a way for external components to check the proxy's health.
It is meant to be used with intelligent load-balancers which can use send/expect
scripts to check for all of their servers' availability. This one simply accepts
the connection, returns the word 'OK' and closes it. If the 'option httpchk' is
set, then the reply will be 'HTTP/1.0 200 OK' with no data, so that it can be
tested from a tool which supports HTTP health-checks. To enable it, simply
specify 'health' as the working mode :
Example :
---------
# simple response : 'OK'
listen health_check 0.0.0.0:60000
mode health
# HTTP response : 'HTTP/1.0 200 OK'
listen http_health_check 0.0.0.0:60001
mode health
option httpchk
From the HAProxy Docs
Example:
frontend www
mode http
acl site_dead nbsrv(dynamic) lt 2
acl site_dead nbsrv(static) lt 2
monitor-uri /site_alive
monitor fail if site_dead
Checkout the reference documentation.
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-monitor-uri
<uri> is the exact URI which we want to intercept to return HAProxy's
health status instead of forwarding the request.
When an HTTP request referencing <uri> will be received on a frontend,
HAProxy will not forward it nor log it, but instead will return either
"HTTP/1.0 200 OK" or "HTTP/1.0 503 Service unavailable", depending on failure
conditions defined with "monitor fail". This is normally enough for any
front-end HTTP probe to detect that the service is UP and running without
forwarding the request to a backend server. Note that the HTTP method, the
version and all headers are ignored, but the request must at least be valid
at the HTTP level. This keyword may only be used with an HTTP-mode frontend.
Monitor requests are processed very early. It is not possible to block nor
divert them using ACLs. They cannot be logged either, and it is the intended
purpose. They are only used to report HAProxy's health to an upper component,
nothing more. However, it is possible to add any number of conditions using
"monitor fail" and ACLs so that the result can be adjusted to whatever check
can be imagined (most often the number of available servers in a backend).
Related
Need some help to dig deeper into why IIS is behaving in a certain way. Edge/Chrome makes an HTTP2.0 request to IIS, using the IPv6 address in the header (https://[ipv6]/) which results in the server generating a 302 response. The ISAPI filter makes some mods to the 302 response and replaces the response buffer. IIS drops the request/response and logs in HTTPERR log:
<date> <time> fe80::a993:90bf:89ff:1a54%2 1106 fe80::bdbf:5254:27d2:33d8%2 443 HTTP/2.0 GET <url> 1 - 1 Connection_Dropped_List_Full <pool>
Suspect related to HTTP2.0, when putting Fiddler in the middle, it isn't HTTP/2.0 anymore, it downgrades to HTTP/1.1 and it works.
When using an IPv4 address, it works. In either case the filter goes through the identical steps. There is no indication in the filter that anything went wrong.
Failed Request Tracing will not write buffers for incomplete/dropped requests that appear in HTTPERR log.
Is there a place where I can find out more detailed information about why IIS is dropping the request?
I did the network capture, and looks like browser is initiating the FIN tear down of session.
Do you use any load balance or reverse proxy before request get in IIS? This error indicates that the log cannot store more lost connections, so the problem is that your connection is lost.
If you use load balance, web application is under heavy load and
because of this no threads are available to currently provide
logging data to HTTP.sys. Check this.
Or before IIS response to client, client has closed the request but
IIS still send response. This is more likely to be a problem with the
application itself not IIS and http.sys. Check this.
One thing I noticed is if you change http2 to 1.1, it can work well. The difference between http1.1 and 2 is performance.
HTTP/1.1 practically allows only one outstanding request per TCP connection (though HTTP pipelining allows more than one outstanding request, it still doesn’t solve the problem completely).
HTTP/2.0 allows using same TCP connection for multiple parallel requests
So it looks like that when you use http2, one connection includes multiple requests and application cannot handle these requests well, especially the request of image.
Aother thing is failed request tracing can catch all request and response, including status code is 200 and 302.
I have an architecture where I have multiple instances but I want to maximize cache hits.
Users are defined in groups and I want to make sure that all users that belong to the same group hit the same server as much as possible.
The application is fully stateless, but having users from the same group hitting the same server will dramatically increase performance and memory load on all instances.
When loading the main page I already know which server I would like to send this user to on the XHR call.
Using the ARRAffinity cookies are not really great is almost impossible in this scenario (cross domain, have to make server call first etc) and I would strongly prefer sending a hint myself through a custom header.
I'm trying manually to do some workarounds with deleting the cookies and assigning them, but it feels very hacky and I don't get it fully working yet. and it doesn't work for XHR calls.
Question:
Is it possible to direct to a specific instance through a header, url or domain instead of a cookie?
Notes
Distributed cache does not work for me in this case. I need the performance of memory cache without extra network hops and serialization/deserialization.
This seems to be possible with Application Gateway, but it seem to need a lot of extra infrastructure and moving parts while all my problems would be fixed by sending the "right" header.
I could fix this by duplicating the web app in its entirety and assigning a different hostname. Also this feels like adding a lot of extra moving parts that can break. Also maintenance will be harder and more confusing, I loss autoscale, etc.
Maybe this can be fixed easily by Kubenetes/Docker Swarm type of architecture (no experience), but as this is a large legacy project and I have a pretty strict deadline I am very cautious of making such a dramatic switch last minute.
If I am understanding correctly you want to set a custom header via a client application and based on that proxy over the connection to some other backend server.
I like to use HAProxy for that, you can also look into Nginx as well for that.
You can install HAProxy on linux from the distribution's package manager, or you can use the available HAProxy docker container.
An example of installing it on ArchLinux:
sudo pacman -S haproxy
sudo systemctl start haproxy
Once its installed you can find out where your haproxy.cfg config file is located and then copy the haproxy.cfg config snippet that I posted here below instead of the existing default config.
In my case haproxy.cfg was in /etc/haproxy/haproxy.cfg
To achieve what you want in HAProxy you would set all clients to communicate with this main HAProxy server, which would then forward the connection to the different backend servers you have based on the value of the custom header that you can set client side, for example "x-mycustom-header: Server-one". As a bonus, you can also enable sticky sessions as well if needed on HAProxy, but it is not mandatory to achieve what you are looking for.
Here is a simple example setup for the HAProxy config file (haproxy.cfg) with only 2 backend servers, but you can add more.
The logic here is that all the clients would make http requests to the HAProxy server listening on port 80, then HAProxy would check the value of the custom header called 'x-mycustom-header' that the clients added and based on that value, it will forward the client to either backend_server_one or backend_server_two.
For testing purposes both HAProxy and the two backends are on the same box but listening on different ports. HAProxy on port 80, server1 is on 127.0.0.1:3000 and server2 is on 127.0.0.1:4000.
cat haproxy.cfg
#---------------------------------------------------------------------
# Example configuration. See the full configuration manual online.
#
# http://www.haproxy.org/download/1.7/doc/configuration.txt
#
#---------------------------------------------------------------------
global
maxconn 20000
log 127.0.0.1 local0
user haproxy
chroot /usr/share/haproxy
pidfile /run/haproxy.pid
daemon
frontend main
bind :80
mode http
log global
option httplog
option dontlognull
option http_proxy
option forwardfor except 127.0.0.0/8
maxconn 8000
timeout client 30s
use_backend backend_server_one if { req.hdr(x-mycustom-header) server-one }
use_backend backend_server_two if { req.hdr(x-mycustom-header) server-two }
default_backend backend_server_one #when the header is something else default to the first backend
backend backend_server_one
mode http
balance roundrobin
timeout connect 5s
timeout server 5s
server static 127.0.0.1:3000 #change this ip to your 1st backend ip address
backend backend_server_two
mode http
balance roundrobin
timeout connect 5s
timeout server 30s
server static 127.0.0.1:4000 #change this ip to your 2nd backend ip address
To test that this works you can open two netcat listeners, one on port 3000, and then the other on port 4000, run them on differnt screens or different ssh sessions.
nc -l 3000 # in the first screen
nc -l 4000 # in a second screen
Then after you do a sudo systemctl reload haproxy to make sure that HAProxy is reloaded with your new config file, you can make an http GET request on port 80 and provide the "x-mycustom-header: Server-one" header.
You will be able to see the request in the output of the netcat instance that is listening on port 3000.
Now change the header to "x-mycustom-header: Server-two" and make a second GET request, and you will see that the request reached to the second netcat instance this time, which is listening on port 4000, which indicates that this works.
Tested on ArchLinux
The Microsoft Team has responded an confirmed this is at the moment not possible.
I have installed HAProxy 1.5.18 on a Centos 7.
In the /etc/haproxy/haproxy.cfg I have following line:
frontend free_api
bind *:80
stats uri /haproxy?stats
mode http
option forwardfor
acl key1 urlp(key) 12345
acl key2 urlp(key) 6789
http-request deny if key1
http-request deny if key2
# use_backend api if api
default_backend api
Right now the URL access gets denied for query string key with a matching value of 12345 or 6789. HAProxy returns a 403 Forbidden status code back.
What I am looking is to simply drop the connection so nothing is returned back to user? How to do that in HAProxy?
Thanks.
use haproxy v1.6 and higher, and directive silent-drop
http-request silent-drop if key1
"silent-drop" : this stops the evaluation of the rules and makes the
client-facing connection suddenly disappear using a system-dependent way
that tries to prevent the client from being notified. The effect it then
that the client still sees an established connection while there's none
on HAProxy. The purpose is to achieve a comparable effect to "tarpit"
except that it doesn't use any local resource at all on the machine
running HAProxy. It can resist much higher loads than "tarpit", and slow
down stronger attackers. It is important to understand the impact of using
this mechanism. All stateful equipment placed between the client and
HAProxy (firewalls, proxies, load balancers) will also keep the
established connection for a long time and may suffer from this action.
On modern Linux systems running with enough privileges, the TCP_REPAIR
socket option is used to block the emission of a TCP reset. On other
systems, the socket's TTL is reduced to 1 so that the TCP reset doesn't
pass the first router, though it's still delivered to local networks. Do
not use it unless you fully understand how it works.
My online whiteboard application has been working previously, but for whatever reason, it no longer works. Now it shows a 503 Varnish cache error as seen here: http://grab.by/eFHG
Do you happen to know where I should start to look to try to resolve this issue?
Thanks!
Donny
This error means that Varnish had no response (even not an HTTP error) from the backend within the timeout limit.
You can troubleshoot that in many ways :
On the backend : do you see requests from Varnish in your webserver
log ?
On Varnish server : Run varnishlog and check the request process. You should have events in this order : RxRequest > TxRequest > RxResponse > TxResponse. Your problem is between TxRequest (request sent to backend) and RxResponse (response received from backend).
On your Varnish server try connecting on the backend using telnet (telnet ). Does it connect ? If it does, try sending a request (e.g. "GET / "). Do you receive a response ?
Probable causes could be : firewall/selinux blocking between varnish & backend, bad varnish or backend web server config (are backend address & port sync ?), webserver stopped, ...
You could always check your /etc/varnish/default.vcl (CentOS).
backend default {
.host = "127.0.0.1";
.port = "80";
}
Make sure the .host value is your server IP Address, and change the port to 8080, and adjust your port setting in /etc/httpd/conf/httpd.conf and make sure Apache listen to 8080.
EDIT
It could also mean web server (Apache) have wrong/default settings.
Check the virtual host ports, the values might be 80, they should be
8080.
Check Listen directive and ServerName, they must have 8080
port.
Make sure you do sudo systemctl enable varnish.
And for the final touch, do sudo reboot (cache programs are bugging me in the development state).
When I tried to reach this page today, I got shown the error page instead with following contents:
Error 503 Service Unavailable
Service Unavailable
Guru Mediation:
Details: cache-hel6832-HEL 1623148717 170212118
So it seems StackOverflow is also using a CDN that was failing today, and the failure was impacting huge number of services at the same time. In these kind of situations, we might be at the mercy of external CDN provider. Today showed us that sometimes the only thing that you can do, is to wait for the others to solve it for you.
Luckily these kind of outages are shortlived.
Suitable long term strategy is to use several CDN services instead of single.
We're planning to add a second varnish server to our infraestructure.
What it's the better method to balance the traffic throw the two servers? I think we can use haproxy in front of the two servers, but how to configure it to load balance the traffic between the 2 varnish? The ideal solution is that if one varnish is down all the traffic goes to the other.
Edit: The ideal behaviour is an active/active conf, with 50% load each and if one goes down haproxy sends 100% load to the other.
Another option would be to put both varnish instances in the same backend, but balance requests by uri or parameter.
This will allow you to have both active in the same backend, and still maintain a high cache-hit ratio since the same uri will be always balanced to the same varnish cache (as long as it is available). Balace uri also takes an optional parameter of length to use when hashing.
Heres a quick example:
frontend http-in
acl my_acl
use_backend varnish if my_acl
backend varnish1
mode http
balance uri
option httpchk GET /check.txt HTTP/1.0
server varnish1 192.168.222.51:6081 check inter 2000
server varnish2 192.168.222.52:6081 check inter 2000
The idea I have come up is use two backends and have in each backed use the other server as backup so when a server is down all the request goes to the alive server.
frontend http-in
acl my_acl <whaever acl to split the traffic>
use_backend varnish2 if my_acl
default_backend varnish1
backend varnish1
mode http
option httpchk GET /check.txt HTTP/1.0
server varnish1 192.168.222.51:6081 check inter 2000
server varnish2 192.168.222.52:6081 check inter 2000 backup
backend varnish2
mode http
option httpchk GET /check.txt HTTP/1.0
server varnish2 192.168.222.51:6081 check inter 2000
server varnish1 192.168.222.52:6081 check inter 2000 backup