currently I am using pfSense on my server with the HAProxy package, because I can easily configure it via the GUI.
I configured HAProxy to act as a reverse proxy corresponding to this guide: https://blog.devita.co/pfsense-to-proxy-traffic-for-websites-using-pfsense/
SSL offloading works like a charm. The problem I have is when I have more than one service (open port) on the same internal IP it seems not to be working.
Example:
I configure service1.domain.com for Service1 with port 8000 (10.100.10.101:8000) and it works flawlessly.
Now I need another port on the same machine (e.g. 10.100.10.101:8082) with another service. If I configure another backend pointing to the same IP but with a different port I can only reach the second servce (service2.domain.com) even if I access service1.domain.com.
My use case is that I am trying to set up Seafile which is using port 8000 for the web GUI and port 8082 for the fileserver. Right now I am able to access the web GUI but I am not able to upload, download or share files.
My configuration:
# Automaticaly generated, dont edit manually.
# Generated on: 2018-09-29 19:24
global
maxconn 1000
stats socket /tmp/haproxy.socket level admin
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 8192
server-state-file /tmp/haproxy_server_state
ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000
frontend shared-frontend-merged
bind X.X.X.X:443 name X.X.X.X:443 ssl crt-list /var/etc/haproxy/shared-frontend.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
http-response set-header Strict-Transport-Security max-age=15768000
acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.domain\.com(:([0-9]){1,5})?$
acl ACL1 var(txn.txnhost) -m str -i test.domain.com
acl ACL2 var(txn.txnhost) -m str -i service1.domain.com
acl ACL3 var(txn.txnhost) -m str -i service2.domain.com
http-request set-var(txn.txnhost) hdr(host)
default_backend test.domain.com_ipv4
default_backend service1.domain.com_ipvANY
default_backend service2.domain.com_ipvANY
frontend http-to-https
bind X.X.X.X:80 name X.X.X.X:80
mode http
log global
option http-keep-alive
timeout client 30000
http-request redirect scheme https
backend test.domain.com_ipv4
mode http
id 10100
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4# usesrc clientip
option httpchk GET /
server testvm-server01 10.100.10.101:54080 id 10101 check inter 1000
backend service1.domain.com_ipvANY
mode http
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server seafile-vm-01 10.100.10.103:8000 id 101 check inter 1000
backend service2.domain.com_ipvANY
mode http
id 104
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server seafile-vm-02 10.100.10.103:8082 id 103 check inter 1000
I would really be glad if anyone can point me in the right direction, thank you in advance and if you need further information please tell me.
Best regards,
Bioneye
I was able to solve my problem with the help of one awesome user over on reddit.
The first problem was that I misconfigured my frontend and thus had 3 default_backends. That was the reason why every services pointed to the same virtual machine. To solve it I just had to add the if condition corresponding to my ACL name.
The second problem was that my Service2 was shown as DOWN on the HAProxy stats page. I had to change the health check method from HTTP to Basic and that finally resolved everything.
This is the working configuration:
# Automaticaly generated, dont edit manually.
# Generated on: 2018-10-02 16:59
global
maxconn 1000
stats socket /tmp/haproxy.socket level admin
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 8192
server-state-file /tmp/haproxy_server_state
ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000
frontend shared-frontend-merged
bind X.X.X.X:443 name X.X.X.X:443 ssl crt-list /var/etc/haproxy/shared-frontend.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
http-response set-header Strict-Transport-Security max-age=15768000
acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.domain\.com(:([0-9]){1,5})?$
acl ACL1 var(txn.txnhost) -m beg -i test.domain.com
acl ACL2 var(txn.txnhost) -m beg -i service1.domain.com
acl ACL3 var(txn.txnhost) -m beg -i service2.domain.com
http-request set-var(txn.txnhost) hdr(host)
use_backend test.domain.com_ipv4 if ACL1
use_backend service1.domain.com_ipvANY if ACL2
use_backend service2.domain.com-seafhttp_ipvANY if ACL3
frontend http-to-https
bind X.X.X.X:80 name X.X.X.X:80
mode http
log global
option http-keep-alive
timeout client 30000
http-request redirect scheme https
backend test.domain.com_ipv4
mode http
id 10100
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4# usesrc clientip
option httpchk GET /
server testvm-server01 10.100.10.101:54080 id 10101 check inter 1000
backend service1.domain.com_ipvANY
mode http
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server seafile-vm-01 10.100.10.103:8000 id 101 check inter 1000
backend service2.domain.com-seafhttp_ipvANY
mode http
id 104
log global
timeout connect 30000
timeout server 30000
retries 3
server seafile-vm-02 10.100.10.103:8082 id 103 check inter 1000
For further details: https://www.reddit.com/r/PFSENSE/comments/9kezl3/pfsense_haproxy_reverse_proxy_with_multiple/?st=jmruoa9r&sh=26d24791
TLDR: I misconfigured my Action Table and had the wrong health check in place.
Greetings,
Bioneye
Related
I stored port number in client side path and i want to use it in webserver in frontend section.
How can i get path content in Haproxy? i dont want to use if command
My code is:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend http80
bind *:2095
mode http
use_backend webs1 if { path -m beg -i /1023 }
use_backend webs2 if { path -m beg -i /5449 }
use_backend webs3 if { path -m beg -i /4855 }
backend webs1
mode http
server webserver1 ip:1023
backend webs2
mode http
server webserver1 ip:5449
backend webs3
mode http
server webserver1 ip:4855
thanks
You can try to set the dst port via http-request set-dst-port
Here a untested example, just that you get the idea
backend webs2
http-request set-var(txn.dst-port) %[url,'regsub("\/","",i)']
http-request set-dst-port %[var(txn.dst-port)]
server webserver1 0.0.0.0:0
Here is the documentation for http-request set-dst
I have the following setup in HAProxy
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
retries 2
option dontlognull
timeout connect 10000
timeout server 600000
timeout client 600000
frontend https
bind 5.x.x.x:443
default_backend https
backend https
mode tcp
balance roundrobin
option tcp-check
server traefik 192.168.128.5:9443 check fall 3 rise 2
And it works as expected, the backend server "traefik" is doing the SSL termination of the requests.
The thing is the client source IP I get in the backend server is the HAProxy's IP and I would like to pass the source IP to the backend server.
Is it possible at all? because I tried all the options I saw in internet.
Thanks.
At the end the solution was to use https://www.haproxy.com/blog/haproxy/proxy-protocol/ as it is supported by HAProxy and traefik.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
retries 2
option dontlognull
timeout connect 10000
timeout server 600000
timeout client 600000
frontend https
bind 5.x.x.x:443
default_backend https
backend https
mode tcp
balance roundrobin
option tcp-check
server traefik 192.168.128.5:9443 check fall 3 rise 2 send-proxy
And enabling traefik's entrypoint Proxy Protocol as described here: https://docs.traefik.io/configuration/entrypoints/#proxyprotocol
what is the equivalent of Nginx pass_proxy this in HAProxy
location /{
proxy_pass https://WebApplication.azurewebsites.net;
}
I tried to test this configuration but I receive 404 when I point to any backend server with the below configuration without using ACLs on root for example with self signed certificate
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log /dev/log local0
log /dev/log local1 notice
user haproxy
group haproxy
maxconn 16000
stats socket /var/lib/haproxy/stats level admin
tune.bufsize 32768
tune.maxrewrite 1024
tune.ssl.default-dh-param 2048
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
#HAProxy Monitoring Config
#---------------------------------------------------------------------
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
# FrontEnd Configuration
#---------------------------------------------------------------------
frontend fe_http_sne_in
bind *:443 ssl crt /etc/ssl/private/mydomain.pem
option forwardfor
default_backend be_default
backend be_default
mode http
option forwardfor
http-request add-header X-Forwarded-For %[src]
server srv02 www.google.com:443 ssl verify none
I receive 404 when pointing to any backend server tested with bing, google as urls ...
I suggest to use the following config
frontend fe_http_sne_in
bind *:443 ssl crt /etc/ssl/private/mydomain.pem
option forwardfor
use_backend be_sne_insecure if { path_beg /test}
default_backend be_default
backend be_default
...
backend be_sne_insecure
mode http
option forwardfor
http-request replace-header Host .* WebApplication.azurewebsites.net
server srv01 WebApplication.azurewebsites.net:443 ssl verify none
In the Blog post Introduction to HAProxy ACLs is the acl explained
I want to make Lighttpd works with websocket (Socket.IO) and it appears the only way to do so is to install an additionnal module: mod_websocket. I followed these steps but I think I don't get the right /path/to/lighttpd_top_srcdir. I used /usr/lib/lighttpd as I saw all the modules in here when I ls-ed it.
Apparently, I need to reinstall Lighttpd, am I right?
So far, I got
copy mod_websocket files into /usr/lib/lighttpd
cp src/mod_websocket*.{h,c} /usr/lib/lighttpd/src
cp: target « /usr/lib/lighttpd/src » is not a directory
I need to do this because the error I get when trying to make websockets work is the following: WebSocket connection to 'ws://<myURL>/socket.io/1/websocket/agXkznae1gmlDTutzJyk' failed: Unrecognized frame opcode: 5 (I use Google Chrome v33.0.1750.154).
Is there another way to make websockets work with Lighttpd or do I need to change webserver?
Many thanks!
I resolved my problem!
I used HAProxy instead of Lighttpd mod_proxy as specified in this question
Here is my conf file (amend <...> per your configuration):
# this config needs haproxy-1.1.28 or haproxy-1.2.1
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
uid 99
gid 99
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option http-use-proxy-header
option redispatch
option http-server-close
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend public
bind *:80
acl is_example hdr_end(host) -i <URL.toyourwebsite.com>
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket path_beg -i /websockets
use_backend ws if is_websocket is_example
default_backend www
backend ws
balance roundrobin
option forwardfor # This sets X-Forwarded-For
timeout queue 5000
timeout server 86400000
timeout connect 86400000
server apiserver localhost:<PORT> weight 1 maxconn 1024 check
And I made Lighttpd listened to the 8080 port (otherwise HAProxy wouldn't start).
I have Ubuntu 12.04LTS running. My webserver is Tomcat 7.0.42 and I use HAProxy as proxy server. My application is a servlet application which uses websockets.
Sometime when I request my page I get "502 Bad Gateway" error on some resources not on all, but on some. I think that this has something to do with my HAProxy configuration, which is the following:
global
maxconn 4096 # Total Max Connections. This is dependent on ulimit
nbproc 1
defaults
mode http
option http-server-close
option httpclose
# option redispatch
no option checkcache # test against 502 error
frontend all 0.0.0.0:80
timeout client 86400000
default_backend www_backend
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend socket_backend if is_websocket
backend www_backend
balance roundrobin
option forwardfor # This sets X-Forwarded-For
timeout server 30000
timeout connect 4000
server apiserver localhost:8080 weight 1 maxconn 1024 check
backend socket_backend
balance roundrobin
option forwardfor # This sets X-Forwarded-For
timeout queue 5000
timeout server 86400000
timeout connect 86400000
server apiserver localhost:8080 weight 1 maxconn 1024 check
What do I have to change to prevent the 502 error?
First, enable haproxy logging. It will simply tell you why it is giving the 502's. My guess is that the backend "localhost:8080" is simply not able to keep up or is not able to get a connection within 4000ms "timeout connect 4000".
You may have exceeded some of the default limits in HAProxy. Try adding the following to global section:
tune.maxrewrite 4096
tune.http.maxhdr 202
Your should replace your defaults with these ones :
# Set balance mode
balance random
# Set http mode
mode http
# Set http keep alive mode (https://cbonte.github.io/haproxy-dconv/2.3/configuration.html#4)
option http-keep-alive
# Set http log format
option httplog
# Dont log empty line
option dontlognull
# Dissociate client from dead server
option redispatch
# Insert X-Forwarded-For header
option forwardfor
Don't use http-server-close, it is likely the cause of your problems.
Keep-alive will have a connection with client and server at both side.
It is working fine with websockets as well.
And if you enable the check on the server you need to as well configure it with something like this :
# Enable http check
option httpchk
# Use server configuration
http-check connect default
# Use HEAD on / with HTTP/1.1 protocol for Host example.com
http-check send meth HEAD uri / ver HTTP/1.1 hdr Host example.com
# Expect status 200 to 399
http-check expect status 200-399