Hoping someone can help me.
I'm using NodeJS v0.8.16 Socket.ID v0.9.13 and HAProxy 1.5-dev17.
My setup is on Amazon AWS using a VPC, HAProxy on a public facing instance and NodeJS on a separate instance which normally is not publicly accessible. I do have an IP address on it for testing.
I have a test setup which logs into the NodeJS server, then opens an authenticated websocket through Socket.IO, session details are saved in Redis to share between them. Once the websocket is connected successfully a request is made through NodeJS which prompts an event to be emitted back to the client.
This flow works correctly when the test references the NodeJS instance directly bypassing HAProxy. When it does route through HAProxy I get the errors from Socket.IO
"reserved fields must be empty"
and
"no handler for opcode " referencing a random opcode
From what I can see, there is an initial byte that is parsed by Socket.IO using bitmasks to work out what the request is for. After routing through HAProxy yhis value is now not an expected one and throws these errors.
My HAProxy configuration is from another StackOverflow question HAProxy + WebSocket Disconnection
global
maxconn 4096 # Total Max Connections. This is dependent on ulimit
nbproc 2
defaults
mode http
retries 3
option redispatch
option http-server-close
frontend all 0.0.0.0:3000
timeout client 5000
default_backend www_backend
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend socket_backend if is_websocket
backend www_backend
balance roundrobin
option forwardfor # This sets X-Forwarded-For
timeout server 5000
timeout connect 4000
server server 10.0.0.214:3000 weight 1 maxconn 1024 check
backend socket_backend
balance roundrobin
option forwardfor # This sets X-Forwarded-For
timeout queue 5000
timeout server 5000
timeout connect 5000
timeout tunnel 3600s
timeout http-keep-alive 1s
timeout http-request 15s
server server1 10.0.0.214:3000 weight 1 maxconn 1024 check
I've also tried other various HAProxy configurations but end up with the same result.
Has anyone come across this issue. I'm not sure what I've done incorrectly.
Related
I'm trying to use Haproxy 1.6.3 2015/12/25 and sticky sessions.
I did everything according to haproxy manual, but, unfortunately, checking client browser I see that cookies aren't being added (the balancer has to return me cookies in the response after the first request but it returns me nothing and works as nothing happened (without cookies)). Everything else perfectly works, but cookies don't. I've attached my haproxy.cfg:
global
log /dev/haproxy/log local0
log /dev/haproxy/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
stats enable
stats auth user:pass
stats uri /haproxy_stats
option httpchk HEAD / HTTP/1.0
option redispatch
balance roundrobin
frontend frontend_http
bind *:80
option forwardfor
default_backend backend_http
backend backend_http
option prefer-last-server
cookie mycookies insert indirect nocache
server server1 196.168.0.125:80 check cookie s1
server server2 196.168.0.126:80 check cookie s2
Also my servers (server1, server2) are deployed on IIS and balancer is deployed on Ubuntu 16.04 LTS
change backend configuration :
backend backend_http
option prefer-last-server
cookie mycookies insert indirect nocache
server server1 196.168.0.125:80 check cookie server1
server server2 196.168.0.126:80 check cookie server2
or
backend backend_http
option prefer-last-server
cookie mycookies insert indirect nocache
server s1 196.168.0.125:80 check cookie s1
server s2 196.168.0.126:80 check cookie s2
I have a Node.js server that uses Server-Sent Events (SSE) to allow push notifications to connected web clients. It works beautifully when the browser talks to Node directly.
However, when I place haproxy in the middle, as it must be for production to meet other requirements, the connections are closed and reopened (thanks to SSE's auto-reconnect) every 30 seconds. I've changed and tried everything I know of and can find online in the haproxy configuration.
Most info out there and in their documentation examples deal with sockets, but there is very little about SSE support. Should it support persistent HTTP connections for SSE? If so, what is the trick in to configure it?
My config follows:
global
daemon
# maximum number of concurrent connections
maxconn 4096
# drop privileges after port binding
user nobody
group nogroup
# store pid of process in the file
pidfile /var/run/haproxy.pid
# create this socket for stats
stats socket /var/run/socket-haproxy
defaults
log global
mode http
# disable logging of null connections
option dontlognull
# I've tried all these to no avail
#option http-server-close
#option httpclose
option http-keep-alive
# Add x-forwarded-for header to forward clients IP to app
option forwardfor
# maximum time to wait for a server connection to succeed. Can be as low as few msec if Haproxy and server are on same LAN.
timeout connect 1s
# maximum inactivity time on client side. Recommended to keep it same as server timeout.
timeout client 24d
# maximum time given to server to respond to a request
timeout server 24d
# Long timeout for WebSocket connections.
timeout tunnel 8h
# timeout for keep alive
timeout http-keep-alive 60s
# maximum time to wait for client to send full request. Keep it like 5s for get DoS protection.
timeout http-request 5s
# enable stats web interface. very helpful to see what's happening in haproxy
stats enable
# default refresh time for web interface
stats refresh 30s
# this frontend interface receives the incoming http requests and forwards to https then handles all SSL requests
frontend public
# HTTP
bind :80
# by default, all incoming requests are sent to Node.js
default_backend node_backend
# redirect to the SSE backend if /ionmed/events (eventum #????)
acl req_sse_path path_beg /ionmed/events
use_backend node_sse_backend if req_sse_path
# redirect to the tomcat backend if Time Clock, ViewerJS, Spell Checker, Tomcat Manager, or eScripts (eventum #1039, #1082)
acl req_timeclock_path path_beg /TimeClock/
acl req_manager_path path_beg /manager/
acl req_spelling_path path_beg /jspellEvolution/
acl req_escripts_path path_beg /ionmed/escripts
use_backend tomcat_backend if req_timeclock_path or req_manager_path or req_spelling_path or req_escripts_path
# for displaying HAProxy statistics
acl req_stats path_beg /stats
use_backend stats if req_stats
# node backend, transfer to port 8081
backend node_backend
# Tell the backend that this is a secure connection,
# even though it's getting plain HTTP.
reqadd X-Forwarded-Proto:\ https
server node_server localhost:8081
# node SSE backend, transfer to port 8082
backend node_sse_backend
# Tell the backend that this is a secure connection,
# even though it's getting plain HTTP.
reqadd X-Forwarded-Proto:\ https
server node_sse_server localhost:8082
# tomcat backend, transfer to port 8888
backend tomcat_backend
# Tell the backend that this is a secure connection,
# even though it's getting plain HTTP.
reqadd X-Forwarded-Proto:\ https
server tomcat_server localhost:8888
I have setup HAProxy for EC2 server where i'm running my nodejs two server on port 3005 and 3006. we have setup this for our multiplayer game. we have used socket.io for our realtime event update on client side and server side. HAProxy is working correctly with "balance source" (I have added working copy of my HAProxy Configuration), in source balancer problem is that its goes all event on same server each and ever time. so i have 40 computer setup in my network so all 40 computer event goes to 3005 port. its not changing port when i'm coming next day. I would like to setup TCP connection sticky with TCP mode in haproxy. is there any way to do with balance roundrobin? I have added my current setting files here. we also trying to used cookie but its not working in our case because we have used mode as tcp.
Also we have flash game which used to load flash policy from port 3843.
Here added my haproxy configuration.
global
debug
log 127.0.0.1 local0 # Enable per-instance logging of events and traffic.
log 127.0.0.1 local1 notice # only send important events
nbproc 1
maxconn 65536
pidfile /var/run/haproxy.pid
defaults
log global
srvtimeout 300s
timeout connect 5s
timeout queue 5s
timeout server 1h
timeout tunnel 1h
frontend flash_policy
bind 0.0.0.0:843
timeout client 5s
default_backend nodejs_flashpolicy
frontend wwws
bind 0.0.0.0:3000 ssl crt /home/certificate/final.crt
timeout client 1h
default_backend flashsocket_backend
tcp-request inspect-delay 500ms
tcp-request content accept if HTTP
use_backend flashsocket_backend if !HTTP
backend flashsocket_backend
mode tcp
option log-health-checks
balance source
cookie JSESSIONID insert indirect nocache
server 3006Game serverip:3006 cookie socket1 weight 1 maxconn 32536 check
server 3005Game serverip:3005 cookie socket2 weight 1 maxconn 32536 check
backend nodejs_flashpolicy
server flashpolicy serverip:3843 weight 1 maxconn 65536 check
# Configuration for HAProxy Stats
listen stats :1900
mode http
timeout client 1h
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth alpesh:alpesh
It is possible by using following options in backend:
stick-table type ip size 50k expire 10m
stick on src
So far I have had a single node.js app. running socket.io. As number of users grows, it reaches 100% CPU most of the day, so I decided to split users to multiple node.js processes. I have split my node.js application logic to allow sharding of users on different subdomains. I also extracted session code into token passing via URL, so cookies are not important.
I'd like to use at least 4 cores of my 8-core machine, so I want to run multiple node.js processes, each serving the app on subdomain. In order for all node.js's to be accessible via port 80, I decided to use HAProxy. Setup looks like this:
domain.com -> haproxy -> node on 127.0.0.1:5000
sub1.domain.com -> haproxy -> node on 127.0.0.1:5001
sub2.domain.com -> haproxy -> node on 127.0.0.1:5002
sub3.domain.com -> haproxy -> node on 127.0.0.1:5003
Now everything works, but reqular part of the application (not using socket.io) is very slow. It's written using Express.js and it works fast when I open the page directly (i.e. not through HAProxy). Also, connecting to socket.io works fast with XHR transport, but for Websocket transport it also takes a long time to establish connection. Once connection is established, it works well and fast.
I have never used HAProxy before, so I probably misconfigured something. Here's my HAProxy config:
global
maxconn 50000
daemon
defaults
mode http
retries 1
contimeout 8000
clitimeout 120000
srvtimeout 120000
frontend http-in
bind *:80
acl is_l1 hdr_end(host) -i sub1.domain.com
acl is_l2 hdr_end(host) -i sub2.domain.com
acl is_l3 hdr_end(host) -i sub3.domain.com
acl is_l0 hdr_end(host) -i domain.com
use_backend b1 if is_l1
use_backend b2 if is_l2
use_backend b3 if is_l3
use_backend b0 if is_l0
default_backend b0
backend b0
balance source
option forwardfor except 127.0.0.1 # stunnel already adds the header
server s1 127.0.0.1:5000
backend b1
balance source
option forwardfor except 127.0.0.1 # stunnel already adds the header
server s2 127.0.0.1:5001
backend b2
balance source
option forwardfor except 127.0.0.1 # stunnel already adds the header
server s2 127.0.0.1:5002
backend b3
balance source
option forwardfor except 127.0.0.1 # stunnel already adds the header
server s2 127.0.0.1:5003
I figured it out. I failed to find this in docs, but global maxconn setting does NOT apply to frontend. Frontend has default of 2000 concurrent connections and everything beyond was queued. Since I have long-lived socket.io connections this created problems.
The solution is to explicitly set maxconn in frontend section.
We're planning to add a second varnish server to our infraestructure.
What it's the better method to balance the traffic throw the two servers? I think we can use haproxy in front of the two servers, but how to configure it to load balance the traffic between the 2 varnish? The ideal solution is that if one varnish is down all the traffic goes to the other.
Edit: The ideal behaviour is an active/active conf, with 50% load each and if one goes down haproxy sends 100% load to the other.
Another option would be to put both varnish instances in the same backend, but balance requests by uri or parameter.
This will allow you to have both active in the same backend, and still maintain a high cache-hit ratio since the same uri will be always balanced to the same varnish cache (as long as it is available). Balace uri also takes an optional parameter of length to use when hashing.
Heres a quick example:
frontend http-in
acl my_acl
use_backend varnish if my_acl
backend varnish1
mode http
balance uri
option httpchk GET /check.txt HTTP/1.0
server varnish1 192.168.222.51:6081 check inter 2000
server varnish2 192.168.222.52:6081 check inter 2000
The idea I have come up is use two backends and have in each backed use the other server as backup so when a server is down all the request goes to the alive server.
frontend http-in
acl my_acl <whaever acl to split the traffic>
use_backend varnish2 if my_acl
default_backend varnish1
backend varnish1
mode http
option httpchk GET /check.txt HTTP/1.0
server varnish1 192.168.222.51:6081 check inter 2000
server varnish2 192.168.222.52:6081 check inter 2000 backup
backend varnish2
mode http
option httpchk GET /check.txt HTTP/1.0
server varnish2 192.168.222.51:6081 check inter 2000
server varnish1 192.168.222.52:6081 check inter 2000 backup