nginx nodejs faye performance issue - node.js

Question

I am working on nginx load balancing for multiple Faye Chat Servers.
I am able to see significant performance on normal http requests. But, websocket connection performance is very low when comparing the results without nginx.
Here is my nginx cofiguration.
upstream backend {
server 127.0.0.1:4000;
server 127.0.0.1:4002;
server 127.0.0.1:4003;
server 127.0.0.1:4004;
}
server {
listen 4001;
root /var/www/html/laughing-robot;
index index.html index.htm;
server_name backend;
location /faye {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
location / {
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_redirect off;
proxy_pass http://backend;
}
}
I am using websocket-bench for benchmarking Faye connections(websocket).
Here is the result with out nginx:
user#machine:/etc/nginx/sites-enabled$ websocket-bench -a 2000 -c 500 -t faye http://127.0.0.1:4000/faye
Launch bench with 2000 total connection, 500 concurent connection
0 message(s) send by client
1 worker(s)
WS server : faye
trying : 500 ...
trying : 1000 ...
trying : 1500 ...
trying : 2000 ...
#### steps report ####
┌────────┬─────────────┬────────┬──────────────┐
│ Number │ Connections │ Errors │ Duration(ms) │
├────────┼─────────────┼────────┼──────────────┤
│ 500 │ 500 │ 0 │ 2488 │
├────────┼─────────────┼────────┼──────────────┤
│ 1000 │ 500 │ 0 │ 2830 │
├────────┼─────────────┼────────┼──────────────┤
│ 1500 │ 500 │ 0 │ 2769 │
├────────┼─────────────┼────────┼──────────────┤
│ 2000 │ 500 │ 0 │ 2144 │
└────────┴─────────────┴────────┴──────────────┘
#### total report ####
┌────────┬─────────────┬────────┬──────────────┬──────────────┬──────────────┐
│ Number │ Connections │ Errors │ Message Send │ Message Fail │ Duration(ms) │
├────────┼─────────────┼────────┼──────────────┼──────────────┼──────────────┤
│ 2000 │ 2000 │ 0 │ 0 │ 0 │ 5150 │
└────────┴─────────────┴────────┴──────────────┴──────────────┴──────────────┘
Total duration is under 6000 ms.
Here is the results with nginx load balancer:
user#machine:/etc/nginx/sites-enabled$ websocket-bench -a 2000 -c 500 -t faye http://127.0.0.1:4001/faye
Launch bench with 2000 total connection, 500 concurent connection
0 message(s) send by client
1 worker(s)
WS server : faye
trying : 500 ...
trying : 1000 ...
trying : 1500 ...
trying : 2000 ...
#### steps report ####
┌────────┬─────────────┬────────┬──────────────┐
│ Number │ Connections │ Errors │ Duration(ms) │
├────────┼─────────────┼────────┼──────────────┤
│ 500 │ 500 │ 0 │ 6452 │
├────────┼─────────────┼────────┼──────────────┤
│ 1000 │ 500 │ 0 │ 9394 │
├────────┼─────────────┼────────┼──────────────┤
│ 1500 │ 500 │ 0 │ 12772 │
├────────┼─────────────┼────────┼──────────────┤
│ 2000 │ 500 │ 0 │ 16163 │
└────────┴─────────────┴────────┴──────────────┘
#### total report ####
┌────────┬─────────────┬────────┬──────────────┬──────────────┬──────────────┐
│ Number │ Connections │ Errors │ Message Send │ Message Fail │ Duration(ms) │
├────────┼─────────────┼────────┼──────────────┼──────────────┼──────────────┤
│ 2000 │ 2000 │ 0 │ 0 │ 0 │ 19173 │
└────────┴─────────────┴────────┴──────────────┴──────────────┴──────────────┘
For total 2000 connections & 500 concurrent connections, nginx loadbalancer's performance is very low.
I have also configured nofile & file-max:
/etc/security/limits.conf
* soft nofile 2048
* hard nofile 65536
/etc/sysctl.conf
fs.file-max = 100000
On Fedora, I am getting a lots of connection refused error on /var/log/nginx/error.log. But on Ubuntu 13.04 no errors.
It would be greatly appreciable, if someone able to put me towards right direction.
Thanks!

Answer 1

do you know https://github.com/observing/balancerbattle ?
If you receive no errors on ubuntu, what is the performance there?
(hopefully the systems are comparable in performance)
Anyhow, take a look at the kernel tuning part, you could also try the nginx.conf they used in their tests and see if that yeilds the same results.
Also you should if possible try multiple load tests. The tests you did are on the server which runs nginx, how do the tests compare from an acutal external ip over your domain?
I would advice to run the load test on a dev machine and not on the actual server.
Also what does top say cpu/memory wise for both nginx and the node processes?
Is nginx maby starving one of your processes and/or the test itself?
Websockets are more stable over ssl, it might be worth wile to test that also.
They used thor for the tests, does that give you the same results?
https://github.com/observing/balancerbattle/blob/master/results/messaging/HAproxy/1k.md