HAProxy + NodeJS gets stuck on TCP Retransmission - node.js

I have a HAProxy + NodeJS + Rails Setup, I use the NodeJS Server for file upload purposes.
The problem I'm facing is that if I'm uploading through haproxy to nodejs and a "TCP (Fast) Retransmission" occurs because of a lost packet the TX rate on the client drops to zero for about 5-10 secs and gets flooded with TCP Retransmissions.
This does not occur if I upload to NodeJS directly (TCP Retransmission happens too but it doesn't get stuck with dozens of retransmission attempts).
My test setup is a simple HTML4 FORM (method POST) with a single file input field.
The NodeJS Server only reads the incoming data and does nothing else.
I've tested this on multiple machines, networks, browsers, always the same issue.
Here's a TCP Traffic Dump from the client while uploading a file:
.....
TCP 1506 [TCP segment of a reassembled PDU]
>> everything is uploading fine until:
TCP 1506 [TCP Fast Retransmission] [TCP segment of a reassembled PDU]
TCP 66 [TCP Dup ACK 7392#1] 63265 > http [ACK] Seq=4844161 Ack=1 Win=524280 Len=0 TSval=657047088 TSecr=79373730
TCP 1506 [TCP Retransmission] [TCP segment of a reassembled PDU]
>> the last message is repeated about 50 times for >>5-10 secs<< (TX drops to 0 on client, RX drops to 0 on server)
TCP 1506 [TCP segment of a reassembled PDU]
>> upload continues until the next TCP Fast Retransmission and the same thing happens again
The haproxy.conf (haproxy v1.4.18 stable) is the following:
global
log 127.0.0.1 local1 debug
maxconn 4096 # Total Max Connections. This is dependent on ulimit
nbproc 2
defaults
log global
mode http
option httplog
option tcplog
frontend http-in
bind *:80
timeout client 6000
acl is_websocket path_beg /node/
use_backend node_backend if is_websocket
default_backend app_backend
# Rails Server (via nginx+passenger)
backend app_backend
option httpclose
option forwardfor
timeout server 30000
timeout connect 4000
server app1 127.0.0.1:3000
# node.js
backend node_backend
reqrep ^([^\ ]*)\ /node/(.*) \1\ /\2
option httpclose
option forwardfor
timeout queue 5000
timeout server 6000
timeout connect 5000
server node1 127.0.0.1:3200 weight 1 maxconn 4096
Thanks for reading! :)
Simon

Try setting "timeout http-request" to 6 seconds globally. It can typically be too low to pickup re-transmits and while it won't explain the cause it might solve your problem.

Try using https://github.com/nodejitsu/node-http-proxy. I am not sure if it will fit in your overall architecture requirement but it would be worth a try.

Related

Unable to connect to local host from 127.0.0.1, localhost and not even with my public ip

I have my Windows server 2012 which is active on production and running 2 websites of .NET. Now I want to run my wordpress site I had configured everything and my wordpress site was working fine before but all of sudden now am unable to connect to local host and even wp admin dashboard is not appearing so I deleted all that stuff uninstalled MYSQL connector MYSQL and web platform installer too. Even now I'm facing the same problem.
Whenever I try to connect 127.0.0.1 /Localhost I get the same message for both "This site can't be
reached" and if I try to connect with my public ip it says "HTTP Error 404. The requested resource is not found."
My netstat results are mentioned below:
C:\Users\Administrator>netsh http show iplisten
IP addresses present in the IP listen list:
173.208.205.34
173.208.205.35
173.208.205.36
C:\Users\Administrator>netstat -ano
Active Connections
1. Proto Local Address Foreign Address State
PID TCP 0.0.0.0:135 0.0.0.0:0
LISTENING
1192 TCP 0.0.0.0:180 0.0.0.0:0
LISTENING 1388 TCP 0.0.0.0:445 0.0.0.0:0
LISTENING 4 TCP 0.0.0.0:1433 0.0.0.0:0
LISTENING 2812 TCP 0.0.0.0:1443 0.0.0.0:0
LISTENING 1388 TCP 173.208.205.34:80 0.0.0.0:0
LISTENING 4 TCP 173.208.205.34:139 0.0.0.0:0
LISTENING 4 TCP 173.208.205.34:443 0.0.0.0:0
LISTENING 4 TCP 173.208.205.34:443
160.153.147.141:35160 TIME_WAIT 0
TCP 173.208.205.34:1433 122.176.28.110:2048 ESTABLISHED 28
Additionally, I have checked the etc/hosts file it have 127.0.0.1 localhost uncommented there.
I have also disabled the firewall that make no change.
Can anyone tell what is wrong with this ?
I notice that there is no 0.0.0.0:80 in the IP listen list. Does your site bind to localhost:80?
The correct IP address in list should include
0.0.0.0:80 (ipv4) and [::]:80 (ipv6)
I think you can add 127.0.0.1 to IP listen list.
netsh http add iplisten ipaddress=127.0.0.1
Then check whether it is in list.

Cannot reach services exposed by docker containers on Ubuntu 18.04

I've been struggling with a strange problem on Ubuntu 18.04. I cannot reach services exposed by containers. I will show you on an example with nginx.
Starting the container:
sudo docker run -it --rm -d -p 8080:80 --name web nginx
docker ps shows:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f09c71db299a nginx "/docker-entrypoint.…" 10 minutes ago Up 10 minutes 0.0.0.0:8080->80/tcp web
listening is on 0.0.0.0:8080 as expected
But curl throws "Connection reset by peer":
$ curl -4 -i -v localhost:8080
* Rebuilt URL to: localhost:8080/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.58.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* stopped the pause stream!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
I used tshark to inspect a network traffic:
$ sudo tshark -i any
66 7.442606878 127.0.0.1 → 127.0.0.1 TCP 68 8080 → 47430 [ACK] Seq=1 Ack=79 Win=65408 Len=0 TSval=4125875840 TSecr=4125875840
67 7.442679088 172.16.100.2 → 172.16.100.1 TCP 56 80 → 37906 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
68 7.442784223 127.0.0.1 → 127.0.0.1 TCP 68 8080 → 47430 [RST, ACK] Seq=1 Ack=79 Win=65536 Len=0 TSval=4125875840 TSecr=4125875840
I see RST within the container(?). I have never had such an issue and I'm a bit lost how to solve it. Can someone help me out?
UPDATE: I used docker inspect f09c71db299a and it shows that:
"Gateway": "172.16.100.1"
"IPAddress": "172.16.100.2"
172.16.100.1 it's my docker0 IP address. It looks it rejects traffic from the container, right?
UPDATE 2: According to NightsWatch's suggestion I checked if the host accepts connection on the 8080. Telnet says:
~$ telnet localhost 8080
Trying ::1...
Connected to localhost.
Escape character is '^]'.
So it looks port is open but the request is blocked :/

nginx tcp SYN packet not receiving ACK

My server setup is nginx directly connects to a node.js server (nginx and node.js are in the same node and nginx is forwarding request to node: 127.0.0.1:8000). The symptom is sometimes there are some 504 logs in nginx log. And node.js log doesn't show any sign of ever receiving the request.
I then enabled tcp log using iptables, which logs all tcp packets to port 8000. After checking the tcp log, it seems that nginx was trying to establish a tcp connection with node.js server, but it never succeeds. It just kept retrying sending SYN packets and then got timed out by nginx. Here's an example (tcp + nginx log):
13:44:44 sp:48103 dp:8000 SYN
13:44:45 sp:48103 dp:8000 SYN
13:44:47 sp:48103 dp:8000 SYN
13:44:51 sp:48103 dp:8000 SYN
13:44:59 sp:48103 dp:8000 SYN
13:45:15 sp:48103 dp:8000 SYN
13:45:44 nginx 504
During the period, the CPU load is pretty light, memory < 50%, incoming request is less than 50 per minute. And other requests were processed normally.
Server is Ubuntu 14.04.2 LTS
Any idea what's going on? Seems like an OS level issue? Thank you in advance.
Check whether something is actually running on TCP port 8000 on your loopback interface. Try some commands like:
lsof -P -n -i tcp:8000
fuser 8000/tcp
ss -4lnt
netstat -4lnt
These should give you some hints on whether something is listening at all. Or it may only be listening on a specific interface/address and not your loopback.

Squid3 suers authenitcate but no data servered

Hi everybody!
I have Squid 3.1 running on Debian 7 from the Wheezy repos.
Users can authenticate and requests sent but pages were not displayed in browser, which sits with white screen loading forever. :(
My log file gives:
Stackoverflow won't let me post this because it thinks I'm posting links, and I lack reputation points.
And my config is:
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443 8443 563 22 # https, snews, ssh
acl SSL_ports port 4443 # rsync
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 81 82 800 8119 # http
acl Safe_ports port 4443 873 # rsync
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 631 # cups
acl purge method PURGE
acl CONNECT method CONNECT
auth_param digest program /usr/lib/squid3/digest_pw_auth -c /etc/squid3/passwords
auth_param digest realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow manager localhost
http_access allow authenticated
http_access deny manager
#http_access deny !Safe_ports
#http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access deny all
http_port 3128 transparent
forwarded_for delete
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
I removed the denys for ports for testing:
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
Maybe this has something to do with IP6 resolution, because my DNS will resolve IP6 addresses, but ping won't ping!
In squid log file: DIRECT/2a01:111:f400:9814::6
And failing ip5 ping.
# ping6 -n ipv6.google.com
PING ipv6.google.com(2607:f8b0:4005:802::1000) 56 data bytes
^C
--- ipv6.google.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms
I'd be so grateful if somebody just gave me some help?
Kind regards,
Sophie
The problem was DNS resolves IP6 address but my VPS server did not have an IP6 address, so squid uses IP6 and won't fall back to IP4.
IP4 can be forced by adding this my squid.conf:
tcp_outgoing_address 123.123.123.123
where 123.123.123.123 is your outgoing IP4 addresss you would like to use.
Squid now works.

TCP: Server sends [RST, ACK] immediately after receiving [SYN] from Client

Host_A tries to send some data to Host_B over TCP. Host_B is listening on port 8181. Both Host_A & Host_B are Linux boxes (Red Hat Enterprise). The TCP layer is implemented using Java NIO API.
Whatever Host_A sends, Host_B is unable to receive. Sniffing the data on wire using WireShark resulted in the following log:
1) Host_A (33253) > Host_B (8181): [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=513413781 TSER=0 WS=7
2) Host_B (8181) > Host_A (33253): [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
The logs show that Host_A sends a [SYN] flag to Host_B in order to establish connection. But instead of [SYN, ACK] Host_B responds with an [RST, ACK] which resets/closes the connection. This behavior is observed always.
I am wondering under what circumstance does a TCP listener sends [RST,ACK] in response to a [SYN]?
RST, ACK means the port is closed. You sure Host_B is listening on the right IP/interface?
Also check your firewall for a -j REJECT --reject-with tcp-reset
It happened to me because I did not set sockaddr_in.sin_family to AF_INET, in the server c++ program.

Resources