From my 2 laptops on LAN (one is 10.0.0.11 and the other is 10.0.0.15). From one laptop, I could create passive connection via listening and everything works fine, the other laptop ACK the communication. But I cannot do the same the other way - making passive listener from the other laptop:
LAP1 = laptop(10.0.0.11), LAP2 = laptop(10.0.0.15)
LAP1
$ tcpdump -i wlp2s0 -t -n src 10.0.0.15 or src 10.0.0.11 > output &
$ nc -nl 4444
hello
hi
^C
LAP2
$nc -n 10.0.0.11 4444
hello
hi
^C
Now everything works and the output is:
IP 10.0.0.15.35410 > 10.0.0.11.4444: Flags [S], seq 3649576842, win 64240, options [mss 1460,sackOK,TS val 1557903826 ecr 0,nop,wscale 7], length 0
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [S.], seq 2552044612, ack 3649576843, win 65160, options [mss 1460,sackOK,TS val 2837963760 ecr 1557903826,nop,wscale 7], length 0
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [S.], seq 2552044612, ack 3649576843, win 65160, options [mss 1460,sackOK,TS val 2837964778 ecr 1557903826,nop,wscale 7], length 0
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [S.], seq 2552044612, ack 3649576843, win 65160, options [mss 1460,sackOK,TS val 2837966794 ecr 1557903826,nop,wscale 7], length 0
IP 10.0.0.15.35410 > 10.0.0.11.4444: Flags [.], ack 1, win 502, options [nop,nop,TS val 1557907412 ecr 2837963760], length 0
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [P.], seq 1:7, ack 1, win 510, options [nop,nop,TS val 2837966936 ecr 1557907412], length 6
IP 10.0.0.15.35410 > 10.0.0.11.4444: Flags [.], ack 7, win 502, options [nop,nop,TS val 1557907421 ecr 2837966936], length 0
IP 10.0.0.15.35410 > 10.0.0.11.4444: Flags [P.], seq 1:4, ack 7, win 502, options [nop,nop,TS val 1557908702 ecr 2837966936], length 3
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [.], ack 4, win 510, options [nop,nop,TS val 2837968227 ecr 1557908702], length 0
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [F.], seq 7, ack 4, win 510, options [nop,nop,TS val 2837971235 ecr 1557908702], length 0
IP 10.0.0.15.35410 > 10.0.0.11.4444: Flags [F.], seq 4, ack 8, win 502, options [nop,nop,TS val 1557911933 ecr 2837971235], length 0
IP 10.0.0.11.4444 > 10.0.0.15.35410: Flags [.], ack 5, win 510, options [nop,nop,TS val 2837971644 ecr 1557911933], length 0
So far so good.
But If I try to initiate the other way around:
LAP2:
$nc -nl 4444
LAP1:
$ tcpdump -i wlp2s0 -t -n src 10.0.0.15 or src 10.0.0.11 > output2 &
$nc -n 10.0.0.15 4444
I do not even get connection, the 10.0.0.11 (LAP1) is only trying to send SEQ many times but the other side do not respond and thus the handshake is not complete:
output of output2:
IP 10.0.0.11.47928 > 10.0.0.15.4444: Flags [S], seq 2268838313, win 64240, options [mss 1460,sackOK,TS val 2837899323 ecr 0,nop,wscale 7], length 0
IP 10.0.0.11.47928 > 10.0.0.15.4444: Flags [S], seq 2268838313, win 64240, options [mss 1460,sackOK,TS val 2837900330 ecr 0,nop,wscale 7], length 0
IP 10.0.0.11.47928 > 10.0.0.15.4444: Flags [S], seq 2268838313, win 64240, options [mss 1460,sackOK,TS val 2837902346 ecr 0,nop,wscale 7], length 0
...
As you can see the other side (10.0.0.15) do not respond if it is passive connection. Were it active one (not listening, just connecting) as in previous case, there is no problem. But if it is listening, it is not responding. What could cause that? How to solve this one-direction connection? Why it connect if active but not if passive?
I have a problem, which I hope somebody can point me to the right direction.
Problem >>>
A) Our external provider (connects via VPN) needs to access "OpenSuse42.3" to specific ports, which
"nmap" or "ncat" tools shows as "filtered" or "refused".
B) No services are listening on these ports.
C) No firewall is running on this server.
D) Security team opened these ports on firewall with evidence that connection get reset by server
"OpenSuse42.3".
Test runs from "10.10.10.2" to "10.10.10.1" (problem server) from provider VPN connection (from my computer)
Example 1 : from "10.10.10.2"
>>> nmap -sT -p1101,3050 10.10.10.1
>>>
PORT STATE SERVICE
1101/tcp filtered pt2-discover
3050/tcp filtered gds_db
Example 2 : from "10.10.10.2"
nc -z -v 10.10.10.1 1101
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.
nc -z -v 10.10.10.1 3050
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection refused.
Example 3: on server "10.10.10.1"
tcpdump -n -i eth0 port 1101 or port 3050 -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:00:28.940582 IP (tos 0x0, ttl 64, id 32383, offset 0, flags [DF], proto TCP (6), length 60)
10.10.10.2.58000 > 10.10.10.1.1101: Flags [S], cksum 0xa3fc (correct), seq 3906215335, win 29200,
options [mss 1460,sackOK,TS val 1388733400 ecr 0,nop,wscale 7], length 0
13:00:28.940662 IP (tos 0x0, ttl 64, id 40440, offset 0, flags [DF], proto TCP (6), length 40)
10.10.10.1.1101 > 10.10.10.2.58000: Flags [R.], cksum 0x347b (correct), seq 0, ack 3906215336,
win 0, length 0
13:00:31.263502 IP (tos 0x0, ttl 64, id 60627, offset 0, flags [DF], proto TCP (6), length 60)
10.10.10.2.40830 > 10.10.10.1.3050: Flags [S], cksum 0x8bc2 (correct), seq 3504308280, win 29200,
options [mss 1460,sackOK,TS val 1388735723 ecr 0,nop,wscale 7], length 0
13:00:31.263569 IP (tos 0x0, ttl 64, id 40888, offset 0, flags [DF], proto TCP (6), length 40)
10.10.10.1.3050 > 10.10.10.2.40830: Flags [R.], cksum 0x2554 (correct), seq 0, ack 3504308281,
win 0, length 0
BUT
As soon as I put something on the server like - "nc -l 1101" or "nc -l 3050"
problem disappears, which probably makes sense. To my knowledge "nmap" tool usually shows port status as "closed" if port is not firewalled and service is not running and "open" if service is running on this port.
Question
Are ports opened or closed ??? What else do I check, because provider keep insisting that ports are closed on "10.10.10.1" and he cannot continue his work.
Please let me knoe if something is unclear in this situation and I will respond.
Appreciate it !!!!
If have set up a pair of virtual ethernet devices veth0 and veth1:
ip link add veth0 type veth peer name veth1
# Bring the interfaces up
sudo ifconfig veth0 up
sudo ifconfig veth1 up
sudo ifconfig veth0 1.1.1.1
sudo ifconfig veth1 1.1.1.2
Inside my application I connect to veth0 using a raw socket. Incoming packets are forwarded to my own TCP/IP stack implementation. Replies from the stack are sent back to the socket.
I also have a simple HTTP server running on my stack. I try to connect to it using the curl:
sudo curl -vvv --interface veth1 1.1.1.1/1/2/10000
* Trying 1.1.1.1...
* Local Interface veth1 is ip 1.1.1.2 using address family 2
* Local port: 0
Now my custom stack receives the SYN, enters the SYN-RECEIVED state and replies a with a SYN-ACK. This is sent down the stack back to the raw socket.
However, it seems that curl is not receiving the SYN-ACK because it keeps retransmitting the original SYN.
According to tcpdump the SYN-ACK does seem to arrive on 1.1.1.2:
$ sudo tcpdump -i veth1 -vv
tcpdump: listening on veth1, link-type EN10MB (Ethernet), capture size 262144 bytes
02:43:41.680087 IP (tos 0x0, ttl 64, id 59135, offset 0, flags [DF], proto TCP (6), length 60)
1.1.1.2.41847 > 1.1.1.1.http: Flags [S], cksum 0x0433 (incorrect -> 0x38a4), seq 446675468, win 29200, options [mss 1460,sackOK,TS val 1266013534 ecr 0,nop,wscale 7], length 0
02:43:41.680345 IP (tos 0x0, ttl 64, id 30106, offset 0, flags [DF], proto TCP (6), length 52)
1.1.1.1.http > 1.1.1.2.41847: Flags [S.], cksum 0x0bbe (correct), seq 697874744, ack 446675469, win 65535, options [mss 1460,wscale 5,nop,sackOK,nop,nop], length 0
02:43:42.690344 IP (tos 0x0, ttl 64, id 59136, offset 0, flags [DF], proto TCP (6), length 60)
1.1.1.2.41847 > 1.1.1.1.http: Flags [S], cksum 0x0433 (incorrect -> 0x34b2), seq 446675468, win 29200, options [mss 1460,sackOK,TS val 1266014544 ecr 0,nop,wscale 7], length 0
02:43:44.706343 IP (tos 0x0, ttl 64, id 59137, offset 0, flags [DF], proto TCP (6), length 60)
1.1.1.2.41847 > 1.1.1.1.http: Flags [S], cksum 0x0433 (incorrect -> 0x2cd2), seq 446675468, win 29200, options [mss 1460,sackOK,TS val 1266016560 ecr 0,nop,wscale 7], length 0
02:43:46.850382 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.1.1.1 tell 1.1.1.2, length 28
02:43:46.850579 ARP, Ethernet (len 6), IPv4 (len 4), Reply 1.1.1.1 is-at 92:c6:e5:d6:03:2f (oui Unknown), length 46
02:43:47.680487 IP (tos 0x0, ttl 64, id 30107, offset 0, flags [DF], proto TCP (6), length 52)
1.1.1.1.http > 1.1.1.2.41847: Flags [S.], cksum 0x0bbe (correct), seq 697874744, ack 446675469, win 65535, options [mss 1460,wscale 5,nop,sackOK,nop,nop], length 0
02:43:48.898343 IP (tos 0x0, ttl 64, id 59138, offset 0, flags [DF], proto TCP (6), length 60)
1.1.1.2.41847 > 1.1.1.1.http: Flags [S], cksum 0x0433 (incorrect -> 0x1c72), seq 446675468, win 29200, options [mss 1460,sackOK,TS val 1266020752 ecr 0,nop,wscale 7], length 0
02:43:57.090346 IP (tos 0x0, ttl 64, id 59139, offset 0, flags [DF], proto TCP (6), length 60)
1.1.1.2.41847 > 1.1.1.1.http: Flags [S], cksum 0x0433 (incorrect -> 0xfc71), seq 446675468, win 29200, options [mss 1460,sackOK,TS val 1266028944 ecr 0,nop,wscale 7], length 0
02:43:59.680648 IP (tos 0x0, ttl 64, id 30108, offset 0, flags [DF], proto TCP (6), length 52)
1.1.1.1.http > 1.1.1.2.41847: Flags [S.], cksum 0x0bbe (correct), seq 697874744, ack 446675469, win 65535, options [mss 1460,wscale 5,nop,sackOK,nop,nop], length 0
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Wireshark screenshot.
Does anyone know why my SYN-ACK doesn't reach the other side of the TCP connection?
The problem was that veth interfaces are using checksum offload by default so the forwarded packets had a bad checksum and were ignored by the kernel. Running 'ethtool --offload IF_NAME rx off tx off' on veth interfaces (the sender and receiver), you may fix it.
We have such server structure: reverse proxy (nginx) -> worker (uwsgi) -> postgresql / memcached. All servers are in local network behind router, with NATed external ip:ports (http/s 80/443 to proxy, and ssh 22 to all servers).
Problem is, that sometimes proxy server netstat reports MILLIONS of SYN_RECV connections. From same IP / same ports. Like that:
nginx ~ # netstat -n | grep 83.238.153.195
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
tcp 0 0 192.168.1.1:80 83.238.153.195:3107 SYN_RECV
[...]
And this is not DDoS, because all IPs affected belongs to our website users. On side note, users says that it's not affecting them. Website is online and working, but... that particular one (from example above) told me that website is down and Firefox can't connect. I've done tcpdump.
19:42:14.826011 IP 83.238.153.195.zephyr-srv > 192.168.1.1.http: Flags [S], seq 1845850583, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:14.826042 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:17.887331 IP 83.238.153.195.zephyr-srv > 192.168.1.1.http: Flags [S], seq 1845850583, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:17.887343 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:19.065497 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:23.918064 IP 83.238.153.195.zephyr-srv > 192.168.1.1.http: Flags [S], seq 1845850583, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:23.918076 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:25.265499 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:37.265501 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:37.758051 IP 83.238.153.195.2107 > 192.168.1.1.http: Flags [S], seq 564208067, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:37.758069 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:40.714360 IP 83.238.153.195.2107 > 192.168.1.1.http: Flags [S], seq 564208067, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:40.714374 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:41.665503 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:46.751073 IP 83.238.153.195.2107 > 192.168.1.1.http: Flags [S], seq 564208067, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:42:46.751087 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:47.665498 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:42:59.865499 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:01.265500 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:13.320382 IP 83.238.153.195.2114 > 192.168.1.1.http: Flags [S], seq 2136055006, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:43:13.320399 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:16.320556 IP 83.238.153.195.2114 > 192.168.1.1.http: Flags [S], seq 2136055006, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:43:16.320569 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:17.665498 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:22.250069 IP 83.238.153.195.2114 > 192.168.1.1.http: Flags [S], seq 2136055006, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:43:22.250080 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:23.665500 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:23.865501 IP 192.168.1.1.http > 83.238.153.195.2107: Flags [S.], seq 3188568660, ack 564208068, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:35.665498 IP 192.168.1.1.http > 83.238.153.195.2114: Flags [S.], seq 3754336171, ack 2136055007, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:37.903038 IP 83.238.153.195.2213 > 192.168.1.1.http: Flags [S], seq 2918118729, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:43:37.903054 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:40.772899 IP 83.238.153.195.2213 > 192.168.1.1.http: Flags [S], seq 2918118729, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:43:40.772912 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:41.865500 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:46.793057 IP 83.238.153.195.2213 > 192.168.1.1.http: Flags [S], seq 2918118729, win 65535, options [mss 1412,nop,wscale 0,nop,nop,sackOK], length 0
19:43:46.793069 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:47.865500 IP 192.168.1.1.http > 83.238.153.195.2213: Flags [S.], seq 4145523337, ack 2918118730, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
19:43:49.465503 IP 192.168.1.1.http > 83.238.153.195.zephyr-srv: Flags [S.], seq 2835837547, ack 1845850584, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
Anyone have some thoughts on that?