Client close the socket first, when there is not much data from server, tcp connection shutdown is okay like:
FIN -->
<-- ACK
<-- FIN, ACK
ACK -->
When the server is busying sending data:
FIN -->
<-- ACK,PSH
RST -->
And the server connection comes to CLOSE_WAIT state and hang on there for a long time.
What's the problem here? client related or server related? This happens on Redhat5 for local sockets.
This article talk about why "RST" is sent, but I do not know why the server connection stuck on CLOSE_WAIT, and do not send a FIN out.
[EDIT]I ignored the most important information, this happens on qemu's slirp network emulation. It seems to be a problem of slirp bug for dealing with close connection.
This means that there is unread data left in in the stream, that the client hasn't finished reading.
You can force it off by using the SO_LINGER option. Here's relevant documentation for Linux (also see the option itself, here), and [here's the matching function2] for Win32.
It's the server side that is remaining open, so it's on the server side you can try disabling SO_LINGER.
It may mean that the server hasn't closed the socket. You can easily tell this by using "lsof" to list the file descriptors open by that process which will include TCP sockets. The fix is to have the process always close the socket when it's finished (even in error cases etc)
This a known defect for qemu.
Related
I recently encountered an TCP related issue and hope someone could shed some light on it.
I have an application1 in container1/pod1 that is connected to a server (client_ip:12345 <-> server_ip:443). After running for a while, the server decided to close this connection, so it sent FIN to the client and the client sends ACK back to the server (saw these two packets in the pcap). This sould leave the client in CLOSE_WAIT and the server in FIN_WAIT2.
In this situation, the client should call close() and send FIN back to the server. But I've found that the application lack close() in its code, so in theory, the client should be stuck in CLOSE_WAIT and the server will be in FIN_WAIT2 until FIN_WAIT2 timeout. The port 12345 on the client side shouldn't be reused by any other new connections.
However, it seems somehow the client_ip:12345 <-> server_ip:443 socket state on the client side was no longer in CLOSE_WAIT state (become fully closed and available), so when another application2 in container2 was up, it randomly pick the same port 12345 (the kernel assign the ephemeral source port from the range) to connect to the server's port 443. Because the server side were still in FIN_WAIT2, so the connection couldn't be established, and thus the service got interrupted until FIN_WAIT2 state timeout (300 secs).
I understand I should fix the application code by adding close(). However, I'm curious about what could make the CLOSE_WAIT state disappear/reset on the client side and let another application be able to pick the same 12345 port to connect to the server?
I found a F5 Bug mentioned a similar situation: "Client side connection has been fully closed. This may occur if a client SSL profile is in use and an 'Encrypted Alert' has been received."
https://cdn.f5.com/product/bugtracker/ID812693.html
I'm wondering if there are any other possibilities that could cause FIN_WAIT2 on one side and fully closed on the other side (not in CLOSE_WAIT)?
For example, the process using this socket was killed? But AFAIK, after killing that process, the socket file descriptor should be closed and a FIN should still be sent by TCP?
I hope someone could shed some light on it!
I have a server running on linux(kernel 3.0.35) and a client running on Windows 7. Every time the client connects to the server a TCP retransmission occurs:
Here is the traffic from wireshark:
http://oi39.tinypic.com/ngsnyb.jpg
What I think happens is this:
The client(192.168.1.100) connects to the server(192.168.1.103), the connection is successful. At some point the client decides to close the connection( FIN,ACK), the server doesn't ACK the FIN.
Then the client starts a new connection, that connection is ACK, and is successful. In the meantime the Windows kernel continues to retransmit the FIN,ACK packet, and finally decides to do a reset.
At the moment the second connection is established I don't receive the data that the client is sending(the packet with 16 bytes length) at server side, I receive these bytes only after the RST packet.
On the server side I'm using the poll() function to check for POLLIN events, and I'm not notified of any data until the RST packet arrives.
So does anyone know why this situation is happening?
Your data bytes are not sent on that 52687 connection but rather the following 52690 connection. I'd guess that the server app is accepting only one connection at a time (the kernel will accept them in advance and just hold the data) and thus not seeing data from the second connection until the first connection is dead and it moves on to the next.
That doesn't explain why your FIN is not being ACK'd. It should. Perhaps there is something in the kernel that doesn't like open-then-close-with-no-data connections? Maybe some kind of attack mitigation? Firewall rules?
recently i've got some port holding on FIN_WAIT_1 state till two days later. The target port is used by one server process ever and client connect to the server process through this port.
The situation is we stopped the server process, and obviously some client is still connecting with the server at that moment. From my understanding, the server process sends a FIN package to client and waiting for the ACK package back. Unfortunately, that ACK package seems not like coming to server side till two days later.
my question is there any config like timeout to FIN_WAIT_1 status. i walked through the internet searching but found nothing there. Please help tell me if you have any experience with this.
BTW, the server process has already gone while the FIN_WAIT_1 happening to the port.
Thanks in advance
The FIN_WAIT_1 state is waiting for the peer to ACK the FIN that this end has just sent. That outgoing FIN is subject to all the normal TCP retry and timeout processing, so if the other end has completely disappeared and never responds, TCP should time out the connection and reset it as a matter of course. That's why you couldn't find a specific FIN_WAIT_1 timeout: there isn't one, just the normal TCP write timers.
All that should have happened within ten minutes or so.
If the state persists and it causes other problems I don't think you don't have much option but to reboot.
Are you sure it's the same ports stuck in FIN_WAIT? It could be due to a load balancer or NAT device that is dropping the connections after an inactivity timeout and silently discarding any further packets, which is the default behavior on some devices.
Is it possible in Node.JS to "drop" a connection in such a way that
The client never receives a response (200, 404 or otherwise)
The client is never notified that the connection is terminated (never receives connection reset or end of stream)
The server's resources are released (the server should not attempt to maintain the connection in any way)
I am specifically asking about Node.JS HTTP Servers (which are really just complex TCP servers) on Solaris., but if there are cases on other OSes (Windows, Linux) or programming languages (C/C++, Java) that permit this, I am interested.
Why do I want this?
To annoy or slow down (possibly single-threaded) robots such as phpMyAdmin Probe.
I know this is not really something that matters, but these types of questions can better help me learn the boundaries of my programs.
I am aware that the client host is likely to re-transmit the packets of the connection since I am never sending reset.
These are not possible in a generic TCP stack (vs. your own custom TCP stack). The reasons are:
Closing a socket sends a RST
Even if you avoid sending a RST, the client continues to think the connection is open while the server has closed the connection. If the client sends any packet on this connection, the server is going to send a RST.
You may want to explore firewalling these robots and block / rate limit their IP addresses with something like iptables (linux) or the equivalent on solaris.
closing a connection should NOT send an RST. There is a 3 way tear down process.
I am doing coding in linux architecture.
I have question regarding socket server and client.
I have made one sample code in which server continue to accept the connection and client is connected to server.
if somehow someone has remove the network cable so i am disconnecting client (client socket disconnected from PC) while in server side connection is still alive because i am not able to notify that client is disconnected because network is unplugged.
How can i know that client got disconnected ?
Thanks,
Neel
You need to either configure keepalive on the socket or send an application level heartbeat message, otherwise the listening end will wait indefinitely for packets to arrive. If you control the protocol, the application level heartbeat may be easier. As a plus side, either solution will help keep the connection alive across NAT gateways in the network.
See this answer: Is TCP Keepalive the only mechanism to determine a broken link?
Also see this Linux documentation: http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/#programming
SIGPIPE for local sockets and eof on read for every socket type.