Folks,
My environment is Ubuntu 12.04.
Here is a pseudo-code for my TCP server application that is listening for a connection:
while(true) {
int hConn = accept(hMain, NULL, NULL);
string s = readClient(hConn);
if (s == "quit") {
close(hConn);
}
}
While my server is running, I telnet to localhost at port nnnn:
$ telnet localhost nnnn
quit
Connection closed by foreign host.
$
When the server receives "quit," it closes the client connection. This causes the telnet client to quit with an appropriate message.
So far so good.
However, when I run netstat, I can still see that the client connection is still alive.
It takes a few seconds for the connection to disappear.
This happens even if I force quit my server app.
If I run my server app once again, I get an error that port "nnnn" is still in use.
I have to wait for a few seconds before I can run my server app once again.
Is there something that I am missing? Is there a way to fix this behavior?
Note that I am indeed closing socket hMain when quitting the server although this is not shown in the above pseudo-code.
Thank you in advance for your help.
Regards,
Peter
You need to be aware of the TIME_WAIT state, which provides that TCP connections which have been closed hang around for a couple of minutes for TCP security/integrity reasons.
The problem with restarting your server can be overcome via the SO_REUSEADDR option.
Related
I recently encountered an TCP related issue and hope someone could shed some light on it.
I have an application1 in container1/pod1 that is connected to a server (client_ip:12345 <-> server_ip:443). After running for a while, the server decided to close this connection, so it sent FIN to the client and the client sends ACK back to the server (saw these two packets in the pcap). This sould leave the client in CLOSE_WAIT and the server in FIN_WAIT2.
In this situation, the client should call close() and send FIN back to the server. But I've found that the application lack close() in its code, so in theory, the client should be stuck in CLOSE_WAIT and the server will be in FIN_WAIT2 until FIN_WAIT2 timeout. The port 12345 on the client side shouldn't be reused by any other new connections.
However, it seems somehow the client_ip:12345 <-> server_ip:443 socket state on the client side was no longer in CLOSE_WAIT state (become fully closed and available), so when another application2 in container2 was up, it randomly pick the same port 12345 (the kernel assign the ephemeral source port from the range) to connect to the server's port 443. Because the server side were still in FIN_WAIT2, so the connection couldn't be established, and thus the service got interrupted until FIN_WAIT2 state timeout (300 secs).
I understand I should fix the application code by adding close(). However, I'm curious about what could make the CLOSE_WAIT state disappear/reset on the client side and let another application be able to pick the same 12345 port to connect to the server?
I found a F5 Bug mentioned a similar situation: "Client side connection has been fully closed. This may occur if a client SSL profile is in use and an 'Encrypted Alert' has been received."
https://cdn.f5.com/product/bugtracker/ID812693.html
I'm wondering if there are any other possibilities that could cause FIN_WAIT2 on one side and fully closed on the other side (not in CLOSE_WAIT)?
For example, the process using this socket was killed? But AFAIK, after killing that process, the socket file descriptor should be closed and a FIN should still be sent by TCP?
I hope someone could shed some light on it!
So I am working on a basic client/server school project. The server is running on port 10'000. The problem is whenever I manually close the server (CTRL+C) the local TCP connection on port 10'000 stays alive for a minute or so. It gets into a "TIME_WAIT" state. Is there any way I can kill the connection right away and start using it directly? Linux btw.
Thanks!
I hope that undestand you correctly. Try fuser -k 10000/tcp it should help
I think you can use setsockopt() function with SO_REUSEADDR option. It should allow you to bind to the same address and port of application which you have just terminated.
Here is example of using SO_REUSEADDR:
How do I use setsockopt(SO_REUSEADDR)?
Consider this small server for node.js
var net = require ('net');
var server = net.createServer(function (socket) {
console.log("Connection detected");
socket.on('end', function() {
console.log('server disconnected');
});
socket.write("Hello World");
socket.end();
});
server.listen("8888");
When I test the server with Chrome on my Macbook Pro, I get three times the "Connection detected" message in the console.
I know one is for well connecting, another for the favicon, but what's the third one all about?
I tested it with Firefox and wget (which is a Linux command line program), as well as telnet to do the deep investigation. Surprisingly, all of these don't make any extra connection (obviously they don't even try to download the favicon). So I fired up Wireshark and captured a session, and quickly discovered that Chorme systematically makes useless connection, ie it just connects (SYN, SYN-ACK, ACK) and then closes the connection (RST, ACK) without sending anything.
Just a quick googlin and I found this bug report (excerpt):
I suspect the "empty" TCP connections are
backup TCP connections,
IPv4/IPv6 parallel connections, or
TCP pre-connections,
A backup TCP connection is made only if the original TCP connection is
not set up within 250 milliseconds. IPv4/IPv6 parallel connections
are made only if the server has both IPv6 and IPv6 addresses and the
IPv6 connection is not set up within 300 milliseconds. Since you're
testing a local server at localhost:8080, you should be able to
connect to it quickly, so I suspect you are seeing TCP
pre-connections.
To verify if the "empty" TCP connections are TCP pre-connections, open
the "wrench" menu > Settings > Under the Hood > Privacy, and clear the
"Predict network actions to improve page load performance" check box.
Shut down and restart Chrome. Are the "empty" TCP connections gone?
For further reference, see the linked thread, which explains more in-depth what backup, parallel and pre-connections are and if/why this is a good optimization.
I am doing coding in linux architecture.
I have question regarding socket server and client.
I have made one sample code in which server continue to accept the connection and client is connected to server.
if somehow someone has remove the network cable so i am disconnecting client (client socket disconnected from PC) while in server side connection is still alive because i am not able to notify that client is disconnected because network is unplugged.
How can i know that client got disconnected ?
Thanks,
Neel
You need to either configure keepalive on the socket or send an application level heartbeat message, otherwise the listening end will wait indefinitely for packets to arrive. If you control the protocol, the application level heartbeat may be easier. As a plus side, either solution will help keep the connection alive across NAT gateways in the network.
See this answer: Is TCP Keepalive the only mechanism to determine a broken link?
Also see this Linux documentation: http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/#programming
SIGPIPE for local sockets and eof on read for every socket type.
I am seeing weird behavior on Linux where I am seeing that remote end and local end are both showing same IP and port combination. Following is the netstat output
netstat -anp | grep 6102
tcp 0 0 139.185.44.123:61020 0.0.0.0:* LISTEN 3361/a.out
tcp 0 0 139.185.44.123:61021 139.185.44.123:61021 ESTABLISHED 3361/a.out
Can anyone tell me if this is even possible ? If yes, then what could be the scenario ?
A connection is identified by a 4-tuple ((source ip, source port), (target ip, target port)), and the source and target ports could conceivably be the same without any issues. This connection could even be established by one process, which would lead to the output you're seeing.
However, I just got bitten by a nasty bug, where a client socket would try to connect to a server socket with a port number in the ephemeral port range on the same machine. The connection operation would be retried operation until it succeeded.
The retry feature was the issue: if the server application wasn't running AND the source port that got picked at random was the same as the target port (which is possible because the target port was in the ephemeral range), the client socket would CONNECT TO ITSELF (which was wreaking havoc on the internal logic of the client application, you can imagine).
Since the client was performing retries as quickly as possible, the 1 in 30.000 odds that this can happen were hit quickly enough.
The following Python script reproduces it:
import socket
host = 'localhost'
port = 35911
ctr = 0
while True:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(5)
ctr += 1
s.connect((host, port))
print "Connected to self after", ctr, "tries"
break
except socket.error, e:
print e
# Retry
Here is an imaginable scenario. The caller of connect could call bind before calling connect. The caller of connect could cooperate with the caller of listen, and intentionally bind to a port number 1 higher than the listening port number.
If the kernel proceeds to reuse the socket number for the listener, I'd call that a kernel bug.
When multi-threaded server software accepts connection, it usually creates another socket, which communicates with newly connected client in separate thread, while original server socket is still listening for new clients in original thread. In such cases ports of both sockets are equal. So, there's no any problem.
It's a slightly odd case, called a TCP "active/active open". A socket has been opened, bound to a local port and then connected to itself, by using connect() with its own address as the destination.
Nothing weird about that. It has a 1 in 63k chance of happening. What you won't see is * two* such ESTABLISHED* connections: that's impossible by the rules of TCP.