I am seeing weird behavior on Linux where I am seeing that remote end and local end are both showing same IP and port combination. Following is the netstat output
netstat -anp | grep 6102
tcp 0 0 139.185.44.123:61020 0.0.0.0:* LISTEN 3361/a.out
tcp 0 0 139.185.44.123:61021 139.185.44.123:61021 ESTABLISHED 3361/a.out
Can anyone tell me if this is even possible ? If yes, then what could be the scenario ?
A connection is identified by a 4-tuple ((source ip, source port), (target ip, target port)), and the source and target ports could conceivably be the same without any issues. This connection could even be established by one process, which would lead to the output you're seeing.
However, I just got bitten by a nasty bug, where a client socket would try to connect to a server socket with a port number in the ephemeral port range on the same machine. The connection operation would be retried operation until it succeeded.
The retry feature was the issue: if the server application wasn't running AND the source port that got picked at random was the same as the target port (which is possible because the target port was in the ephemeral range), the client socket would CONNECT TO ITSELF (which was wreaking havoc on the internal logic of the client application, you can imagine).
Since the client was performing retries as quickly as possible, the 1 in 30.000 odds that this can happen were hit quickly enough.
The following Python script reproduces it:
import socket
host = 'localhost'
port = 35911
ctr = 0
while True:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(5)
ctr += 1
s.connect((host, port))
print "Connected to self after", ctr, "tries"
break
except socket.error, e:
print e
# Retry
Here is an imaginable scenario. The caller of connect could call bind before calling connect. The caller of connect could cooperate with the caller of listen, and intentionally bind to a port number 1 higher than the listening port number.
If the kernel proceeds to reuse the socket number for the listener, I'd call that a kernel bug.
When multi-threaded server software accepts connection, it usually creates another socket, which communicates with newly connected client in separate thread, while original server socket is still listening for new clients in original thread. In such cases ports of both sockets are equal. So, there's no any problem.
It's a slightly odd case, called a TCP "active/active open". A socket has been opened, bound to a local port and then connected to itself, by using connect() with its own address as the destination.
Nothing weird about that. It has a 1 in 63k chance of happening. What you won't see is * two* such ESTABLISHED* connections: that's impossible by the rules of TCP.
Related
My company releases a special TCP stack for special purposes and I'm tasked with implementing RFC793 compliant closing sequence. One of the unit tests has a server working on top of the special TCP stack talking to a normal Linux TCP client, and I'm running into some strange behaviour that I'm not sure whether is caused by programming error on my part or is to be expected.
Before my work started, we used to send a RST packet when the user application calls close(). I've implemented the FIN handshake, but I noticed that in the case of simultaneous TCP termination (FIN_WAIT_1 -> CLOSING -> TIME_WAIT on both ends, see the picture), the standard Linux TCP client cannot connect to the same destination address and port again, with connect() returning with EADDRNOTAVAIL, until after TIME_WAIT passes into CLOSED.
Now, the standard Linux client application sets the option SO_REUSEADDR, binds the socket to port 8888 each time, and connects to destination port 6666. My question is, why does bind() succeed and why does connect() fail? I would have thought SO_REUSEADDR could take over a local TIME_WAIT port, which it did, but what does connect() have against talking to the destination-ip:6666 again?
Is my code doing something it shouldn't or is this expected behaviour?
I can confirm no SYN packet for the failed connect() makes it out of the client machine at all. I've attached a screenshot of the FIN handshake for the above session.
Your previous implementation used RST to end the connection. Receipt of an RST packet immediately removes the connection from the active connection table. That's because there is no further possibility of receiving a valid packet on that connection: the peer has just told your system that that session is not valid.
If you do a proper session termination with FIN, on the other hand, there is the last packet problem: how do you know for sure whether the peer actually received the last acknowledgment you sent to their FIN (this is the definition of TCP's TIME_WAIT state)? And if the peer didn't receive it, they might validly send another copy of the FIN packet which your machine should then re-ACK.
Now, your bind succeeds because you're using SO_REUSEADDR, but you still cannot create a new connection with the exact same ports on both sides because that entry is still in your active connection table (in TIME_WAIT state). The 4-tuple (IP1, port1, IP2, port2) must always be unique.
As #EJP suggested in the comment, it is unusual for the client to specify a port, and there is typically no reason to. I would change your test.
This question already has answers here:
Does the port change when a server accepts a TCP connection?
(3 answers)
Closed 4 years ago.
I understand the basics of how ports work. However, what I don't get is how multiple clients can simultaneously connect to say port 80. I know each client has a unique (for their machine) port. Does the server reply back from an available port to the client, and simply state the reply came from 80? How does this work?
First off, a "port" is just a number. All a "connection to a port" really represents is a packet which has that number specified in its "destination port" header field.
Now, there are two answers to your question, one for stateful protocols and one for stateless protocols.
For a stateless protocol (ie UDP), there is no problem because "connections" don't exist - multiple people can send packets to the same port, and their packets will arrive in whatever sequence. Nobody is ever in the "connected" state.
For a stateful protocol (like TCP), a connection is identified by a 4-tuple consisting of source and destination ports and source and destination IP addresses. So, if two different machines connect to the same port on a third machine, there are two distinct connections because the source IPs differ. If the same machine (or two behind NAT or otherwise sharing the same IP address) connects twice to a single remote end, the connections are differentiated by source port (which is generally a random high-numbered port).
Simply, if I connect to the same web server twice from my client, the two connections will have different source ports from my perspective and destination ports from the web server's. So there is no ambiguity, even though both connections have the same source and destination IP addresses.
Ports are a way to multiplex IP addresses so that different applications can listen on the same IP address/protocol pair. Unless an application defines its own higher-level protocol, there is no way to multiplex a port. If two connections using the same protocol simultaneously have identical source and destination IPs and identical source and destination ports, they must be the same connection.
Important:
I'm sorry to say that the response from "Borealid" is imprecise and somewhat incorrect - firstly there is no relation to statefulness or statelessness to answer this question, and most importantly the definition of the tuple for a socket is incorrect.
First remember below two rules:
Primary key of a socket: A socket is identified by {SRC-IP, SRC-PORT, DEST-IP, DEST-PORT, PROTOCOL} not by {SRC-IP, SRC-PORT, DEST-IP, DEST-PORT} - Protocol is an important part of a socket's definition.
OS Process & Socket mapping: A process can be associated with (can open/can listen to) multiple sockets which might be obvious to many readers.
Example 1: Two clients connecting to same server port means: socket1 {SRC-A, 100, DEST-X,80, TCP} and socket2{SRC-B, 100, DEST-X,80, TCP}. This means host A connects to server X's port 80 and another host B also connects to the same server X to the same port 80. Now, how the server handles these two sockets depends on if the server is single-threaded or multiple-threaded (I'll explain this later). What is important is that one server can listen to multiple sockets simultaneously.
To answer the original question of the post:
Irrespective of stateful or stateless protocols, two clients can connect to the same server port because for each client we can assign a different socket (as the client IP will definitely differ). The same client can also have two sockets connecting to the same server port - since such sockets differ by SRC-PORT. With all fairness, "Borealid" essentially mentioned the same correct answer but the reference to state-less/full was kind of unnecessary/confusing.
To answer the second part of the question on how a server knows which socket to answer. First understand that for a single server process that is listening to the same port, there could be more than one socket (maybe from the same client or from different clients). Now as long as a server knows which request is associated with which socket, it can always respond to the appropriate client using the same socket. Thus a server never needs to open another port in its own node than the original one on which the client initially tried to connect. If any server allocates different server ports after a socket is bound, then in my opinion the server is wasting its resource and it must be needing the client to connect again to the new port assigned.
A bit more for completeness:
Example 2: It's a very interesting question: "can two different processes on a server listen to the same port". If you do not consider protocol as one of the parameters defining sockets then the answer is no. This is so because we can say that in such a case, a single client trying to connect to a server port will not have any mechanism to mention which of the two listening processes the client intends to connect to. This is the same theme asserted by rule (2). However, this is the WRONG answer because 'protocol' is also a part of the socket definition. Thus two processes in the same node can listen to the same port only if they are using different protocols. For example, two unrelated clients (say one is using TCP and another is using UDP) can connect and communicate to the same server node and to the same port but they must be served by two different server processes.
Server Types - single & multiple:
When a server processes listening to a port that means multiple sockets can simultaneously connect and communicate with the same server process. If a server uses only a single child process to serve all the sockets then the server is called single-process/threaded and if the server uses many sub-processes to serve each socket by one sub-process then the server is called a multi-process/threaded server. Note that irrespective of the server's type a server can/should always use the same initial socket to respond back (no need to allocate another server port).
Suggested Books and the rest of the two volumes if you can.
A Note on Parent/Child Process (in response to query/comment of 'Ioan Alexandru Cucu')
Wherever I mentioned any concept in relation to two processes say A and B, consider that they are not related by the parent-child relationship. OS's (especially UNIX) by design allows a child process to inherit all File-descriptors (FD) from parents. Thus all the sockets (in UNIX like OS are also part of FD) that process A listening to can be listened to by many more processes A1, A2, .. as long as they are related by parent-child relation to A. But an independent process B (i.e. having no parent-child relation to A) cannot listen to the same socket. In addition, also note that this rule of disallowing two independent processes to listen to the same socket lies on an OS (or its network libraries), and by far it's obeyed by most OS's. However, one can create own OS which can very well violate this restriction.
TCP / HTTP Listening On Ports: How Can Many Users Share the Same Port
So, what happens when a server listen for incoming connections on a TCP port? For example, let's say you have a web-server on port 80. Let's assume that your computer has the public IP address of 24.14.181.229 and the person that tries to connect to you has IP address 10.1.2.3. This person can connect to you by opening a TCP socket to 24.14.181.229:80. Simple enough.
Intuitively (and wrongly), most people assume that it looks something like this:
Local Computer | Remote Computer
--------------------------------
<local_ip>:80 | <foreign_ip>:80
^^ not actually what happens, but this is the conceptual model a lot of people have in mind.
This is intuitive, because from the standpoint of the client, he has an IP address, and connects to a server at IP:PORT. Since the client connects to port 80, then his port must be 80 too? This is a sensible thing to think, but actually not what happens. If that were to be correct, we could only serve one user per foreign IP address. Once a remote computer connects, then he would hog the port 80 to port 80 connection, and no one else could connect.
Three things must be understood:
1.) On a server, a process is listening on a port. Once it gets a connection, it hands it off to another thread. The communication never hogs the listening port.
2.) Connections are uniquely identified by the OS by the following 5-tuple: (local-IP, local-port, remote-IP, remote-port, protocol). If any element in the tuple is different, then this is a completely independent connection.
3.) When a client connects to a server, it picks a random, unused high-order source port. This way, a single client can have up to ~64k connections to the server for the same destination port.
So, this is really what gets created when a client connects to a server:
Local Computer | Remote Computer | Role
-----------------------------------------------------------
0.0.0.0:80 | <none> | LISTENING
127.0.0.1:80 | 10.1.2.3:<random_port> | ESTABLISHED
Looking at What Actually Happens
First, let's use netstat to see what is happening on this computer. We will use port 500 instead of 80 (because a whole bunch of stuff is happening on port 80 as it is a common port, but functionally it does not make a difference).
netstat -atnp | grep -i ":500 "
As expected, the output is blank. Now let's start a web server:
sudo python3 -m http.server 500
Now, here is the output of running netstat again:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
So now there is one process that is actively listening (State: LISTEN) on port 500. The local address is 0.0.0.0, which is code for "listening for all". An easy mistake to make is to listen on address 127.0.0.1, which will only accept connections from the current computer. So this is not a connection, this just means that a process requested to bind() to port IP, and that process is responsible for handling all connections to that port. This hints to the limitation that there can only be one process per computer listening on a port (there are ways to get around that using multiplexing, but this is a much more complicated topic). If a web-server is listening on port 80, it cannot share that port with other web-servers.
So now, let's connect a user to our machine:
quicknet -m tcp -t localhost:500 -p Test payload.
This is a simple script (https://github.com/grokit/dcore/tree/master/apps/quicknet) that opens a TCP socket, sends the payload ("Test payload." in this case), waits a few seconds and disconnects. Doing netstat again while this is happening displays the following:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.10:500 192.168.1.13:54240 ESTABLISHED -
If you connect with another client and do netstat again, you will see the following:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:500 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.10:500 192.168.1.13:26813 ESTABLISHED -
... that is, the client used another random port for the connection. So there is never confusion between the IP addresses.
Normally, for every connecting client the server forks a child process that communicates with the client (TCP). The parent server hands off to the child process an established socket that communicates back to the client.
When you send the data to a socket from your child server, the TCP stack in the OS creates a packet going back to the client and sets the "from port" to 80.
Multiple clients can connect to the same port (say 80) on the server because on the server side, after creating a socket and binding (setting local IP and port) listen is called on the socket which tells the OS to accept incoming connections.
When a client tries to connect to server on port 80, the accept call is invoked on the server socket. This creates a new socket for the client trying to connect and similarly new sockets will be created for subsequent clients using same port 80.
Words in italics are system calls.
Ref
http://www.scs.stanford.edu/07wi-cs244b/refs/net2.pdf
Reading the following article: 10M concurrent websockets
So, there are 1000 websocket servers listening on ports 10000-11000. When a connection is made to one of these servers, I assume they continue communication from a random established TCP connection with random ports. So, as one IP is used, and there are 64K ports, how can one maintain 10M connections? Are connections identified by IP-Port pairs? Can two different connections from different IPs to same port be established? How does this work under the hood?
When a connection is made to one of these servers, I assume they continue communication from a random established TCP connection with random ports.
Wrong assumption. They communicate with the clients using the same local port number they are listening on.
So, as one IP is used, and there are 64K ports, how can one maintain 10M connections?
Not a problem.
Are connections identified by IP-Port pairs?
Yes.
Can two different connections from different IPs to same port be established?
Yes.
How does this work under the hood?
See above. IP:port pairs. You answered your own question.
Sorry for totally changing my answer.
Linux can easily support millions of open sockets if the machine has enough memory and processing power. The TCP/IP stack allows this because the socket the OS targets for a given TCP packet is determined by the source and destination IP and port tuple.
The server implementing the websocket protocol need only listen to a single TCP socket, often defined by the HTTP or HTTPS port number, but not in this example. As part of standard TCP handshaking, the server OS and application open a unique socket for the TCP connection to the new client when the HTTP request which is a websocket request is received. The websocket package takes care of upgrading the protocol used on this new socket from standard HTTP to websocket.
In the example, a goroutine is started for each websocket socket.
The client side, the side initiating the TCP connections, is limited by the number of ephemeral ports its OS can open for a given destination host and port. Honestly, I don't know if this is a limitation of the client OS or the TCP/IP specification itself.
I think the part you are missing is a TCP connection is actually two pairs of IP:PORT.
One for the server, one for the client.
The listening side of a tcp socket is generally always the same IP/Port pair.
Example: net.Listen("tcp", ":8080") is listening on port 8080 (on all interfaces in this case)
The connecting (client) side is usually uses a single outgoing IP along with a random port.
Example: net.Dial("tcp","server:8080) Selects a random available ephemeral port and then attempts to connect to server:8080.
So, in the above example, that connection is: client.ip:32768 -> server.ip:8080 (where 32768 is the ephemeral port selected)
the two pairs combined make a unique connection.
The server side can take as many connections from a single client as there are available (client side) ports. It can also take as many clients are there are IP addresses.
Think of it as, for one listening socket, you can theoretically have 2^16(ports) * 2^32(ipv4 addrs) connections.
In reality, there are reserved IPs, ports, memory limitations, etc so the number is far smaller.
For exmaple, the ephemeral port range on Linux is 32768 - 61000. Which means I'll start getting errors if I net.Dial("tcp", "server:8080") more than 28232 times as I will have exhausted my ephemeral port range for the given server address. But if the server is listening on 2 separate ports, I can do 28232 to the first port, and another 28232 to the second port.
When you see people do the 10MM connection tests, they have to use multiple client IPs or multiple server IPs/Ports to achieve this (or a combo of both to get 10MM unique client:ip/server:ip pairs)
I have a small web application which opens a TCP socket connection, issues a command, reads the response and then closes the connection for every request to a particular REST endpoint.
I've started load testing the endpoint using Apache JMeter and am noticing that after running for some time, I start seeing errors like "Cannot assign requested address", the code opening this connection is:
def lookup(word: String): Option[String] = {
try {
val socket = new Socket(InetAddress.getByName("localhost"), 2222)
val out = new PrintStream(socket.getOutputStream)
val reader = new BufferedReader(new InputStreamReader(socket.getInputStream, "utf8"))
out.println("lookup " + word)
out.flush()
var curr = reader.readLine()
var response = ""
while (!curr.contains("SUCC") && !curr.contains("FAIL")) {
response += curr + "\n"
curr = reader.readLine()
}
socket.close()
curr match {
case code if code.contains(SUCCESS_CODE) => {
Some(response)
}
case _ => None
}
}
catch {
case e: Exception => println("Got an exception "+ e.getMessage); None
}
}
When I run netstat I also see a lot of the following TIME_WAIT connection statues, which implies to me that I am running out of ports in the ephemeral space.
tcp6 0 0 localhost:54646 localhost:2222 TIME_WAIT
tcp6 0 0 localhost:54638 localhost:2222 TIME_WAIT
tcp6 0 0 localhost:54790 localhost:2222 TIME_WAIT
tcp6 0 0 localhost:54882 localhost:2222 TIME_WAIT
I am wondering what the best solution is for this issue. My current thought is that creating a connection pool where connections to this service running on port 2222 can be reused by different HTTP requests, rather than creating new requests each time. Is this a sensible way of fixing the issue and making the application scale better? It seems like a lot of overhead to introduce and definitely makes my application more complicated.
Are there any other solutions for helping this application scale and overcome this port issue that I'm not seeing? My web application is running in an Ubuntu linux VM.
Yes, creating a connection pool is a good solution. However, an easier solution is to have the server close connections, rather than the client. In this case, the server's sockets, rather than the client's, would end up in TIME_WAIT state, so the client would not run out of ports. On the server side, connections in TIME_WAIT state would not make the server run out of ports because they all use the same local port.
To make sure that the connection is closed by the server, you need to read from the socket (on the client) until you reach end-of-file condition. At this moment, it is safe to close the socket on the client side because the server has closed it already. Of course, you need to make sure that the server would close the socket, rather than, say, waiting for a new request.
Alternatively, if you have root access, there are some sysctl options that you can tweak:
net.ipv4.ip_local_port_range – range of ephemeral ports. Increase it to make more ports available for outgoing connections.
net.ipv4.tcp_tw_recycle – enable faster recycling of connections in TIME_WAIT state.
net.ipv4.tcp_tw_reuse – enable reuse of connections in TIME_WAIT state. Not recommended.
See man pages ip(7) and tcp(7) for more information.
Connection-pooling will solve this problem for you.
I want to point out that the net.ipv4.tcp_tw_recycle has been completely removed from Linux 4.12 per commit 95a22caee396
Linux will randomize timestamp offsets for each connection, making this option completely broken, with or without NAT.
When the remote host is a NAT device, the condition on timestamps will forbid all the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems.
Source
Machine is RHEL 5.3 (kernel 2.6.18).
Some times I notice in netstat that my application has connection, established TCP connection when Local Address and Foreign Address are same.
Here same problem reported by someone else too.
The symptoms are same as described in link - client connects to port X port of server running locally. After some time netstat shows that client has connection from 127.0.0.1:X to 127.0.0.1:X
How it's possible?
Edit 01
Simultaneous open is causing the problem (thanks a lot to Hasturkun). You can see it on classical TCP state diagram in transition from SYN_SENT state to SYNC_RECEIVED
This may be caused by TCP simultaneous connect (mentioned on this post to LKML, see also here).
It's possible for a program looping on trying to connect to a port within the dynamic local port range (which can be seen in /proc/sys/net/ipv4/ip_local_port_range),to succeed while the server is not listening on that port.
On a large enough number of attempts, the socket being used to connect may be bound to the same port being connected to, which succeeds due to previously mentioned simultaneous connect. You now magically have a client connected to itself
A TCP connection is uniquely identified by this tuple (local address, local port #, foreign address, foreign port #). There is no requirement that local address and foreign address, or even that the port numbers be different (though that would be exceedingly strange). But there is at most 1 TCP connection that has the same values for a given tuple.
When a computer connects to itself, it's local address and foreign address are almost always the same. After all, the 'local' side and 'foreign' side are actually the same computer. In fact, when this happens your computer should be showing two connections that have the same 'local' and 'foreign' addresses, but reversed port numbers. For example:
$ ssh localhost
will result in two connections that look something like this:
$ netstat -nA inet | fgrep :22
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:56039 127.0.0.1:22 ESTABLISHED
tcp 0 0 127.0.0.1:22 127.0.0.1:56039 ESTABLISHED
As you can see, the local address and foreign addresses are the same, but the port numbers are reversed. The unique tuple for this TCP connection is (127.0.0.1, 56039, 127.0.0.1, 22). There will be no other TCP connection that has these same four fields.
The fact you see two is because your computer is both ends of the connection. Each end has its own view of which one is 'foreign' and which is 'local'.
You can even connect to yourself on the same port, and while this is not a common occurrence, it is not forbidden by the spec. Here is a sample program in Python which will do this:
import socket
import time
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('127.0.0.1', 56443))
s.connect(('127.0.0.1', 56443))
time.sleep(30)
This code works because one way in which it's possible to open a TCP connection is to have the other side of the connection try to open one with you simultaneously. This is known as simultaneous SYN exchange, and the linked to StackOverflow answer describes what that's about.
I also have a paper on using simultaneous SYN exchange to get through NAT, though in that case the source and foreign would be completely different.