Too many TIME_WAIT connections, getting "Cannot assign requested address" - linux

Question

I have a small web application which opens a TCP socket connection, issues a command, reads the response and then closes the connection for every request to a particular REST endpoint.
I've started load testing the endpoint using Apache JMeter and am noticing that after running for some time, I start seeing errors like "Cannot assign requested address", the code opening this connection is:
def lookup(word: String): Option[String] = {
try {
val socket = new Socket(InetAddress.getByName("localhost"), 2222)
val out = new PrintStream(socket.getOutputStream)
val reader = new BufferedReader(new InputStreamReader(socket.getInputStream, "utf8"))
out.println("lookup " + word)
out.flush()
var curr = reader.readLine()
var response = ""
while (!curr.contains("SUCC") && !curr.contains("FAIL")) {
response += curr + "\n"
curr = reader.readLine()
}
socket.close()
curr match {
case code if code.contains(SUCCESS_CODE) => {
Some(response)
}
case _ => None
}
}
catch {
case e: Exception => println("Got an exception "+ e.getMessage); None
}
}
When I run netstat I also see a lot of the following TIME_WAIT connection statues, which implies to me that I am running out of ports in the ephemeral space.
tcp6 0 0 localhost:54646 localhost:2222 TIME_WAIT
tcp6 0 0 localhost:54638 localhost:2222 TIME_WAIT
tcp6 0 0 localhost:54790 localhost:2222 TIME_WAIT
tcp6 0 0 localhost:54882 localhost:2222 TIME_WAIT
I am wondering what the best solution is for this issue. My current thought is that creating a connection pool where connections to this service running on port 2222 can be reused by different HTTP requests, rather than creating new requests each time. Is this a sensible way of fixing the issue and making the application scale better? It seems like a lot of overhead to introduce and definitely makes my application more complicated.
Are there any other solutions for helping this application scale and overcome this port issue that I'm not seeing? My web application is running in an Ubuntu linux VM.

Answer 1

Yes, creating a connection pool is a good solution. However, an easier solution is to have the server close connections, rather than the client. In this case, the server's sockets, rather than the client's, would end up in TIME_WAIT state, so the client would not run out of ports. On the server side, connections in TIME_WAIT state would not make the server run out of ports because they all use the same local port.
To make sure that the connection is closed by the server, you need to read from the socket (on the client) until you reach end-of-file condition. At this moment, it is safe to close the socket on the client side because the server has closed it already. Of course, you need to make sure that the server would close the socket, rather than, say, waiting for a new request.
Alternatively, if you have root access, there are some sysctl options that you can tweak:
net.ipv4.ip_local_port_range – range of ephemeral ports. Increase it to make more ports available for outgoing connections.
net.ipv4.tcp_tw_recycle – enable faster recycling of connections in TIME_WAIT state.
net.ipv4.tcp_tw_reuse – enable reuse of connections in TIME_WAIT state. Not recommended.
See man pages ip(7) and tcp(7) for more information.

Answer 2

Connection-pooling will solve this problem for you.

Answer 3

I want to point out that the net.ipv4.tcp_tw_recycle has been completely removed from Linux 4.12 per commit 95a22caee396
Linux will randomize timestamp offsets for each connection, making this option completely broken, with or without NAT.
When the remote host is a NAT device, the condition on timestamps will forbid all the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems.
Source