I have a server (nginx on Ubuntu) which has listen 80 so_keepalive=4m::;. I have a windows client which I set the TCP Keep-Alive timeout with http://msdn.microsoft.com/en-us/library/windows/desktop/dd877220(v=vs.85).aspx . I set the server timeout to 4 minutes and the client timeout to 5 minutes. What I notice now, capturing the data between them with Wireshark is that after 4 minutes the server sends the TCP Keep-Alive and then one minute later (at 5 minutes from the start) the client sends its own TCP Keep-Alive. Why isn't the timer reset upon receipt of the servers Keep-Alive packet? Is there any way to change this behavior?
The TCP Keep-Alives is kind of a hack that was added later to the protocol.
There is the relevant RFC-1122 paragraph about the question you have:
4.2.3.6 TCP Keep-Alives
[...snip...]
To confirm that an idle connection is still active, these implementations send a probe segment designed to elicit a response from the peer TCP. Such a segment generally contains SEG.SEQ = SND.NXT-1 and may or may not contain one garbage octet of data. Note that on a quiet connection SND.NXT = RCV.NXT, so that this SEG.SEQ will be outside the window. Therefore, the probe causes the receiver to return an acknowledgment segment, confirming that the connection is still live. If the peer has dropped the connection due to a network partition or a crash, it will respond with a RST instead of an acknowledgment segment.
[...snip...]
In other words, a TCP Keep-Alive segment uses an old sequence number (SND.NXT-1) and expects a valid reply from it. There isn't really anything else that distinguish that segment from a normal segment. The TCP stack on the other end does not need to implement anything special to know that a Keep-Alive segment was sent. As long as the protocol was properly implemented, with or without the Keep-Alive feature, it will work transparently.
As mentioned in the RFC, there are cases where a TCP implementation will not answer such a request properly unless it includes at least one octet of data. I do not know which ones (Microsoft?)...
Further, the documentation says that only the server side should make use of the Keep-Alive feature. For sure, if you know that a server has the Keep-Alive turned on, then there is no need for you to have it in your client.
Related
I'm writing a HTTP/1.1 client in (asyncio) Python, and wondering if sockets should be created with the SO_KEEPALIVE option
import socket
sock = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM, proto=socket.IPPROTO_TCP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
Should it always be enabled or disabled? Are there certain situations where it is better to enable it or not? Are there tradeoffs to be made? It the answer different if it's HTTPS?
I am specifically thinking in reference to connections used for more than one HTTP request (i.e. using HTTP Keep-Alive).
TCP keep-alive is used to detect a loss of connectivity for TCP connection which are idle (i.e. no data transfer) for a longer duration. HTTP/1 usually does not fit this use case and thus it makes not much sense to have TCP keep-alive active. But it also does no harm. In fact, it likely makes no difference at all what shows up on the wire.
In HTTP/1 the client sends a request which more or less immediately is followed by a response from the server. And if HTTP keep-alive is active then another request might follow which again results in a response. The duration between these requests is usually short, i.e. it is not common to keep an idle connection open for long. It is also expected that client and server can close the connection at any time after the request-response was done and that they must also be able to handle such connection close from the peer. Thus it is likely that the connection either got closed or that new data got transferred before the TCP keep-alive timer could trigger the delivery of the empty keep-alive packet.
As a way to learn how raw sockets work, I programmed a dummy firewall which drops the packets based on the TCP destination port. It is working but the problem is that the client retries for quite some time until the time out is finally reached.
I was wondering if perhaps the client retries for so long because it does not receive any answer. In that case, would it help if the firewall replies with a TCP RST to the TCP SYNC messages from the client? If not, is there any way to force the client to stop retrying (not reducing the timeout time in the Linux but more, getting a specific answer to its packets which will make the client stop)?
You can think of your firewall as the same case as if the port were closed on the host OS. What would the host OS's TCP/IP stack do?
RFC 793 (original TCP RFC) has the following to say about this case:
If the connection does not exist (CLOSED) then a reset is sent
in response to any incoming segment except another reset. In
particular, SYNs addressed to a non-existent connection are rejected
by this means.
You should read the TCP RFCs and make sure your TCP RST packet conforms to the requirements for this case. A malformed RST will be ignored by the client.
RFC 1122 also indicates that ICMP Destination Unreachable containing codes 2-4 should cause an abort in the connection. It's important to note the codes because 0, 1, and 5 are listed as a MUST NOT for aborting the connection
Destination Unreachable -- codes 2-4
These are hard error conditions, so TCP SHOULD abort
the connection.
Your firewall is behaving correctly. It is a cardinal principle of information scurity not to disclose any information to the attacker. Sending an RST would disclose that the host exists.
There were firewalls that did that 15-20 years ago but there were frowned on. Nowadays they behave like yours: they just drop the packet and do nothing in reply.
It is normal for the client to retry a few times before giving up if there is no response, but contrary to what you have been told in comments, a client will give up immediately with 'connection refused' if it receives an RST. It only retries if there is no response at all.
I have a requirement to enable TCP keepalive on any connections and now I am struggling with the results from our test case. I think this is because I do not really understand when the first keepalive probe is sent. I read the following in the documentation for tcp_keepalive_time on Linux:
the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the
connection is marked to need keepalive, this counter is not used any
further
Some other sources state that this is the time a connection is idle, but they do not further define what this means. I also looked into Stevens to find a more formal definition of this, because I am wondering what "the last data packet sent" actually means when considering retransmissions.
In my test case, I have a connection where data is only sent from a server to a client at rather high rates. To test keepalive, we unplugged the cable on the client's NIC. I can now see that the network stack tries to send the data and enters the retransmission state, but no keep alive probe is sent. Is it correct that keep alive probes are not sent during retransmission?
I have a connection where data is only sent from a server to a client
at rather high rates.
Then you'll never see keepalives. Keepalives are sent when there is "silence on the wire". RFC1122 has some explanation re keepalives.
A "keep-alive" mechanism periodically probes the other end of a
connection when the connection is otherwise idle, even when there is
no data to be sent
Back to your question:
Some other sources state that this is the time a connection is idle,
but they do not further define what this means.
This is how long TCP will wait before poking the peer "hoy! still alive?".
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
In other words, you've been using a TCP connection and it has been great. However, for the past 2 hours there hasn't been anything to send. Is it reasonable to assume the connection is still alive? Is it reasonable to assume all the middleboxes in the middle still have state about your connection? Opinions vary and keepalives aren't part of RFC793.
The TCP specification does not include a keep-alive mechanism it
could: (1) cause perfectly good connections to break during transient
Internet failures; (2) consume unnecessary bandwidth ("if no one is
using the connection, who cares if it is still good?")
To test keepalive, we unplugged the cable on the client's NIC.
This isn't testing keepalive. This is testing your TCPs retransmit strategy, i.e. how many times and how often TCP will try to get your message across. On a Linux box this (likely) ends up testing net.ipv4.tcp_retries2:
How may times to retry before killing alive TCP connection. RFC 1122
says that the limit should be longer than 100 sec. It is too small
number. Default value 15 corresponds to 13-30min depending on RTO.
But RFC5482 - TCP User Timeout Option provides more ways to influence it.
The TCP user timeout controls how long transmitted data may remain
unacknowledged before a connection is forcefully closed.
Back to the question:
Is it correct that keep alive probes are not sent during retransmission
It makes sense: TCP is already trying to elicit a response from the other peer, an empty keepalive would be superfluous.
Linux-specific (2.4+) options to influence keepalive
TCP_KEEPCNT The maximum number of keepalive probes TCP should send before dropping the connection.
TCP_KEEPIDLE The time (in seconds) the connection needs to remain idle before TCP starts sending keepalive probes, if the socket option SO_KEEPALIVE has been set on this socket
TCP_KEEPINTVL The time (in seconds) between individual keepalive probes
Linux-specific (2.6.37+) option to influence TCP User Timeout
TCP_USER_TIMEOUT The maximum amount of time in
milliseconds that transmitted data may remain unacknowledged before
TCP will forcibly close connection.
So for example your application could use this option to determine how long the connection survives when there is no connectivity (similar to your NIC-unplugging example). E.g. if you have reason to believe the client will come back (perhaps they closed the laptop lid? spotty wireless access?) you can specify a timeout of 12 hours and when they do come back the connection will still function.
Machine - linux, 3.10.19 kernel
This is in a large distributed system, there are several servers and clients (on same as well as different nodes/machines) having TCP connections with each other.
Test case:
The client program node/machine is switched off (on purpose, test case) and the only way for server to know about his disconnection is via keepalive timer (idle time=40 sec, 4 probes, probe time=10 sec).
Good case:
This works fine in most of the cases, the server gets to know that the client has gone down in [40,70] sec.
Bad case:
But I am hitting another unique situation where while keepalive timer is running, the server tries sending some data to the client, and this in turn starts the TCP retransmission timer which overrides/kills the keepalive timer. It takes ~15 min for the retransmission timer to detect that the other end is not there anymore.
15 min is a lot of time for server to realize this. I am looking for ways how others handle such a situation. Do I need to tweak my retransmission timer values?
Thanks!
There is a completely separate configuration for retransmission timeouts.
From Linux's tcp.7 man page:
tcp_retries2 (integer; default: 15; since Linux 2.2)
The maximum number of times a TCP packet is retransmitted in
established state before giving up. The default value is 15, which
corresponds to a duration of approximately between 13 to 30 minutes,
depending on the retransmission timeout. The RFC 1122 specified
minimum limit of 100 seconds is typically deemed too short.
This is likely the value you'll want to adjust to change how long it takes to detect if your connection has vanished.
I have the same issue with a linux kernel release 4.3.0-1-amd64:
I used a server and a client, connected to the same switch.
TCP keep-alive mecanism works correctly for client and server in the following cases:
When no message is sent between the cable disconnection and the socket disconnection (by the tcp keep-alive mecanism).
When the cable is disconnected between the client/server and the switch (which sets the link state to down) even if the client/server application tries to send a message.
When the wire is unpluged on the other side of the switch, TCP Keep-Alive frames are transmitted until an applicative message is sent. Then, TCP Retransmission frames are sent and TCP keep-alive frames stop being sent, which prevents the socket to be closed.
I just want to ask in net module of Node.js because I did not fully understand in the docs. what will happen if I implement the setKeepAlive() ?. what is the behavior of this setKeepAlive() ?
var net = require('net');
var server = net.createServer(function(socket){
socket.setKeepAlive(true,60000); //1 min = 60000 milliseconds.
socket.on('data',function(data){
///receiving data here
});
socket.on('end',function(data){
});
});
server.listen(1333,'127.0.0.1', function () {
console.log("server is listening in port 1333!");
});
Thank you in advance.
The .setKeepAlive() method enables/disables TCP keep alive. This is done at the TCP level in the OS, so it is enabled by the node.js socket library, but the keep-alive functionality is actually implemented in the TCP stack in the host OS.
Here's a pretty good summary of what the keep alive feature does: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html.
Here's a piece of that article that should give you an overview:
The keepalive concept is very simple: when you set up a TCP
connection, you associate a set of timers. Some of these timers deal
with the keepalive procedure. When the keepalive timer reaches zero,
you send your peer a keepalive probe packet with no data in it and the
ACK flag turned on. You can do this because of the TCP/IP
specifications, as a sort of duplicate ACK, and the remote endpoint
will have no arguments, as TCP is a stream-oriented protocol. On the
other hand, you will receive a reply from the remote host (which
doesn't need to support keepalive at all, just TCP/IP), with no data
and the ACK set.
If you receive a reply to your keepalive probe, you can assert that
the connection is still up and running without worrying about the
user-level implementation. In fact, TCP permits you to handle a
stream, not packets, and so a zero-length data packet is not dangerous
for the user program.
This procedure is useful because if the other peers lose their
connection (for example by rebooting) you will notice that the
connection is broken, even if you don't have traffic on it. If the
keepalive probes are not replied to by your peer, you can assert that
the connection cannot be considered valid and then take the correct
action.
Since you are setting keep alive on incoming connections to your server, the effect of the keep alive setting will depend entirely upon what happens with these incoming sockets. If they are short lived (e.g. they connected, exchange some data and then disconnect like a typical HTTP connection without going inactive for any significant amount of time), then the keep-alive setting will not even come into play.
If, on the other hand, the client connects to the server and holds that connection open for a long time, then the keep-alive setting will come into play and you will see the different behaviors that are called out in the above referenced article. In addition, if the client is a battery-powered device (phone, tablet, etc...) and it holds a long running connection, then it may consume more battery power and a small bit more bandwidth responding to the regular keep-alive packets because the device has to wake up to receive incoming packets and then has to transmit to send responses.