I'm writing a HTTP/1.1 client in (asyncio) Python, and wondering if sockets should be created with the SO_KEEPALIVE option
import socket
sock = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM, proto=socket.IPPROTO_TCP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
Should it always be enabled or disabled? Are there certain situations where it is better to enable it or not? Are there tradeoffs to be made? It the answer different if it's HTTPS?
I am specifically thinking in reference to connections used for more than one HTTP request (i.e. using HTTP Keep-Alive).
TCP keep-alive is used to detect a loss of connectivity for TCP connection which are idle (i.e. no data transfer) for a longer duration. HTTP/1 usually does not fit this use case and thus it makes not much sense to have TCP keep-alive active. But it also does no harm. In fact, it likely makes no difference at all what shows up on the wire.
In HTTP/1 the client sends a request which more or less immediately is followed by a response from the server. And if HTTP keep-alive is active then another request might follow which again results in a response. The duration between these requests is usually short, i.e. it is not common to keep an idle connection open for long. It is also expected that client and server can close the connection at any time after the request-response was done and that they must also be able to handle such connection close from the peer. Thus it is likely that the connection either got closed or that new data got transferred before the TCP keep-alive timer could trigger the delivery of the empty keep-alive packet.
Related
Objective:
Never close connection between client and SOCKS proxy + reuse it to send multiple HTTPS requests to different targets (example targets: google.com, cloudflare.com) without closing the socket during the switch to different target.
Step 1:
So I have client which connects to SOCKS proxy server over TCP connection. That is client socket(and only socket(file descriptor) used in this project).
client -> proxy
Step 2:
Then after connection is established and verified. Then it does TLS connect to the target server which can be for example google.com (DNS lookup is done before this).
Now we have connection:
client -> proxy -> target
Step 3:
Then client sends HTTPS request over it and receives response successfully.
Issue appears:
After that I want to close connection explicitly between proxy and target so I can send request to another target. For this it is required to close TLS connection and I don't know how to do it without closing connection between client and proxy which is not acceptable.
Possible solutions?:
1:
Would sending Connection: close\n\r request to current target close connection only between proxy and target and not close the socket.
2:
If I added Connection: close\n\r to headers of every request, would that close the socket and thus it's not valid solution?
Question:
(NodeJS) I made custom https Agent which handles Agent-s method -> callback(req, opts) where opts argument is request options from what client sent to target (through proxy). This callback returns tls socket after it's connected, I built tls socket connection outside of the callback and passed it to agent. Is it possible to use this to close connection between proxy and target using req.close(), would this close the socket? Also what is the point of req in Agent's callback, can it be used in this case?
Any help is appreciated.
If you spin up wireshark and look at what is happening through your proxy, you should quickly see that HTTP/S requests are connection oriented, end-to-end (for HTTPS) and also time-boxed. If you stop and think about it, they are necasarily so, to avoid issues such as the confused deputy problem etc.
So the first bit to note is that for HTTPS, the proxy will only see the initial CONNECT request, and then from there on everything is just a TCP stream of TLS bytes. Which means that the proxy won't be able to see the headers (that is, unless your proxy is a MITM that intercepts the TLS handshake, and you haven't mentioned this, so I've assumed not).
The next bit is that the agent/browser will open connections in parallel (typically a half-dozen for a browser) and will also use pipelining and keep-alive to send multiple requests down the same connection.
Then there are connection limits imposed by the browser, and servers. These typically cap the number of requests, and the duration that they are held open, before speculatively closing them. If they didn't, any reasonably busy server would quickly exhaust all their TCP sockets.
So all-in, what you are looking to achieve isn't going to work.
That said, if you are looking to improve performance, the node client has a few things you can enable and tweak:
Enable TLS session reuse, which will make connections much more
efficient to establish.
Enable keep-alive, which will funnel multiple requests through
the same connection.
I have a requirement to enable TCP keepalive on any connections and now I am struggling with the results from our test case. I think this is because I do not really understand when the first keepalive probe is sent. I read the following in the documentation for tcp_keepalive_time on Linux:
the interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe; after the
connection is marked to need keepalive, this counter is not used any
further
Some other sources state that this is the time a connection is idle, but they do not further define what this means. I also looked into Stevens to find a more formal definition of this, because I am wondering what "the last data packet sent" actually means when considering retransmissions.
In my test case, I have a connection where data is only sent from a server to a client at rather high rates. To test keepalive, we unplugged the cable on the client's NIC. I can now see that the network stack tries to send the data and enters the retransmission state, but no keep alive probe is sent. Is it correct that keep alive probes are not sent during retransmission?
I have a connection where data is only sent from a server to a client
at rather high rates.
Then you'll never see keepalives. Keepalives are sent when there is "silence on the wire". RFC1122 has some explanation re keepalives.
A "keep-alive" mechanism periodically probes the other end of a
connection when the connection is otherwise idle, even when there is
no data to be sent
Back to your question:
Some other sources state that this is the time a connection is idle,
but they do not further define what this means.
This is how long TCP will wait before poking the peer "hoy! still alive?".
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
In other words, you've been using a TCP connection and it has been great. However, for the past 2 hours there hasn't been anything to send. Is it reasonable to assume the connection is still alive? Is it reasonable to assume all the middleboxes in the middle still have state about your connection? Opinions vary and keepalives aren't part of RFC793.
The TCP specification does not include a keep-alive mechanism it
could: (1) cause perfectly good connections to break during transient
Internet failures; (2) consume unnecessary bandwidth ("if no one is
using the connection, who cares if it is still good?")
To test keepalive, we unplugged the cable on the client's NIC.
This isn't testing keepalive. This is testing your TCPs retransmit strategy, i.e. how many times and how often TCP will try to get your message across. On a Linux box this (likely) ends up testing net.ipv4.tcp_retries2:
How may times to retry before killing alive TCP connection. RFC 1122
says that the limit should be longer than 100 sec. It is too small
number. Default value 15 corresponds to 13-30min depending on RTO.
But RFC5482 - TCP User Timeout Option provides more ways to influence it.
The TCP user timeout controls how long transmitted data may remain
unacknowledged before a connection is forcefully closed.
Back to the question:
Is it correct that keep alive probes are not sent during retransmission
It makes sense: TCP is already trying to elicit a response from the other peer, an empty keepalive would be superfluous.
Linux-specific (2.4+) options to influence keepalive
TCP_KEEPCNT The maximum number of keepalive probes TCP should send before dropping the connection.
TCP_KEEPIDLE The time (in seconds) the connection needs to remain idle before TCP starts sending keepalive probes, if the socket option SO_KEEPALIVE has been set on this socket
TCP_KEEPINTVL The time (in seconds) between individual keepalive probes
Linux-specific (2.6.37+) option to influence TCP User Timeout
TCP_USER_TIMEOUT The maximum amount of time in
milliseconds that transmitted data may remain unacknowledged before
TCP will forcibly close connection.
So for example your application could use this option to determine how long the connection survives when there is no connectivity (similar to your NIC-unplugging example). E.g. if you have reason to believe the client will come back (perhaps they closed the laptop lid? spotty wireless access?) you can specify a timeout of 12 hours and when they do come back the connection will still function.
I just want to ask in net module of Node.js because I did not fully understand in the docs. what will happen if I implement the setKeepAlive() ?. what is the behavior of this setKeepAlive() ?
var net = require('net');
var server = net.createServer(function(socket){
socket.setKeepAlive(true,60000); //1 min = 60000 milliseconds.
socket.on('data',function(data){
///receiving data here
});
socket.on('end',function(data){
});
});
server.listen(1333,'127.0.0.1', function () {
console.log("server is listening in port 1333!");
});
Thank you in advance.
The .setKeepAlive() method enables/disables TCP keep alive. This is done at the TCP level in the OS, so it is enabled by the node.js socket library, but the keep-alive functionality is actually implemented in the TCP stack in the host OS.
Here's a pretty good summary of what the keep alive feature does: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html.
Here's a piece of that article that should give you an overview:
The keepalive concept is very simple: when you set up a TCP
connection, you associate a set of timers. Some of these timers deal
with the keepalive procedure. When the keepalive timer reaches zero,
you send your peer a keepalive probe packet with no data in it and the
ACK flag turned on. You can do this because of the TCP/IP
specifications, as a sort of duplicate ACK, and the remote endpoint
will have no arguments, as TCP is a stream-oriented protocol. On the
other hand, you will receive a reply from the remote host (which
doesn't need to support keepalive at all, just TCP/IP), with no data
and the ACK set.
If you receive a reply to your keepalive probe, you can assert that
the connection is still up and running without worrying about the
user-level implementation. In fact, TCP permits you to handle a
stream, not packets, and so a zero-length data packet is not dangerous
for the user program.
This procedure is useful because if the other peers lose their
connection (for example by rebooting) you will notice that the
connection is broken, even if you don't have traffic on it. If the
keepalive probes are not replied to by your peer, you can assert that
the connection cannot be considered valid and then take the correct
action.
Since you are setting keep alive on incoming connections to your server, the effect of the keep alive setting will depend entirely upon what happens with these incoming sockets. If they are short lived (e.g. they connected, exchange some data and then disconnect like a typical HTTP connection without going inactive for any significant amount of time), then the keep-alive setting will not even come into play.
If, on the other hand, the client connects to the server and holds that connection open for a long time, then the keep-alive setting will come into play and you will see the different behaviors that are called out in the above referenced article. In addition, if the client is a battery-powered device (phone, tablet, etc...) and it holds a long running connection, then it may consume more battery power and a small bit more bandwidth responding to the regular keep-alive packets because the device has to wake up to receive incoming packets and then has to transmit to send responses.
I have a server (nginx on Ubuntu) which has listen 80 so_keepalive=4m::;. I have a windows client which I set the TCP Keep-Alive timeout with http://msdn.microsoft.com/en-us/library/windows/desktop/dd877220(v=vs.85).aspx . I set the server timeout to 4 minutes and the client timeout to 5 minutes. What I notice now, capturing the data between them with Wireshark is that after 4 minutes the server sends the TCP Keep-Alive and then one minute later (at 5 minutes from the start) the client sends its own TCP Keep-Alive. Why isn't the timer reset upon receipt of the servers Keep-Alive packet? Is there any way to change this behavior?
The TCP Keep-Alives is kind of a hack that was added later to the protocol.
There is the relevant RFC-1122 paragraph about the question you have:
4.2.3.6 TCP Keep-Alives
[...snip...]
To confirm that an idle connection is still active, these implementations send a probe segment designed to elicit a response from the peer TCP. Such a segment generally contains SEG.SEQ = SND.NXT-1 and may or may not contain one garbage octet of data. Note that on a quiet connection SND.NXT = RCV.NXT, so that this SEG.SEQ will be outside the window. Therefore, the probe causes the receiver to return an acknowledgment segment, confirming that the connection is still live. If the peer has dropped the connection due to a network partition or a crash, it will respond with a RST instead of an acknowledgment segment.
[...snip...]
In other words, a TCP Keep-Alive segment uses an old sequence number (SND.NXT-1) and expects a valid reply from it. There isn't really anything else that distinguish that segment from a normal segment. The TCP stack on the other end does not need to implement anything special to know that a Keep-Alive segment was sent. As long as the protocol was properly implemented, with or without the Keep-Alive feature, it will work transparently.
As mentioned in the RFC, there are cases where a TCP implementation will not answer such a request properly unless it includes at least one octet of data. I do not know which ones (Microsoft?)...
Further, the documentation says that only the server side should make use of the Keep-Alive feature. For sure, if you know that a server has the Keep-Alive turned on, then there is no need for you to have it in your client.
I want to proxy WebSocket connections to multiple node.js servers using Amazon Elastic Load Balancer. Since Amazon ELB does not provide actual WebSocket support, I would need to use its vanilla TCP messaging. However, I'm trying to understand how this would work without some sort of sticky session functionality.
I understand that WebSockets work by first sending an HTTP Upgrade request from the client, which is handled by the server by sending a response which correctly handles key authentication. After the server sends that response and it is approved by the client, there is a bidirectional connection between that client and server.
However let's say the client, after approving the server response, sends data to the server. If it sends the data to the load balancer, and the load balancer then relays that data to a different server that did not handle the original WebSocket Upgrade request, then how will this new server be aware of the WebSocket connection? Or will the client automatically bypass the load balancer and send data directly to the server that handled the initial upgrade?
I think what we need to understand in order to answer this question is how exactly the underlying TCP connection evolves during the whole WebSocket creation process. You will realize that the sticky part of a WebSocket connection is the underlying TCP connection itself. I am not sure what you mean with "session" in the context of WebSockets.
At a high level, initiating a "WebSocket connection" requires the client to send an HTTP GET request to an HTTP server whereas the request includes the Upgrade header field. Now, for this request to happen the client needs to have established a TCP connection to the HTTP server (that might be obvious, but I think here it is important to point this out explicitly). The subsequent HTTP server response is then sent through the same TCP connection.
Note that now, after the server response has been sent, the TCP connection is still open/alive if not actively closed by either the client or the server.
Now, according to RFC 6455, the WebSocket standard, at the end of section 4.1:
If the server's response is validated as provided for above, it is
said that The WebSocket Connection is Established and that the
WebSocket Connection is in the OPEN state
I read from here that the same TCP connection that was initiated by the client before sending the initial HTTP GET (Upgrade) request will just be left open and will from now on serve as the transport layer for the full-duplex WebSocket connection. And this makes sense!
With respect to your question this means that a load balancer will only play a role before the initial HTTP GET (Upgrade) request is made, i.e. before the one and only TCP connection involved in said WebSocket connection creation is established between the two communication end points. Thereafter, the TCP connection stays established and cannot become "redirected" by a network device in between.
We can conclude that -- in your session terminology -- the TCP connection defines the session. As long as a WebSocket connection is alive (i.e. is not terminated), it by definition provides and lives in its own session. Nothing can change this session. Speaking in this picture, two independent WebSocket connections, however, cannot share the same session.
If you referred to something else with "session", then it probably is a session that is introduced by the application layer and we cannot comment on that one.
Edit with respect to your comments:
so you're saying that the load balancer is not involved in the TCP
connection
No, that is not true, at least in general. It definitely can take influence upon TCP connection establishment, in the sense that it can decide what to do with the client connection attempt. The specifics depend on the exact type of load balancer (* , see below). Important: After the connection is established between two endpoints -- whereas I don't consider the load balancer to be an endpoint, I refer to WebSocket client and WebSocket server -- the two endpoints will not change anymore for the lifetime of the WebSocket connection. The load balancer might* still be in the network path, but can be assumed to not take influence anymore.
Therefore the full-duplex connection is between the client and the
end server?
Yes!
***There are different types of load balancing. Depending on the type, the role of the load balancer is different after connection establishment between the two end points. Examples:
If the load balancing happens on DNS basis, then the load balancer is not involved in the final TCP connection at all. It just tells the client to which host is has to connect directly.
If the load balancer works like the Layer 4 ELB from AWS (docs here), then it so to say proxies the TCP connection. So the client would actually see the ELB itself as the server. What happens, however, is that the ELB just forwards the packages in both directions, without change. Hence, it is still heavily involved in the TCP connection, just transparently. In this case there are actually two permanent TCP connections involved: one from you to the ELB, and one from the ELB to the server. These are again permanent for the lifetime of your WebSocket connection.
WebSocket uses a persistent TCP connection, and hence requires all IP packets for that TCP connection to be forwarded to the same backend server (for the lifetime of the TCP connection).
It needs to be sticky. This is different from L7 HTTP LBs which are able to dispatch on a per HTTP-request basis.
A LB can work sticky by different approaches, i.e.
hash the source IP/port to the set of alive backend servers
upon TCP connection establishment, choose a backend server and remember that