Objective:
Never close connection between client and SOCKS proxy + reuse it to send multiple HTTPS requests to different targets (example targets: google.com, cloudflare.com) without closing the socket during the switch to different target.
Step 1:
So I have client which connects to SOCKS proxy server over TCP connection. That is client socket(and only socket(file descriptor) used in this project).
client -> proxy
Step 2:
Then after connection is established and verified. Then it does TLS connect to the target server which can be for example google.com (DNS lookup is done before this).
Now we have connection:
client -> proxy -> target
Step 3:
Then client sends HTTPS request over it and receives response successfully.
Issue appears:
After that I want to close connection explicitly between proxy and target so I can send request to another target. For this it is required to close TLS connection and I don't know how to do it without closing connection between client and proxy which is not acceptable.
Possible solutions?:
1:
Would sending Connection: close\n\r request to current target close connection only between proxy and target and not close the socket.
2:
If I added Connection: close\n\r to headers of every request, would that close the socket and thus it's not valid solution?
Question:
(NodeJS) I made custom https Agent which handles Agent-s method -> callback(req, opts) where opts argument is request options from what client sent to target (through proxy). This callback returns tls socket after it's connected, I built tls socket connection outside of the callback and passed it to agent. Is it possible to use this to close connection between proxy and target using req.close(), would this close the socket? Also what is the point of req in Agent's callback, can it be used in this case?
Any help is appreciated.
If you spin up wireshark and look at what is happening through your proxy, you should quickly see that HTTP/S requests are connection oriented, end-to-end (for HTTPS) and also time-boxed. If you stop and think about it, they are necasarily so, to avoid issues such as the confused deputy problem etc.
So the first bit to note is that for HTTPS, the proxy will only see the initial CONNECT request, and then from there on everything is just a TCP stream of TLS bytes. Which means that the proxy won't be able to see the headers (that is, unless your proxy is a MITM that intercepts the TLS handshake, and you haven't mentioned this, so I've assumed not).
The next bit is that the agent/browser will open connections in parallel (typically a half-dozen for a browser) and will also use pipelining and keep-alive to send multiple requests down the same connection.
Then there are connection limits imposed by the browser, and servers. These typically cap the number of requests, and the duration that they are held open, before speculatively closing them. If they didn't, any reasonably busy server would quickly exhaust all their TCP sockets.
So all-in, what you are looking to achieve isn't going to work.
That said, if you are looking to improve performance, the node client has a few things you can enable and tweak:
Enable TLS session reuse, which will make connections much more
efficient to establish.
Enable keep-alive, which will funnel multiple requests through
the same connection.
Related
I have a nodejs TLS client socket on my laptop, connected to a TLS server socket on a different computer (server). The server cannot connect to my laptop. The laptop needs to initiate the connection.
Now I want the server to make requests to my laptop. The idea is to reuse the HTTP protocol. Is there a way to create a HTTP server using the existing TLS client socket?
This way, the server machine can make a HTTP request, and the client TLS receives it, and the HTTP server would parse it? Or am I missing something?
Once you have a TCP socket open between laptop and server, you can send data either way over that socket. So, if the server wants to send some query to the laptop, it can do so just fine. You will have to invent your own protocol on top of TCP to do that, but it could be as simple as a text/line based protocol if you want.
Or, instead of making a plain TCP connection, you can make a webSocket or socket.io connection from the laptop to the server (instead of the plain TCP connection) and then either side can send messages either way and the protocol part is already taken care of. If you use socket.io, it will automatically reconnect if the connection is interrupted too.
There is no simple way to attach an HTTP server to an existing TCP socket and it would be fraught with difficulties too because an HTTP connection is generally not a continuous connection over which you send many separate requests (advanced versions of http can do that, but I doubt you want to get into implementing all that logic on both ends). You could use the HTTP protocol over your existing TCP socket, but that would probably be substantially more work to implement than just use the webSocket/socket.io idea above.
I have a node server and a web page connected via socket.io. I noticed in the browser console that it is outputting
XHR finished loading: GET "http://my_url/socket.io/?EIO=3&transport=polling&t=1418944327412-412&sid=vqLTUtW3QhNLwQG8AAAA".
and
XHR finished loading: POST "http://my_url/socket.io/?EIO=3&transport=polling&t=1418944385398-415&sid=vqLTUtW3QhNLwQG8AAAA".
every few seconds. Should it be doing this or am I missing a setting. I'm really only looking to send data back and forth explicitly via the socket. Perhaps I'm missing something in the set up.
Client side is basically
var socket = io("http://my_url");
with the usual event listeners. Server side is
var io = require('socket.io')(server);
I tried placing this on the server side
io.set('transports', ['websocket']);
but that seemed to kill it.
The socket.io implementation (when using webSockets) sends regular (every few seconds) heartbeat and response packets to constantly verify that the connection is alive and well. This is normal.
These packets are not actual http requests (they are websocket data packets) so there should not be full-on http packets going on unless socket.io is not actually using the webSocket protocol, but is instead using HTTP long polling. socket.io will use the webSocket protocol as long as it is supported in the client (which it should be in all modern browsers nowadays).
You may have to be careful about how you interpret requests in a debugger. A socket.io connection starts its life as an http request with some custom headers and all debuggers will show this initial http request. If webSocket is supported at both ends, then the server will return a response which "upgrades" the connection to the webSocket protocol. That same TCP socket which started out as a TCP request, then becomes a webSocket connection. Subsequent webSockets messages sent on the webSocket then flow over that TCP socket. It is up to the debugger on how it might display that traffic. In the Chrome debugger, you have to open the original http connection and then ask to see websocket traffic and only then can you actually see webSocket packets. But, I could imagine in other debuggers that weren't as webSocket saavy, they might show subsequent packets as related to that original HTTP connection (I haven't looked at how debuggers other than Chrome show webSocket traffic).
The only other reason I can think of that a client would be repeatedly sending HTTP connection requests is if the connection keeps dropping for some reason so the client keeps reconnecting every time the connection drops. socket.io has settings that can control how often/vigorously the client tries to reconnect when the connection is lost, though if you have connection issues, then you really need to figure out why there are connection issues rather than change the reconnect settings.
I'm new in node.js
So the question can be quite naive.
I want to use node.js as a proxy between the javascript client and a windows program which has a API working through a defined port.
So browser sends HTTP request to node.js.
Node.js opens connection with the windows program, sends request, get a respons and returns the response to the javascript that called node.js ( AJAX )
Actually it has been realized so and works.
The problem is that windows program wants to work persistent.
So once the connection is opened it should stay alive.
And my node.js script opens connection. And the next call to node.js try to open connection again. And that leads to error.
So the question - what is the right way and middles to reuse the TCP connection in node.js.
So that next call won't open new connection but go on with already opened one.
It is possible, but requires some coding on your side.
If it is not sticky (the same HTTP client does not require to use the same TCP connection):
Open an connection pools of TCP connections to the windows program
Listen to HTTP requests
If a HTTP request arrives find an unused TCP connection (if none is available, make a new one or wait)
Query the windows program, return the results to who ever called the HTTP request
Mark the TCP connection as free
If it is sticky (always use the same TCP connection for the same HTTP client):
If a HTTP clients connect and he hasn't one, give him session.
If not available for the session: create a TCP connection
Make the request, return the result
Store the TCP "connection" somewhere where you could reach it when the next request comes in and you could identify it by the HTTP clients session (Maybe make a timeout for clearing up)
I want to proxy WebSocket connections to multiple node.js servers using Amazon Elastic Load Balancer. Since Amazon ELB does not provide actual WebSocket support, I would need to use its vanilla TCP messaging. However, I'm trying to understand how this would work without some sort of sticky session functionality.
I understand that WebSockets work by first sending an HTTP Upgrade request from the client, which is handled by the server by sending a response which correctly handles key authentication. After the server sends that response and it is approved by the client, there is a bidirectional connection between that client and server.
However let's say the client, after approving the server response, sends data to the server. If it sends the data to the load balancer, and the load balancer then relays that data to a different server that did not handle the original WebSocket Upgrade request, then how will this new server be aware of the WebSocket connection? Or will the client automatically bypass the load balancer and send data directly to the server that handled the initial upgrade?
I think what we need to understand in order to answer this question is how exactly the underlying TCP connection evolves during the whole WebSocket creation process. You will realize that the sticky part of a WebSocket connection is the underlying TCP connection itself. I am not sure what you mean with "session" in the context of WebSockets.
At a high level, initiating a "WebSocket connection" requires the client to send an HTTP GET request to an HTTP server whereas the request includes the Upgrade header field. Now, for this request to happen the client needs to have established a TCP connection to the HTTP server (that might be obvious, but I think here it is important to point this out explicitly). The subsequent HTTP server response is then sent through the same TCP connection.
Note that now, after the server response has been sent, the TCP connection is still open/alive if not actively closed by either the client or the server.
Now, according to RFC 6455, the WebSocket standard, at the end of section 4.1:
If the server's response is validated as provided for above, it is
said that The WebSocket Connection is Established and that the
WebSocket Connection is in the OPEN state
I read from here that the same TCP connection that was initiated by the client before sending the initial HTTP GET (Upgrade) request will just be left open and will from now on serve as the transport layer for the full-duplex WebSocket connection. And this makes sense!
With respect to your question this means that a load balancer will only play a role before the initial HTTP GET (Upgrade) request is made, i.e. before the one and only TCP connection involved in said WebSocket connection creation is established between the two communication end points. Thereafter, the TCP connection stays established and cannot become "redirected" by a network device in between.
We can conclude that -- in your session terminology -- the TCP connection defines the session. As long as a WebSocket connection is alive (i.e. is not terminated), it by definition provides and lives in its own session. Nothing can change this session. Speaking in this picture, two independent WebSocket connections, however, cannot share the same session.
If you referred to something else with "session", then it probably is a session that is introduced by the application layer and we cannot comment on that one.
Edit with respect to your comments:
so you're saying that the load balancer is not involved in the TCP
connection
No, that is not true, at least in general. It definitely can take influence upon TCP connection establishment, in the sense that it can decide what to do with the client connection attempt. The specifics depend on the exact type of load balancer (* , see below). Important: After the connection is established between two endpoints -- whereas I don't consider the load balancer to be an endpoint, I refer to WebSocket client and WebSocket server -- the two endpoints will not change anymore for the lifetime of the WebSocket connection. The load balancer might* still be in the network path, but can be assumed to not take influence anymore.
Therefore the full-duplex connection is between the client and the
end server?
Yes!
***There are different types of load balancing. Depending on the type, the role of the load balancer is different after connection establishment between the two end points. Examples:
If the load balancing happens on DNS basis, then the load balancer is not involved in the final TCP connection at all. It just tells the client to which host is has to connect directly.
If the load balancer works like the Layer 4 ELB from AWS (docs here), then it so to say proxies the TCP connection. So the client would actually see the ELB itself as the server. What happens, however, is that the ELB just forwards the packages in both directions, without change. Hence, it is still heavily involved in the TCP connection, just transparently. In this case there are actually two permanent TCP connections involved: one from you to the ELB, and one from the ELB to the server. These are again permanent for the lifetime of your WebSocket connection.
WebSocket uses a persistent TCP connection, and hence requires all IP packets for that TCP connection to be forwarded to the same backend server (for the lifetime of the TCP connection).
It needs to be sticky. This is different from L7 HTTP LBs which are able to dispatch on a per HTTP-request basis.
A LB can work sticky by different approaches, i.e.
hash the source IP/port to the set of alive backend servers
upon TCP connection establishment, choose a backend server and remember that
I've just started learning SSL and boy is it confusing
Q1 - How long does SSL connection between a client and server persist? Until client surfs to some other URL or…?
Q2
A) Assume a client (browser) establishes a SSL connection with a IIS server.
Now how does IIS figure out on each postback that it is dealing with same authenticated client/browser and thus that it already has a SSL connection established with that client?
B) Assuming SSL connection isn’t lost if browser surfs to some other URL:
Suppose that moments after SSL connection is established, client surfs to some other URL, and shortly there after it again requests ( via https ) the original page ( one with which it has SSL connection established).
How will IIS server be able to figure out that current request for a page comes from a client that already has SSL connection established with that page and thus will use already established SSL connection?
thanx
EDIT:
Assuming browser surfs to some other URL and if on returning back to original page the SSL connection is still established, how will browser "remember" the value of symetric encryption key, which the two sides used for communicating?
I realize it depends on what browser you use, but with IE and Firefox, I assume when you close a browser, it sends Connection.Close() ( or something to that effect ) to the server and thus SSL connection is immediately closed?
But if you browse away to some other URL, then if browsers doesn't send any notification to the server, wouldn't then SSL connection remain established for quite some time ( even 10 or more minutes ) and thus browser could easily surf back to that page as if nothing happened?!
I appreciate it
Q1. The SSL connection is only good for a single TCP connection between the client and the server. Current browsers (anything with HTTP/1.1 support) can reuse a single connection for downloading multiple resources. Current browsers also make multiple TCP connections to a server in order to download multiple resources in parallel. Because of this, you'll see multiple SSL connections for one page view.
Q2A. If the browser still has a TCP connection open with that server, it can reuse that connection. Otherwise, a new TCP connection with SSL and IIS authentication is negotiated.
Q2B. Same as Q2A. You can't depend on this, but the TCP connections won't be disposed of immediately. There's a chance you could reuse an existing one depending on your browser.
A1. An SSL connection persists until either the client or server closes it. When that happens depends on the protocol being used. For HTTP, most modern clients will make a few parallel connections to the server to fetch the page and its resources, and reuse those connections until the page is loaded.
A2A. The client must authenticate itself on each request if the authentication uses HTTP auth. If the client is using SSL certificate authorization, then this is obviously maintained on a per-connection basis so that subsequent requests on the same connection retain the same credentials.
A2B. The server would know this because presumably the request would come in on that already established SSL connection.
post-edit answers:
What I think you may be missing is that SSL is linked intrinsically to TCP. You cannot have an SSL "connection" to the server that doesn't ride on top of a TCP connection. You break one, you break the other.
Most SSL implementations include "shortcut" negotiation where subsequent new connections can leverage the public key encryption that has already taken place and instead directly use the most recently negotiated symmetric key. The details of this, however, are hidden within the SSL implementation. From the point of view of the user and/or client software, the fiction is maintained that the entire negotiation took place just like it did on the first connection.
If the SSL connection is still established, then it follows that the symmetric key information is still maintained on both ends.
Yes.
Yes, although it would be improbable for the client to keep a connection to a server once it has navigated away to some other site.