I have an interesting network security challenge that I can't figure out the best way to attack.
I need to provide a way to allow two computers (A and B) that are behind firewalls to make a secure connection to each other using only a common "broker" untrusted server on the internet (somewhere like RackSpace). (the server is considered untrusted because the customers behind the firewalls won't trust it since it is on an open server) I can not adjust the firewall settings to allow the networks to directly connect to each other because the connections are no known ahead of time.
This is very similar to a NAT to NAT connection problem like that handled by remote desktop help tools (crossloop, copilot, etc).
What I would really like to find is a way to open an SSL connection between the two hosts and have the public server broker the connection. Preferably when host A tries to connect to host B, it should have to provide a token that the broker can check with host B before establishing the connection.
To add another wrinkle to this, the connection mechanism needs to support two types of communication. First, HTTP request/response to a REST web service and second persistent socket connection(s) to allow for real-time message passing.
I have looked at the techniques I know about like OpenSSL using certificates, OAuth, etc, but I don't see anything that quite does what I need.
Has anyone else handled something like this before? Any pointers?
You can solve your problem with plain SSL.
Just have the untrusted server forward connections between the client hosts as opaque TCP connections. The clients then establish an end-to-end SSL connection over that forwarded TCP tunnel - with OpenSSL, one client calls SSL_accept() and the other calls SSL_connect().
Use certificates, probably including client certificates, to verify that the other end of the SSL connection is who you expect it to be.
(This is conceptually similar to the way that HTTPS connections work over web proxies - the browser just says "connect me to this destination", and establishes an SSL connection with the desired endpoint. The proxy just forwards encrypted SSL data backwards and forwards, and since it doesn't have the private key for the right certificate, it can't impersonate the desired endpoint).
In general, SSL is packet-based protocol (for the purpose of solving your task). If you can have the host forward the packets back and forth, you can easily have SSL-secured communication channel. One thing you need is something like our SSL/TLS components, which allow any transport and not just sockets. I.e. the component tells your code "send this packet to the other side" or "do you have anything for me to receive?" and your code communicates with your intermediate server.
Related
I want to understand how my local IP address (localhost) can be exposed to Internet. For that I've read [here] a method of port forwarding using SSH. Which basically does routing from publicly available server to our localhost using SSH.
But I wonder how the service like 'LocalTunnel' works? In above article it's written as following:
There services (localtunnel for example) creates a tunnel from their server back to port 3000 on your dev box. It functions pretty much exactly like an SSH tunnel, but doesn’t require that you have a server.
I've tried reading code from it's github repository and what I understood is:
These services have a server which is publicly available, which generates different URLs, and when we hit that URL, It forward the request to localhost corresponding to that URL.
So basically it's works like a proxy server.
Is these understanding correct? If yes then what I don't understand is that how these server has access to some localhost running on my computer? How it perform request to it? What I'm missing here? Here is the code which I referred.
The Short Answer (TL;DR)
The Remote (i.e. the localtunnel software on your computer) initializes the connection to the Relay (i.e. localtunnel.me) acts as a multiplexing proxy and when Clients (i.e. web browsers) connect, the relay multiplexes the connections between remotes and clients by appending special headers with network information.
Browser <--\ /--> Device
Browser <---- M-PROXY Service ----> Device
Browser <--/ \--> Device
Video Presentation
If you prefer a video preso, I just gave a talk on this at UtahJS Conf 2018, in which I talk a little about all of the other potential solutions as well: SSH Socksv5 proxies (which you mentioned), VPN, UPnP, DHT, Relays, etc:
Access Ability: Access your Devices, Share your Stuff
Slides: http://telebit.cloud/utahjs2018
How localtunnel, ngrok, and Telebit work (Long Answer)
I'm the author of Telebit, which provides service with similar features to what ngrok, localtunnel, and libp2p provide (as well as open source code for both the remote/client and relay/server to run it yourself).
Although I don't know the exact internals of how localtunnel is implemented, I can give you an explanation of how it's generally done (and how we do it), and it's most likely nearly identical to how they do it.
The magic that you're curious about happens in two places: the remote socket and the multiplexer.
How does a remote client access the server on my localhost?
1. The Remote Socket
This is pretty simple. When you run a "remote" (telebit, ngrok, and localtunnel all work nearly the same in this regard), it's actually your computer that initiates the request.
So imagine that the relay (the localtunnel proxy in your case) uses port 7777 to receive traffic from "remotes" (like your computer) and your socket number (randomly chosen source address on your computer) is 1234:
Devices: [Your Computer] (tcp 1234:7777) [Proxy Server]
Software: [Remote] -----------------------> [Relay]
(auth & request 5678)
However, the clients (such as browsers, netcat, or other user agents) that connect to you actually also initiate requests with the relay.
Devices: [Proxy Service] (tcp 5678) [Client Computer]
Software: [Relay] <------------------------ [netcat]
If you're using tcp ports, then the relay service keeps an internal mapping, much like NAT
Internal Relay "Routing Table"
Rule:
Socket remote[src:1234] --- Captures ------> ALL_INCOMING[dst:5678]
Condition:
Incoming client[dst:5678] --- MATCHES -------> ALL_INCOMING[dst:5678]
Therefore:
Incoming client[dst:5678] --- Forwards To ---> remote[src:1234]
Both connections are "incoming" connections, but the remote connection on the "south end" is authorized to receive traffic coming from another incoming source (without some form of authorized session anyone could claim use of that port or address and hijack your traffic).
[Client Computer] (tcp 5678) [Proxy Service] (tcp 1234) [Your Computer]
[netcat] --------------> <--[Relay]--> <------------ [remote]
2. The Multiplexer
You may have noticed that there's a critical flaw in the description above. If you just route the traffic as-is, your computer (the remote) could only handle one connection at a time. If another client (browser, netcat, etc) hopped on the connection, your computer wouldn't be able to tell which packets came from where.
Browser <--\ /--> Device
Browser <---- M-PROXY Service ----> Device
Browser <--/ \--> Device
Essentially what the relay (i.e. localtunnel proxy) and the remote (i.e. your computer) do is place a header in front of all data that is to be received by the client. It needs to be something very similar to HAProxy's The PROXY Protocol, but works for non-local traffic as well. It could look like this:
<src-address>,<src-port>,<sni>,<dst-port>,<protocol-guess>,<datalen>
For example
172.2.3.4,1234,example.com,443,https,1024
That info could be sent exactly before or append to each data packet that's routed.
Likewise, when the remote responds to the relay, it has to include that information so that the relay knows which client the data packet is responding to.
See https://www.npmjs.com/package/proxy-packer for long details
Sidenote/Rant: Ports vs TLS SNI
The initial explanation I gave using tcp ports, because it's easy to understand. However, localtunnel, ngrok, and telebit all have the ability to use tls servername indicator (SNI) instead of relying on port numbers.
[Client Computer] (https 443) [Proxy Service] (wss 443) [Your Computer]
[netcat+openssl] --------------------> <--[Relay]--> <------------ [remote]
(or web browser) (sni:xyz.somerelay.com) (sni:somerelay.com)
MITM vs p2p
There are still a few different ways you can go about this (and this is where I want to give a shameless plug for telebit because if you're into decentralization and peer-to-peer, this is where we shine!)
If you only use the tls sni for routing (which is how localtunnel and ngrok both work by default last time I checked) all of the traffic is decrypted at the relay.
Anther way requires ACME/Let's Encrypt integration (i.e. Greenlock.js) so that the traffic remains encrypted, end-to-end, routing the tls traffic to the client without decrypting it. This method functions as peer-to-peer channel for all practical purposes (the relay acts as just another opaque "switch" on the network of the Internet, unaware of the contents of the traffic).
Furthermore, if https is used both for remotes (for example, via Secure WebSockets) and the clients, then the clients look just like any other type of https request and won't be hindered by poorly configured firewalls or other harsh / unfavorable network conditions.
Now, solutions built on libp2p also give you a virtualized peer connection, but it's far more indirect and requires routing through untrusted parties. I'm not a fan of that because it's typically slower and I see it as more risky. I'm a big believer than network federation will win out over anonymization (like libp2p) in the long. (for our use case we needed something that could be federated - run by independently trusted parties- which is why we built our solution the way that we did)
I have a question about SSL. As I know, when we use browser to request from https server, it will make an SSL handshake first then all data will be encryption in the connection. But if I make a request without browser (like request module in nodejs, postman...), will it be an SSL handshake and data encryption on the connection?
Anyone know please explain to me, thank you.
First, stop saying SSL. Its successor is TLS, and it will have 20 years next January.
TLS is a protocol sitting on top of TCP typically (other variants can also use UDP), and provides on top of TCP features some new features about endpoints authentication and transport confidentiality and integrity.
In a way, you can understand it as being sandwiched between TCP and the higher level application protocol, like HTTP.
Saying otherwise you can use many others protocols on top of TLS: you have all email related ones (SMTP, IMAP, POP, etc.), you can have FTP on top of it (while probably not a good idea nowadays), XMPP for realtime communications, etc.
In short, any protocol using TCP could use TLS with some adaptation.
So HTTP is one case among others. HTTP is between an HTTP client and an HTTP server, or webserver for short.
A browser is an HTTP client. One among many ones. When you use curl or wget you are also an HTTP client. So if any HTTP client access an http:// link it will first do the TLS handshake, after the TCP connection and before starting to do anything really related to the HTTP protocol.
You have specialized libraries dealing with TLS so that not all program need to recode everything about this again, since it is also complicated.
I'm working on p2p application intended for LAN. Users are supposed to connect directly to each other without any server. At the moment application works on Linux, but I think about porting it on Windows and Android.
At the moment I simply establish unencrypted TCP connection, which is fine in early development stage. Later I'd like to use secure channels and here's my question - how can I establish secure connection (assuring confidence and authentication like TLS) in distributed, p2p solution for LAN? Is it even possible?
I cannot use TLS, because I will not have any central server working as certificate authority. Moreover my LAN have no internet connection. Anonymous TLS is partial solution (will it work on Android?).
Because users will be in the same LAN, I may assume they will be e.g. sitting next to. Therefore they may "physically" authenticate themselves and e.g. share key or part of key (PIN)?
I know p2p software like Skype is using UDP hole punching for that. But what if one of the clients is a web browser which needs to download a file from another client (TCP connection instead of UDP)? Is there any technique for such case?
I can have an intermediate public server which can marry the clients but I can't afford all the traffic between these clients go through this server. The public server can only establish the connection between the clients, like Skype does, and that's all. And this must work via TCP (more exactly, HTTP) to let the downloading client be a web browser.
Both clients must not be required to setup anything in their routers or anything like that.
I'll plan to code this in C/C++ but at the point I'm wondering if this idea is possible at all.
I previously wrote up a very consolidated rough answer on how P2P roughly works with some discussion on various protocols and corresponding open-source libraries. You can read it here.
The reliability of P2P is ultimately a result of how much you invest in it from both a client coding perspective and a service configuration (i.e. signaling servers and relays). You can settle for easy NAT traversal of UDP with no firewall support. Maybe a little more effort and you get TCP connectivity. And you can go "all the way" and have relays that have HTTPS listeners for clients behind the hardest of firewalls to traverse.
As to the answer of your question about firewalls. Depends on how the Firewall is configured. Many firewalls are just glorified NATs with security to restrict traffic to certain ports and block unsolicited incoming connections. Others are extremely restrictive and just allow HTTP/HTTPS traffic over a proxy.
The video conference apps will ultimately fallback to emulating an HTTPS connection over the PC's configured proxy server to port 443 (or 80) of a remote relay server if it can't get directly connected. (And in some cases, the remote client will try to listen on port 80 or port 443 so it can connect direct).
You are absolutely right to assume that having all the clients going through a relay will be expensive to maintain. If your goal is 100% connectivity no matter what type of firewall the clients is behind, some relay solution will have to exist. If you don't support a relay solution, you can invest heavily in getting the direct connectivity to work reliably and only have a small percentage of clients blocked.
Hope this helps.
PeerConnection, part of WebRTC solves this in modern browsers.
Under the hood it uses ICE which is an RFC for NAT hole-punching.
For older browsers, it is possible to use the P2P support in Flash.
I'm writing a piece of P2P software, which requires a direct connection to the Internet. It is decentralized, so there is no always-on server that it can contact with a request for the server to attempt to connect back to it in order to observe if the connection attempt arrives.
Is there a way to test the connection for firewall status?
I'm thinking in my dream land where wishes were horses, there would be some sort of 3rd-party, public, already existent servers to whom I could send some sort of simple command, and they would send a special ping back. Then I could simply listen to see if that arrives and know whether I'm behind a firewall.
Even if such a thing does not exist, are there any alternative routes available?
Nantucket - does your service listen on UDP or TCP?
For UDP - what you are sort of describing is something the STUN protocol was designed for. It matches your definition of "some sort of simple command, and they would send a special ping back"
STUN is a very "ping like" (UDP) protocol for a server to echo back to a client what IP and port it sees the client as. The client can then use the response from the server and compare the result with what it thinks its locally enumerated IP address is. If the server's response matches the locally enumerated IP address, the client host can self determinte that it is directly connected to the Internet. Otherwise, the client must assume it is behind a NAT - but for the majority of routers, you have just created a port mapping that can be used for other P2P connection scenarios.
Further, you can you use the RESPONSE-PORT attribute in the STUN binding request for the server to respond back to a different port. This will effectively allow you to detect if you are firewalled or not.
TCP - this gets a little tricky. STUN can partially be used to determine if you are behind a NAT. Or simply making an http request to whatismyip.com and parsing the result to see if there's a NAT. But it gets tricky, as there's no service on the internet that I know of that will test a TCP connection back to you.
With all the above in mind, the vast majority of broadband users are likely behind a NAT that also acts as a firewall. Either given by their ISP or their own wireless router device. And even if they are not, most operating systems have some sort of minimal firewall to block unsolicited traffic. So it's very limiting to have a P2P client out there than can only work on direct connections.
With that said, on Windows (and likely others), you can program your app's install package can register with the Windows firewall so your it is not blocked. But if you aren't targeting Windows, you may have to ask the user to manually fix his firewall software.
Oh shameless plug. You can use this open source STUN server and client library which supports all of the semantics described above. Follow up with me offline if you need access to a stun service.
You might find this article useful
http://msdn.microsoft.com/en-us/library/aa364726%28v=VS.85%29.aspx
I would start with each os and ask if firewall services are turned on. Secondly, I would attempt the socket connections and determine from the error codes if connections are being reset or timeout. I'm only familiar with winsock coding, so I can't really say much for Linux or mac os.