Recently I am working on a p2p application and i found out how to program UDP holepunching with c# using ICE and STUN protocols.
Its good but there is some limits for symmetric NATs and routers using symmetric NAT.\
I am looking for a way to connect two system over any NAT type .
Anyway I have tried UPNP and port forwarding too but due to firewall and router blacklist this option is out of my mind, if its your answer please describe how should I use this method correctly.
Thanks in advance.
Upnp and NAT-PMP are quite useful for those routers which support these protocols, I haven't meet any problem like "firewall and router blacklist".
In circumstances that router doesn't support those protocols, ICE is what I use. As you said, symmetric NAT is a type that keeps highest security, and if the port increases randomly it is hard to punch a hole. Using a server as a relay (ICE can do it) is much easier than overcome random algorithm.
If you are interested in the source code of symmetric NAT, it is in some file like nf_nat_core.c, it's easy to trace and understand how it works.
Related
I investigated a lot about this topic but most of the guides just teach how to exchange data between devices on the same network and, regarding exchanging data between devices on different networks, no source was totally clear to me. I hope with this question somebody can give me (and other users) a good overview. If you have any guide or book about it I’d be super interested (for Java would also be fine).
First of all I’m interested in the difference between programs that
need to exchange data quickly (it may be an online videogame) versus
programs that need to exchange data accurately (it may be a message
app). My understanding is that the difference between the two is the
protocol used: in the first case is UDP (where no checks are done to
ensure there is no packets loss), in the second case is TCP (where
checks are done and data is exchanged more slowly). Is this correct?
So in an hypothetical Python script in the first case the socket
created would look like this:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
While in the second case would look like this:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
My understanding is that to exchange data between different networks
you have to use port forwarding (very good explanation here),
concept that is clear to me. However, do you have any source that
suggests how to do it in Python? Also, is port forwarding
everything you need to do in order to exchange data between
different networks? Finally, I’m not sure I understand the role UPnP
plays in port forwarding. Based on this question it seems UPnP
is a way to automatically port forwarding. Is it correct? Can I use
miniupnpc library to do it automatically?
Finally, if I switch off and on my router, the private IP addresses
assigned to the devices connected to the network change (so the
private IP of my phone connected to my home WiFi could change, for
example, from 192.168.1.2 to 192.168.1.11). Does this represent a
problem in networking programming? If I set on the router a certain
port and the traffic that comes to that port is directed to a
certain private IP address and then this IP changes I suppose there is a
problem. Is this correct? If it is what is the solution?
Your understanding of use cases for UDP and TCP seem roughly
accurate. UDP ensure lower latency (not always) so for apps that
require lowest latency possible while also not caring about missed
packets, UDP is used. So if you think about video streaming, once a
packet is missed, it makes no sense to hold up every future packet
for that one old packet. This is because a small amount of data that
is missed doesn't really affect a user's watching experience. For
gaming, we want the newest data as soon as possible, so waiting for
old data also doesn't matter. But if you're implementing a protocol
or something that requires all data to be transmitted, TCP makes
sense since its absolutely vital that all information gets to the
receiver and in order.
There are a few methods to exchange data between two private networks. Port forwarding is certainly one method, and both machines on either network would have to have port forwarding. I don't know anything about automated port forwarding like you mention, but you can go into your router settings and set it up pretty easily. Another method of talking across networks is something like webRTC. Its a protocol that uses the STUN TURN and ICE protocols to perform something called NAT traversal. Short story shorter, it tricks your routers into letting your machines talk to each other(analogous to temporary port forwarding).
You're right that this could be an issue. However you should be able to setup static IP addresses in your router. So you can assign one machine to have a static IP address, setup port forwarding, and bam you have a permanent(hopefully) open connection.
I have created an P2P application which is FULLY decentralized and is using a Kademlia algorithm to make it so. This has been tested on the local network and it completely successful.
I heard about UDP Hole Punching, however hole punching requires the peer to know about the clients IP and vice versa, however as said due to design it is impossible. As it would require each peer to keep on asking the supernode\server for new arrivals and keep them assigned and so naturally it is not too dependable especially if the supernode is down.
Due to the design of the algorithm 1 peer does not know the other peers IP address so I need 1 peer to completely open a port up for PUBLIC to be able to connect to, how can I do this, in Windows? could someone perhaps also give me suitable links that might give me a direction?
It would be preferable (but not absolute) if they use c++\c as example
I think you will have to change your architecture a little. There is no other way for NAT traversal unless you configure your NAT for port forwarding (I think you don't want this). You might need to implement three layers:
Lower layer 1: it knows about IP addresses and ports and can solve problems like hole punching or dealing with servers (which could be down but first you have no choice and second you can add alternatives for connectivity).
Layer 2: implements special naming, addressing and location services for your solution (instead of using IP addresses).
Upper layer 3: implements your p2p solution using lower layer naming and location services.
First of all, you need to examine your design. If it needs 100% connectivity between all nodes (without relays) it's probably going to fail under IPv4 since not all NATs are traversable. And possibly under IPv6 due to stateful firewalls.
Now, for the nat traversal: A solution for DHT-assisted NAT traversal is for NATed nodes to have a rendezvous node.
To keep the UDP NAT mappings open it has to regularly ping that rendezvous node.
Additionally it has to announce the address of the rendezvous point on the DHT, e.g. under hash("rendezvous" + node ID) or simply on its reachable neighbor nodes.
The rendezvous node can then act as coordination point for hole punching.
This does not require any special "supernode", just other (possibly multiple, thus eliminating the SPOF) nodes in the network that are not NATed and can assist.
Additional mechanisms such as UPnP IGP, NAT-PMP, PCP and ultimately instructing users to forward the needed ports can also help to reduce the need for nat traversal.
Due to the design of the algorithm 1 peer does not know the other peers IP address so I need 1 peer to completely open a port up for PUBLIC to be able to connect to
First of all this will only work for Full cone NAT. For other types of NAT that public IP:Port you open will only work for specific destination. In your case you don't know the destination so it's impossible.
In case of full cone NAT, you send a packet to a random address with low TTL value so that the packet drops in the middle and doesn't reach that address. If it reaches than that address's NAT might block you. If you do this then a port will be opened for anybody to send you a packet. You need to keep sending packets after some short interval for that port to be remained open. Here is a problem that you can't choose which port to open in the NAT. The NAT will assign you a free port on its own.
Finally I don't see any point doing any of the above if peers can't exchange their IP information with each other. You should use a signalling protocol like SIP or XMPP to exchange IP information between peers.
To learn more about NATs please read this answer.
I know p2p software like Skype is using UDP hole punching for that. But what if one of the clients is a web browser which needs to download a file from another client (TCP connection instead of UDP)? Is there any technique for such case?
I can have an intermediate public server which can marry the clients but I can't afford all the traffic between these clients go through this server. The public server can only establish the connection between the clients, like Skype does, and that's all. And this must work via TCP (more exactly, HTTP) to let the downloading client be a web browser.
Both clients must not be required to setup anything in their routers or anything like that.
I'll plan to code this in C/C++ but at the point I'm wondering if this idea is possible at all.
I previously wrote up a very consolidated rough answer on how P2P roughly works with some discussion on various protocols and corresponding open-source libraries. You can read it here.
The reliability of P2P is ultimately a result of how much you invest in it from both a client coding perspective and a service configuration (i.e. signaling servers and relays). You can settle for easy NAT traversal of UDP with no firewall support. Maybe a little more effort and you get TCP connectivity. And you can go "all the way" and have relays that have HTTPS listeners for clients behind the hardest of firewalls to traverse.
As to the answer of your question about firewalls. Depends on how the Firewall is configured. Many firewalls are just glorified NATs with security to restrict traffic to certain ports and block unsolicited incoming connections. Others are extremely restrictive and just allow HTTP/HTTPS traffic over a proxy.
The video conference apps will ultimately fallback to emulating an HTTPS connection over the PC's configured proxy server to port 443 (or 80) of a remote relay server if it can't get directly connected. (And in some cases, the remote client will try to listen on port 80 or port 443 so it can connect direct).
You are absolutely right to assume that having all the clients going through a relay will be expensive to maintain. If your goal is 100% connectivity no matter what type of firewall the clients is behind, some relay solution will have to exist. If you don't support a relay solution, you can invest heavily in getting the direct connectivity to work reliably and only have a small percentage of clients blocked.
Hope this helps.
PeerConnection, part of WebRTC solves this in modern browsers.
Under the hood it uses ICE which is an RFC for NAT hole-punching.
For older browsers, it is possible to use the P2P support in Flash.
I'm developing an application that accepts connections from clients over the internet. All these clients are on fixed IP addresses and will establish a non-encrypted TCP connection.
The current plan is for the server to check which IP the connections come from and allow only client connections from a list of know IPs?
How safe is that against IP spoofing?
My reasoning is that since this is a TCP connection, an attacker couldn't just fake its sender IP (which is easy), but would have to assure that the packets travel back to him and thus he would have to hack all routers on the path, which seems rather hard.
I know I could use encryption, like SSH, but lets stick with the question of how safe the plain TCP connection would be.
Restricting connections by IP address is generally a good practice when practical, as it greatly reduces the attack surface and makes the complexity of an attack much higher. As stated in other answers, you would now have to do something like IP spoofing, or attacking the network itself (false BGP routes, etc).
That said, IP address restriction should be used as one layer of a defense-in-depth approach. Could you encrypt the TCP stream without too much rework? Maybe SSL? If you can't modify the program, how about the network? Site ti site IPSEC VPN tunnels are not difficult to establish, as almost any commercial firewall supports them. Even some soho routers can be modified to support IPSEC (with OpenWrt plus OpenSwan, for example).
Lastly, could you require the client and server to mutually authenticate?
Not safe. BGP gateways are not immune to attack, and with that, false routes can be advertised and IPs can be spoofed.
First of all, using the IP you are not identifying the client, but just some numbers. Even if the IP is right, there still can be a troyan on user's computer, authenticating in place of the user itself (as I don't know what kind of service you provide, I assume that this might make sense).
Now, if one has access to one of the routers via which the packets between the client and the server go, then he can do almost anything - he can send and receive packets in the name of the client or he can modify them (as the data goes unencrypted). Moreover, the attacker doesn't need to hack all or one of routers - he just needs to have access (including legitimate one) to the channel where the data goes, be it the router itself or the cable (which can be cut and the router can be inserted).
So to summarize, IP can be used as one of the component that hardens spoofing to some extent, but it can't be the main security measure.
How can an application be designed such that two peers can communicate directly with each other (assuming both know each other's IPs), but without outgoing connections? That's, no ports will be opened. Bitorrent for example does it, but multiplayer games (as far as I know) require port forwarding.
I'm not sure what you mean by No Outgoing Connections, I'm going to assume like everyone else you meant no Incoming Connections (they are behind a NAT/FW/etc).
The most common one mentioned so far is UPNP, which in this context is a protocol that allows you as a computer to talk to the Gateway and say forward me this port because I want someone on the outside to be able to talk to me. UPNP is also designed for other things, but this is the common thing for home networking (Actually it's one of many definitions).
There are also more common and slightly more reliable ways if you don't own the network. The most common is called STUN but if I recall correctly there are a few variants. Basically you use a third party server that allows incoming connections to try and coordinate a communication channel. Basically, what you do is send a UDP packet to you're peer, which will open up you're NAT for a response, but gets dropped on you're peer's NAT (since no forwarding rule exists yet). Through the connection to the intermediary, they are then told to do the same, which now opens up their NAT, and matches the existing rule in you're NAT. Now the communications can proceed. Their is a variant of this which will allow a TCP/IP connection as well by sending SYN and SYN-ACK messages with some coordination.
The Wikipedia articles I've linked to has links to the relevant rfc's for these protocols on precisely how they work. Essentially it comes down to, there isn't an easy answer, as this is a very network centric problem.
You need a "meeting point" in the network somewhere: the participants "meet" at a "gateway" of some sort and the said "gateway function" takes care of the forwarding.
At least that's one way of doing it: I won't try to comment on the details of Bittorrent... I am sure you can google for links.
UPNP dealt with this mostly in the recent years, but the need to open ports is because the application has been coded to listen on a specific port for a response.
Ports beneath 1024 are called "registered" because they've been assigned a port number because a company paid for it. This doesn't mean you couldn't use port 53 for a webserver or SSH, just that most will assume when they see it that they are dealing with DNS. Ports above 1024 are unregistered, so there's no association - your web browser, be it Internet Explorer/Firefox/etc, is using an unregistered port to send the request to the StackOverflow webserver(s) on port 80. You can use:
netstat -a
..on windows hosts to see what network connections are currently established, including the port involved.
UPNP can be used to negotiate with the router to open and forward a port to your application. Even bit-torrent needs at least one of the peers to have an open port to enable p2p connections. There is no need for both peers to have an open port however, since they both communicate with the same server (tracker) that lets them negotiate and determine who has an open port.
An alternative is an echo-server / relay-server somewhere on the internet that both peers trust, and have that relay all the traffic.
The "problem" with this solution is that the echo-server needs to have lots of bandwidth to accomodate all connected peers since it relays all the traffic rather than establish p2p connections.
Check out EchoWare: http://www.echogent.com/tech.htm