I have created an P2P application which is FULLY decentralized and is using a Kademlia algorithm to make it so. This has been tested on the local network and it completely successful.
I heard about UDP Hole Punching, however hole punching requires the peer to know about the clients IP and vice versa, however as said due to design it is impossible. As it would require each peer to keep on asking the supernode\server for new arrivals and keep them assigned and so naturally it is not too dependable especially if the supernode is down.
Due to the design of the algorithm 1 peer does not know the other peers IP address so I need 1 peer to completely open a port up for PUBLIC to be able to connect to, how can I do this, in Windows? could someone perhaps also give me suitable links that might give me a direction?
It would be preferable (but not absolute) if they use c++\c as example
I think you will have to change your architecture a little. There is no other way for NAT traversal unless you configure your NAT for port forwarding (I think you don't want this). You might need to implement three layers:
Lower layer 1: it knows about IP addresses and ports and can solve problems like hole punching or dealing with servers (which could be down but first you have no choice and second you can add alternatives for connectivity).
Layer 2: implements special naming, addressing and location services for your solution (instead of using IP addresses).
Upper layer 3: implements your p2p solution using lower layer naming and location services.
First of all, you need to examine your design. If it needs 100% connectivity between all nodes (without relays) it's probably going to fail under IPv4 since not all NATs are traversable. And possibly under IPv6 due to stateful firewalls.
Now, for the nat traversal: A solution for DHT-assisted NAT traversal is for NATed nodes to have a rendezvous node.
To keep the UDP NAT mappings open it has to regularly ping that rendezvous node.
Additionally it has to announce the address of the rendezvous point on the DHT, e.g. under hash("rendezvous" + node ID) or simply on its reachable neighbor nodes.
The rendezvous node can then act as coordination point for hole punching.
This does not require any special "supernode", just other (possibly multiple, thus eliminating the SPOF) nodes in the network that are not NATed and can assist.
Additional mechanisms such as UPnP IGP, NAT-PMP, PCP and ultimately instructing users to forward the needed ports can also help to reduce the need for nat traversal.
Due to the design of the algorithm 1 peer does not know the other peers IP address so I need 1 peer to completely open a port up for PUBLIC to be able to connect to
First of all this will only work for Full cone NAT. For other types of NAT that public IP:Port you open will only work for specific destination. In your case you don't know the destination so it's impossible.
In case of full cone NAT, you send a packet to a random address with low TTL value so that the packet drops in the middle and doesn't reach that address. If it reaches than that address's NAT might block you. If you do this then a port will be opened for anybody to send you a packet. You need to keep sending packets after some short interval for that port to be remained open. Here is a problem that you can't choose which port to open in the NAT. The NAT will assign you a free port on its own.
Finally I don't see any point doing any of the above if peers can't exchange their IP information with each other. You should use a signalling protocol like SIP or XMPP to exchange IP information between peers.
To learn more about NATs please read this answer.
Related
I investigated a lot about this topic but most of the guides just teach how to exchange data between devices on the same network and, regarding exchanging data between devices on different networks, no source was totally clear to me. I hope with this question somebody can give me (and other users) a good overview. If you have any guide or book about it I’d be super interested (for Java would also be fine).
First of all I’m interested in the difference between programs that
need to exchange data quickly (it may be an online videogame) versus
programs that need to exchange data accurately (it may be a message
app). My understanding is that the difference between the two is the
protocol used: in the first case is UDP (where no checks are done to
ensure there is no packets loss), in the second case is TCP (where
checks are done and data is exchanged more slowly). Is this correct?
So in an hypothetical Python script in the first case the socket
created would look like this:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
While in the second case would look like this:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
My understanding is that to exchange data between different networks
you have to use port forwarding (very good explanation here),
concept that is clear to me. However, do you have any source that
suggests how to do it in Python? Also, is port forwarding
everything you need to do in order to exchange data between
different networks? Finally, I’m not sure I understand the role UPnP
plays in port forwarding. Based on this question it seems UPnP
is a way to automatically port forwarding. Is it correct? Can I use
miniupnpc library to do it automatically?
Finally, if I switch off and on my router, the private IP addresses
assigned to the devices connected to the network change (so the
private IP of my phone connected to my home WiFi could change, for
example, from 192.168.1.2 to 192.168.1.11). Does this represent a
problem in networking programming? If I set on the router a certain
port and the traffic that comes to that port is directed to a
certain private IP address and then this IP changes I suppose there is a
problem. Is this correct? If it is what is the solution?
Your understanding of use cases for UDP and TCP seem roughly
accurate. UDP ensure lower latency (not always) so for apps that
require lowest latency possible while also not caring about missed
packets, UDP is used. So if you think about video streaming, once a
packet is missed, it makes no sense to hold up every future packet
for that one old packet. This is because a small amount of data that
is missed doesn't really affect a user's watching experience. For
gaming, we want the newest data as soon as possible, so waiting for
old data also doesn't matter. But if you're implementing a protocol
or something that requires all data to be transmitted, TCP makes
sense since its absolutely vital that all information gets to the
receiver and in order.
There are a few methods to exchange data between two private networks. Port forwarding is certainly one method, and both machines on either network would have to have port forwarding. I don't know anything about automated port forwarding like you mention, but you can go into your router settings and set it up pretty easily. Another method of talking across networks is something like webRTC. Its a protocol that uses the STUN TURN and ICE protocols to perform something called NAT traversal. Short story shorter, it tricks your routers into letting your machines talk to each other(analogous to temporary port forwarding).
You're right that this could be an issue. However you should be able to setup static IP addresses in your router. So you can assign one machine to have a static IP address, setup port forwarding, and bam you have a permanent(hopefully) open connection.
How we can share any type of data over the two separate networks without violating security mechanisms using UDP ?
There are a few things you'll have to remember:
Every network has its firewall, and it depends on the firewall rules, whether to allow your traffic into the network or not. First, ask your client or receiver to make changes in the firewall so that it accepts your IP address and also remember most of the systems have an edge firewall too.
Be clear with the type of connection i.e., p2p (or) server & client. It's better if it is a client & server type connection.
UDP by definition is NOT a connection-oriented protocol, so there is no state to keep track of as far as OSI layers 2-4 are concerned. All incoming UDP connections are treated as "new" or the same.
Also, see that none of the systems is under NAT connection, as the router will remember the IP and port of the device just for a while. And if there is any delay in response from client-side then the system under NAT will not know the IP or the port of the device, where it is supposed to send the traffic.
I recall reading an article about a proposed way to do this. If I recall correctly, the researchers successfully created a connection to a client on another network without port forwarding by sending HTTP packets to each other (Alice pretends that Bob is an HTTP web server while Bob pretends Alice is a web server).
I'm not sure if that makes sense, but does anyone know where I can find the article or does anyone have any other ideas how to connect two clients together without a central server or port forwarding?
Is it even possible?
Edit: I would know the IPs of both computers and port the program listens on.
It is possible. I see at least 2 parts to your question. (It is not going to be HTTP packet. It is a lot more complex than that.)
First off, I believe you might be talking about a concept called decentralized P2P network. The main idea behind a decentralized peer-to-peer network is the fact that nodes conjoint in such a network will not require central server or group of servers.
As you might already know, most common centralized peer-to-peer networks require such centralized system to exchange and maintain interconnectivity among nodes. The basic concept is such, a new node will connect to one of the main servers to retrieve information about other nodes on the network to maintain its connectivity and availability. The central system gets maintained through servers constantly synchronizing network state, relevant information, and central coordination among each other.
Decentralized network, on the other hand, does not have any structure or predetermined core. This peer-to-peer model is also called unstructured P2P networks. Any new node will copy or inherit original links from the "parent" node and will form its own list over time. There are several categories of decentralization of such unstructured networks.
Interestingly enough, the absence of central command and control system makes it solution of choice for modern malware botnets. A great example could be Storm botnet, which employed so-called Passive P2P Monitor (PPM). PPM was able to locate the infected hosts and build peer list regardless whether or not infected hosts are behind a firewall or NAT. Wikipedia's article Storm botnet is an interesting read. There is also great collaborative study called Towards Complete Node Enumeration in a Peer-to-Peer Botnet, which provides excellent conceptual analysis and techniques employed by Storm botnet network.
Second of all, you might be talking about UDP hole punching. This is a technique or algorithm used to maintain connectivity between 2 hosts behind NATed router/gateway using 3rd comment host by means of a third rendezvous server.
There is a great paper by Bryan Ford, Pyda Srisuresh, and Dan Kegel called Peer-to-Peer Communication Across Network Address Translators.
As answered, a peer-to-peer connection requires establishment of a connection between two (presumably) residential computers, which will necessitate punching holes through both of their firewalls. For a concrete example of hole punching, see pwnat: "The only tool to punch holes through firewalls/NATs without a third party". The process, put simply, goes like this:
The "server" (who doesn't know the client's IP address, but the client knows the server's) pings a very specific ICMP Echo Request packet to 1.2.3.4 every 30 seconds. The NAT, during translation, takes note of this packet in case it gets a response.
The client sends an ICMP Time Exceeded packet to the server, which is a type of packet that usually contains the packet that failed to deliver. The client, knowing in advance the exact packet that the server has been sending to 1.2.3.4, embeds that whole packet in the Data field.
The NAT recognizes the Echo Request packet and happily relays the whole Time Exceeded packet, source IP and all, to the correct user, i.e. the server. Voila, now the server knows the client's IP and port number.
Now that the server knows the address, it begins to continually send UDP packets to the client, despite the fact that the client's NAT did not expect them and will therefore ignore them all.
The client begins sending UDP packets to the server, which will be recognized by the server's NAT as a response to the server's packets and route them appropriately.
Now that the client is sending UDP packets to the server, the server's stream of UDP packets starts getting properly routed by the client's NAT.
And, in 6 easy steps, you have established a UDP connection between a client and a server penetrating two residential firewalls. Take that, ISP!
I am studying the TCP hole punching technique and have got some knowledge from this article: TCP_hole_punching. But I don't understand below part:
Network Drawing
Peer A ←→ Gateway A ← .. Network .. → Gateway B ←→ Peer B
Types of NAT
The availability of the TCP-hole-punching technique depends
on the type ofcomputer port allocation used by the NAT. When
two peers, A and B, instantiate TCP connections by binding
to local ports Pa and Pb, respectively, **they need to know
the remote endpoint NAT port in order to make the connection**.
Here are some questions, could somebody help explain? Any help or suggestion will be greatly appreciated!!
Q1. Suppose we have a client app running behind NAT1 and a server app
running behind NAT2. They will have messaging and file transfer communication.
The pattern will be the server listens and accepts connection
from the client. Do they need to use the TCP hole punching to keep the tcp
connection work?
Q2. Is the TCP hole punching technique necessary only when it's peer to peer
communication and both parties are connecting to each other (e.g., 2x CONNECT(), no LISTEN(), ACCEPT(), etc.)?
Q3. In the above text, what does it mean of "they need to know the remote endpoint NAT port in order to make the connection"? Why do they need to know the remote endpoint NAT port?Don't they only need to know the local port of the other machine?
For example, suppose the pair are (ClientA, publicIPA, LocalPortA), (ClientB, publicIPB, LocalPortB). If the ClientA wants to communicate with ClientB via TCP, it may do something like this:
clientBAddr.port = LocalPortB;
clientBAddr.ip = inet_addr(publicIPB);
connect(fdA, clientBAddr,...);
why would it need to know information such as NATPortA and NATPortB? Is there any problem if the clientA and clientB don't care about the NATA and NATB?
Since this post is reopened and nobody leaves an answer until this moment, I would like to answer it myself based on this article: Peer-to-Peer Communication Across Network Address Translators
About Q1, yes. Without mechanism like hole-punching, the packets one side use to sync with the other side will be dropped by the routers.
About Q2, no exactly. Listening sockets won't work directly. You need other tricks.
How can an application be designed such that two peers can communicate directly with each other (assuming both know each other's IPs), but without outgoing connections? That's, no ports will be opened. Bitorrent for example does it, but multiplayer games (as far as I know) require port forwarding.
I'm not sure what you mean by No Outgoing Connections, I'm going to assume like everyone else you meant no Incoming Connections (they are behind a NAT/FW/etc).
The most common one mentioned so far is UPNP, which in this context is a protocol that allows you as a computer to talk to the Gateway and say forward me this port because I want someone on the outside to be able to talk to me. UPNP is also designed for other things, but this is the common thing for home networking (Actually it's one of many definitions).
There are also more common and slightly more reliable ways if you don't own the network. The most common is called STUN but if I recall correctly there are a few variants. Basically you use a third party server that allows incoming connections to try and coordinate a communication channel. Basically, what you do is send a UDP packet to you're peer, which will open up you're NAT for a response, but gets dropped on you're peer's NAT (since no forwarding rule exists yet). Through the connection to the intermediary, they are then told to do the same, which now opens up their NAT, and matches the existing rule in you're NAT. Now the communications can proceed. Their is a variant of this which will allow a TCP/IP connection as well by sending SYN and SYN-ACK messages with some coordination.
The Wikipedia articles I've linked to has links to the relevant rfc's for these protocols on precisely how they work. Essentially it comes down to, there isn't an easy answer, as this is a very network centric problem.
You need a "meeting point" in the network somewhere: the participants "meet" at a "gateway" of some sort and the said "gateway function" takes care of the forwarding.
At least that's one way of doing it: I won't try to comment on the details of Bittorrent... I am sure you can google for links.
UPNP dealt with this mostly in the recent years, but the need to open ports is because the application has been coded to listen on a specific port for a response.
Ports beneath 1024 are called "registered" because they've been assigned a port number because a company paid for it. This doesn't mean you couldn't use port 53 for a webserver or SSH, just that most will assume when they see it that they are dealing with DNS. Ports above 1024 are unregistered, so there's no association - your web browser, be it Internet Explorer/Firefox/etc, is using an unregistered port to send the request to the StackOverflow webserver(s) on port 80. You can use:
netstat -a
..on windows hosts to see what network connections are currently established, including the port involved.
UPNP can be used to negotiate with the router to open and forward a port to your application. Even bit-torrent needs at least one of the peers to have an open port to enable p2p connections. There is no need for both peers to have an open port however, since they both communicate with the same server (tracker) that lets them negotiate and determine who has an open port.
An alternative is an echo-server / relay-server somewhere on the internet that both peers trust, and have that relay all the traffic.
The "problem" with this solution is that the echo-server needs to have lots of bandwidth to accomodate all connected peers since it relays all the traffic rather than establish p2p connections.
Check out EchoWare: http://www.echogent.com/tech.htm