In my new weekend project I decided to write an bittorrent client from scratch, no ready to use libraries at all. After two days looking for documentation I'm already about to give up :smile:. I know there are the BEPs, but they are far from enough to understand all the specification. After reading a lot more I think the tracker and peer protocols seems to be old and easy to understand/implement (yes, I know, to write a good code with balance, peer selection, optimizations, this is not easy as I just said, but all I want to is to do the basics to learn, not to compete with tens of good clients out there.)
So, I have decided to start by the DHT which seems to the the more complex part and also the less documented. When you stop looking for bittorrent DHT or mainline DHT and start looking for kademlia DHT you have a lot more information but it not so obvious how to put it all together.
Here is what I understand so far (and there are gaps which I hope to fill in):
I start with my DHT tree empty
use find_nodes on my bootstrap node
add the received nodes to my own tree, so I can then select the ones closer to my own ID
start issuing find_nodes to the selected ones and add their responses to my tree
go back to 3 until I stop receiving unknown/new nodes
if I receive an announce_peer with an info_hash than I should save its information on a local DB (the info_hash and ip/port of the sender)
if a node uses get_peers with an info_hash I have in my DB then I send the information otherwise I should send a list of closer nodes I have in my own tree (closest to that info_hash)
when I use get_peers on other nodes I will receive peers or nodes, in the later case I think the nodes are closer to the info_hash and not to my own nodeId so, should I add these nodes to my tree or start a new tree based on them?
when I want to announce I am interested on an info_hash should I use announce_peer everywhere or just to the nodes with nodeId closer to the target info_hash? How much is closer enough?
At this point I have a lot of nodes which IDs are closer to my own ID, and informations about info_hash'es I am not really interested.
I am afraid that I have a giant stupid question: why I did that?
I mean: my selfish reason to do all this work is to locate peers to the info_hash I'm interested in. I understand that the information of one info_hash is likely to be saved on a node which ID is closer to that info_hash. So my chances to find its information is bigger if I create a tree of nodes closer to the info_hash and not closer to my own ID (at this point, if you know the subject, you already noticed how lost I am).
Should I create multiples trees? One for me (to be there to save the information of info_hashes closer to my nodeID people send me), and other tree closer to each one of my target info_hashes so I can retrieve their information?
Should I create a single tree closer to my node ID and hope for the best when querying this tree for the info_hashes I need?
Should I give up since I have completely misunderstood the idea behind DHT at all?
Well, any real documentation, flowcharts, any thing will be welcome!
So, I have decided to start by the DHT which seems to the the more complex part and also the less documented.
The original kademlia paper "Kademlia: A Peer-to-peer Information System Based on the XOR Metric" by Peter Maymounkov and David Mazieres is required reading. It is referenced fairly early in BEP-5
if I receive an announce_peer with an info_hash than I should save its information on a local DB (the info_hash and ip/port of the sender)
You only accept announces when they contain a token previously handed out via get_peers.
when I use get_peers on other nodes I will receive peers or nodes, in the later case I think the nodes are closer to the info_hash and not to my own nodeId so, should I add these nodes to my tree or start a new tree based on them?
You use a temporary tree - or a list ordered by contact-ID relative to the target ID - for iterative lookups since they are not balanced towards your node ID.
when I want to announce I am interested on an info_hash should I use announce_peer everywhere or just to the nodes with nodeId closer to the target info_hash? How much is closer enough?
You perform a get_peers lookup and when it is done you announce to the 𝑲 closest nodes set that returned a write token and verify the responses to make sure you actually get 𝑲. In case of bittorrent 𝑲 = 8.
my selfish reason to do all this work is to locate peers to the info_hash I'm interested in. I understand that the information of one info_hash is likely to be saved on a node which ID is closer to that info_hash. So my chances to find its information is bigger if I create a tree of nodes closer to the info_hash and not closer to my own ID (at this point, if you know the subject, you already noticed how lost I am).
When doing lookups you do not just visit nodes in your routing table, you also visit nodes included in the responses. This makes them iterative. The bias of each node's routing table towards their own ID ensures that the responses include neighbors closer and closer towards the target.
So the deal is that you are responsible for information close to your node ID and other nodes will provide information close to their node IDs that you are interested in. So your routing table layout serves others, their routing table layout serves you.
Note that all the information contained in this answer can be found in the BEP or Kademlia paper.
Related
I am new to this tech but I am close to the concept of Channel. I understand that the channel are ledgers within a bigger ledger( i.e. small chain specific to some users only within the entire blockchain). Based on this I have few questions.
Lets say there are retail markets where there is scope for negotiations in rate and hence each vendor can sale same thing to different people with different prices. So assuming that there are 1000 people and 1000 vendors possibly there could be many channels. How these are managed?
I understand that the blockchains are linear data structure (unless two blocks are created simultaneously, there is a word for this but I forgot it), If I have to access previous data for certain user. How efficient it is going to be for such operations? i.e. take an example of bank credit card transactions. If I want to see all my transactions for last 5 months. How efficient it is going to be?
I am not claiming that I am completely correct in my understanding but these things are bugging me.It would be very nice if someone help me clear these?
Thanks :)
Update
I have gone through this link it also talks about something related to my questions in below comments. i.e. related to businesses (questions private blockchain).
I like more the definition of channels as different blockchains using a common network or common parts of a network. It allows privacy and different organisation structures.
Yes, it could have possibly so many channels as vendor-people pair. Of course, it depends on which privacy you want to achieve.
If you want to access previous datas for previous operations, you have the possibility to have a state database running alongside the peers, which are a state representation of the linear structure. You can use LevelDB or CouchDB. CouchDB allows you to use complex rich queries to access your data.
ledger can only exists in channel. And the channel is composed of peers. the peers out of the channel cannot get access to the ledger data.
I am not sure what you mean by "Operation". You can create it and close it(this will be supported in later version) When you need it ,create it and you can close after use.
If the business is independent, you have to create so many channels. you just say vendor can sale same thing to different people with different prices. . each of them will be used for different scenario.
if you want to get access to the previous data, of course you can. But not maybe as efficient as you the bank card now. But this is doable. Now we use sdk to access. maybe later with more development of the sdk, more graphic tool will be developed, enabling it used as easy as possible.
The Wikipedia page of BitTorrent says regarding Multitracker torrents, "One disadvantage to this is that it becomes possible to have multiple unconnected swarms for a single torrent where some users can connect to one specific tracker while being unable to connect to another. This can create a disjoint set which can impede the efficiency of a torrent to transfer the files it describes."
Can someone please give me an example of this?
Thanks.
The information on the wikipedia page is old and no longer relevant.
Split swarms was only a problem between the introduction of:
Multi-trackers extension:BEP12 around 2004
and the introduction of:
PeerEXchange(PEX):BEP11 and the DHT distributed tracker:BEP5 around 2005.
These three extensions working together creates a single unified swarm.
I know how to implement union find in general, but I was thinking of whether there would be a way to utilize the set structure in python to achieve the same result.
For example, we can union sets pretty easily. But I'm not sure how to determine if two elements are in the same set using just sets.
So, I am wondering if there is a data structure in python that would support such operation, other than the usual implementation?
You could always solve this problem by visualizing it as a tree and its nodes connecting to each other via the root, and then looking up the tree if you want to know if two nodes are connected. If the two nodes you are comparing has the same root (they are in the same tree), than they are connected.
To connect two nodes, just go to the root of each tree they are in, and make one root become the parent of the other.
This video will give you a great intuition about it:
https://www.youtube.com/watch?v=YIFWCpquoS8&list=PLUX6FBiUa2g4YWs6HkkCpXL6ru02i7y3Q&index=1
The connection between the tree nodes can be made via pointers in a language which supports it, but if your language dont (python), than you can create your own pointers by storing positions and links via an array.
The array would be such that its positions would represent your nodes, and the values inside it represents the connection of the specific node to its root. On the beginning, the position in the array is filled with the node number because the nodes has initially no parent, but as you connect nodes, the roots changes, and the array has to represent this. Actually, the value stored there is the identificator of the root.
But try visualizing the problem visually first instead of thinking of arrays and too much mathematical artificats. Visually dealing with it makes the solution sound banal, and can be a good guidance while writing code.
I say this because I have watched the video from Robert Sedgewick I just posted, with a graphical simulation of the solution, and implemented myself without paying too much attention to the code on his book. The intuition the video gave me is much more valuable than any mathematics.
It will help you to encapsulate the nodes into a class, with the following methods:
climbTreeFromNodeUpToRoot
setNewParentToThisNodeAndUpdateHeights
The first method, as the name says, takes you from a node and goes up the tree until finding the root of it, which is then returned.
If you compare two nodes with this method (actually, the roots returned by it), you know easily if they are connected by just comparing their roots.
Once you want to connected them, you go up the trees of both nodes, and ask one root to take the other one as its parent.
The trees can grow very big in height (sorry I dont use the official nomeclature, but this is the one that makes sense to me), so this simple approach will get very slow when you have to climb the tree at a later time.
To prevent trees from becoming to high, dont just set one root as the parent to another without criterium, but attach the smallest tree (in terms of height, not quantity of elements) to the highest one.
For this, you need to know the heights of each tree, and this information you can store on their respective root (via an extra array in your case, or an extra pointer from each node in other languages). This information should be updated everytime another tree connects to it.
It is not possible for a tree to know that she just got a new tree attached to it, so its important that every tree attaching to a second one informs the second as to update its height.
This information can be sent to the root of the second tree, and later used to judge (as writen before) which tree is the smallest. Remember, attaching a small tree to a big one instead of the opposite will save you incredible amounts of time.
Do you want something like this?
myset = ...
all(elt in myset for elt in (a,b))
I'm getting familiar with DHT and I mostly understand how it works. However, I don't quite understand what happens if you want to have separate DHTs with different entry types in each. Is this possible?
If I use a popular DHT library, does that mean I put and get entries using the same DHT as every user of said library? Or is DHT universal for everyone? How do you define an owner of a DHT, or how do you define a separate, contained DHT?
Yes, you can make separate DHTs. However you need to make the 'on the wire' protocols slightly diffrent so they can't speak to each other and get mixed up.
You can actually have unlimited numbers of DHTs using the same protocol as long as peers don't know each other.
This is the important part when you set up a network. You have to know a second peer to intitially create the network. The next peer would have to know one of the two initial nodes, the next one needs knowledge of one of the three above and so forth.
You are also able to be connected to multiple DHTs on the same host without the two interferring (at least in terms of data exchange, not in terms of local ressources). And you are also able to joins those two DHTs by telling one of them about the peers you are connected to in the other DHT. Though that might be not as easy as it sounds.
Recently, I've read a document of the Kademlia Protocol, I tried to understand the protocol, but I still have some question:
Why a node must find another node when he knows its ID but ip or port?
Why he has the ID while he doesn't know the ip or port, where did he get the ID?
I think the "distance" between two different nodes is not a routing distance or real distance, it's only a virtual distance that can be used the algorithm to find the node quickly, it's that right?
Maybe my English is not very clear because English is not my mother tongue, but I'll try to express myself clear if you need.
Thanks very much!
As cHao said, the distributed nature of the network means that nodes need to publish their IDs and their contact details to other nodes they talk to. There is no central place where IDs are mapped to contact info, so each node must keep this mapping for a subset of the nodes on the network in its own routing table.
Kademlia routing tables are structured so that nodes will have detailed knowledge of the network close to them, and exponentially decreasing knowledge further away.
The use of bitwise XOR as a measure of notional distance between IDs has the advantage that for a given target ID, no two IDs can have the same distance to the target.
Imagine a simple example where the IDs are in the range 00 to 63. If Kademlia used e.g. pure mathematical difference as a measure of distance, 15 and 35 would be the same distance to 25 - both would have a distance of 10. Using XOR, the distance between 15 and 25 is 22, and between 25 and 35 it's 58.
In this way, the group of k closest IDs to a target ID can be calculated unambiguously.
The constant k has a couple of uses in Kademlia, but it's primarily the replication factor. In other words, a piece of data is stored on the k closest nodes to the data's ID.
The lookup process is designed to return either a group of k nodes (before storing data on each of them) or return a single piece of data (from the first node holding it during the lookup iterations).
Because of this, pure Kademlia isn't best suited to finding just a single node, so I'm not sure that part of your question is too relevant. If you did want to use Kademlia to find a single node, it would probably be worth modifying the lookup process to finish early as soon as any node returns the target node's contact details (in the same way that the lookup finishes early if a target value is found during the process).
Since the network is distributed, by definition, there's no one master table of ID->address mappings. Nodes don't have to (and usually don't) know about all the other nodes. The process of "finding" a node is basically to ask known nodes "closest" to the target not so much about the target node directly, but about what nodes are closer to the target. The result of that query gives you the next group of nodes to query, and the process repeats -- and because a node would return results that are closer than it is, each iteration tends to find nodes closer and closer to the target til you finally reach a node that can say "Oh, node X? He's right over there."
At least that's what i'm understanding of it.