Initially this was a different question. But the questions I ended up answering which gave me a good idea of how everything works were:
How are buckets organized?- how does the range system work.
How does the xor matrix works for distance?
And how is the routing table organized?
This is an idea of how I think this works:
Routing table is composed of a binary tree composed of 1 and 0. A bucket range is composed of nodes with similar prefix, on a branch number composed of the amount of similar bits on the prefix that are similar. So for example branch 2 composed of 1-0 can have a bucket of (1011,1000,1011) and so on. using your own Id you can xor another nodes Id and the amount of 0's in the prefix will be the branch number and the prefix making the bucket range would be the same as your own node. When needing to insert a new node into a full bucket, if your id fits within the range you'll split the bucket. This way you'll have a routing table composed of nodes similar to you and a bit that don't. Also need to mention the deeper you go on the tree the closer you'll get to your id
Related
In the section "Routing" in the original paper, the problem of the normal bucket splitting rule is described as follows:
Every node with prefix 001 would have an empty k-bucket into which u should be inserted, yet u's bucket refresh would only notify [$ k]of the node.
I don't understand why it is a problem. Why does every node with prefix 001 needs to receive u's bucket refresh?
I have read the original paper and have spent quite amount of time researching the bucket splitting, but I couldn't figure out. Can any one explain?
Thank you
That's the highly unbalanced tree case. For kademlia to work properly it must have nearly-exact view of its neighborhood while only needing a subsampled views of more remote portions of the keyspace.
The initially described bucket splitting algorithm would only split a bucket when its ID-range covers the node's own ID. But if a node's nearest neighbors don't fall into this bucket then maintaining an exhaustive view of the neighborhood requires a non-covering bucket to be split (or in S/Kademlia: an explicit neighborhood list to be maintained).
Okay, I've been reading articles and the paper about Kademlia recently to implement a simple p2p program that uses kademlia dht algorithm. And those papers are saying, those 160-bit key in a Kademlia Node is used to identify both nodes (Node ID) and the data (which are stored in a form of tuple).
I'm quite confused on that 'both' part.
As far as my understanding goes, each node in a Kademlia binary tree uniquely represents a client(IP, port) who each holds a list of files.
Here is the general flow on my understanding.
Client (.exe) gets booted
Creates a node component
Newly created node joins the network (bootstrapping)
Sends find_node(filehash) to k-closest nodes
Let's say hash is generated by hashing file binary named file1.txt
Received nodes each finds the queried filehash in its different hash table
Say, a hash map that has a list of files(File Hash, file location)
Step 4,5 repeated until the node is found (meanwhile all associated nodes are updating the buckets)
Does this flow look all right?
Additionally, bootstrapping method of Kademlia too confuses me.
When the node gets created (user executes the program), It seems like it uses bootstrapping node to fill up the buckets. But then what's bootstrapping node? Is it another process that's always running? What if the bootstrapping node gets turned off?
Can someone help me better understand the concept?
Thanks for the help in advance.
Does this flow look all right?
It seems roughly correct, but your wording is not very precise.
Each node has a routing table by which it organizes the neighbors it knows about and another table in which it organizes the data it is asked to store by others. Nodes have a quasi-random ID that determines their position in the routing keyspace. The hashes of keys for stored data don't precisely match any particular node ID, so the data is stored on the nodes whose ID is closest to the hash, as determined by the distance metric. That's how node IDs and key hashes are used for both.
When you perform a lookup for data (i.e. find_value) you ask the remote nodes for the k-closest neighbor set they have in their routing table, which will allow you to home in on the k-closest set for a particular target key. The same query also asks the remote node to return any data they have matching that target ID.
When you perform a find_node on the other hand you're only asking them for the closest neighbors but not for data. This is primarily used for routing table maintenance where you're not looking for any data.
Those are the abstract operations, if needed an actual implementation could separate the lookup from the data retrieval, i.e. first perform a find_node and then use the result set to perform one or more separate get operations that don't involve additional neighbor lookups (similar to the store operation).
Since kademlia is UDP-based you can't really serve arbitrary files because those could easily exceed reasonable UDP packet sizes. So in practice kademlia usually just serves as a hash table for small binary values (e.g. contact information, public keys and such). Bulk operations are either performed by other protocols bootstrapped off those values or by additional operations beyond those mentioned in the kademlia paper.
What the paper describes is only the basic functionality for a routing algorithm and most basic key value storage. It is a spherical cow in a vacuum. Actual implementations usually need additional features or work around security and reliability problems faced on the public internet.
But then what's bootstrapping node? Is it another process that's always running? What if the bootstrapping node gets turned off?
That's covered in this question (by example of the bittorrent DHT)
I redesign our system (P2P application) that was built using "flat model" of k-buckets - each distance has its own k-backet. The distance is length of identifier minus length of shared prefix (XOR). Everything is clear here.
Now we want to use binary tree to hold buckets (as in last Kadelmia docs).
The tree approach doesn't deal with distance "directly" when we "look for bucket" to put new contact in. This confuses me, since paper says that k-bucket should be split if new node is closer to local then K-closest node.
My question: how to calculate distance in this case? It cannot be prefix (path) of bucket, since bucket may contain nodes with different prefixes.
What is a convenient way to find K-closest node?
Thanks in advance.
It cannot be prefix (path) of bucket, since bucket may contain nodes with diffrent prefixes.
In the tree layout each bucket does have a prefix, but it is not implicit in its position in the routing table data structure, it must be tracked explicitly during split and merge operations instead, e.g. as base address plus prefix length, similar to CIDR notation.
An empty routing table starts out with a prefix covering the entire keyspace (i.e. 0x00/0), after some splitting one of the buckets might cover the range 0x0CFA0 - 0x0CFBF, which would be be the bucket prefix 0x0CFA/15.
See this answer on another question which contains an example routing table layout
Additionally see this answer for a simple and more advanced bucket splitting algorithm
How to find the matching bucket for a given ID depends on the data structure used. A sorted list will require binary search, a Patricia-Trie with nearest-lookups is another option. Even the brute force approach can be adequate as long as you do not have to handle too many operations per second.
I'm trying to understand how Kademlia works in regards to finding a resource. There is pretty good description of now to build a node tree which is closest to the self node, how to find the distance between nodes, how to initiate the process etc. What I don't understand is how the file infohash fits into this picture. All descriptions tell us how to get into the play and build your own part of the distributed hash table but it is not it. We are doing this to actually find a resource, a file with a certain infohash. How it is stored in this node tree or there is a separate one? How is it works to find nodes which have this infohash, consequently having the file.
There is brief mentioning of the fact that the node id and infohash having the same 20 bytes length codes and something that node id XOR infohash is the distance between the node and the resource but I cannot imagine how is that and how it helps to find the resource? After all a node id actually having the resource can have the greatest XOR distance to the resource.
Thank you,
Alex
I recommend that you don't just read the bittorrent DHT specification but also the original kademlia paper, since the former is fairly concise and only mentions some things in passing.
Bittorrent's get_peers lookup is equivalent to the find_value operation described in the paper.
In short: just like you can do an iterative lookup to find the K-closest-node-set - closest based on xor-distance relative to the target key - for your own node's ID you can do so for any other ID.
For get_peers you simply use the infohash as target key.
The K-closest-node-set for a particular infohash is the set of nodes considered responsible to store the data for said infohash. Although due to inaccuracies of implementations and node churn more than K nodes around the target key may be storing data of interest.
Currently in my Node.Js app I use node-uuid module for giving unique IDs to my database objects.
Using uuid.v1() function from that module I get something like
81a0b3d0-e4d0-11e3-ac56-73f5c88681be
Now, my requests are quite long, sometimes hundreds of nodes and edges in one query. So you can imagine they become huge, because every node and edge has to have a unique ID.
Do you know if I could use a shorter ID system in order to not run into any problems after the number of my items grow? I mean I know I could get away with just the first 8 symbols (as there are 36^8 > 2 Trl combinations, but how well will it perform when they are randomly generated? As the number of my nodes increase, what is the chance that the randomly generated ID will not fall into the already existing ones?
Thank you!
If you're really concerned about uuid collisions you can always simply do a lookup to make sure you don't already have a row with that uuid. The point is that there is always a very low but non-zero chance of a collision given current uuid generators, especially with shorter strings.
Here's a post that discusses it in more detail
https://softwareengineering.stackexchange.com/questions/130261/uuid-collisions
One alternative would be to use a sequential ID system (autoincrement) instead of uuids.