How does NodeId generation in a storj kademlia node take place?
Why does it take long time to generate a nodeid and how does this contribute to the security of the node?
According to this blogpost they require hashcash when generating a nodeID, as proposed in the S/Kademlia paper, to make make sybil attacks more costly.
Related
I m trying to implement a DHT using Kademlia paper as a way of understanding better how these systems work.
I have read some other articles that referes to this way of implementing a distributed hash table but there is something I can't wrap my head around.
In a p2p exchange files network, key ids could be implemented as digest of filename for consistency throught search mecanism.
But what about node id itself ?
Should I, for example, take the digest of "WAN-IP:PORT" combination or simply generate a completly random id from scratch ?
In the second case scenario, there is alway a risk that two nodes generate the same id. But using my WAN-IP:PORT I rely on the fact that my client is running a node behind a WAN-IP that should never change.
How should it come from the network itself ? I mean, on first contact, the peer work some magic and give the new node an id ?
I would appreciate some inputs on how to implement Kademlia node id generation.
As stated in Kademlia paper, "Node IDs are currently just random 160-bit identifiers, though they could equally be constructed as in Chords".
Chords uses the SHA-1 digest of a node IP.
Okay, I've been reading articles and the paper about Kademlia recently to implement a simple p2p program that uses kademlia dht algorithm. And those papers are saying, those 160-bit key in a Kademlia Node is used to identify both nodes (Node ID) and the data (which are stored in a form of tuple).
I'm quite confused on that 'both' part.
As far as my understanding goes, each node in a Kademlia binary tree uniquely represents a client(IP, port) who each holds a list of files.
Here is the general flow on my understanding.
Client (.exe) gets booted
Creates a node component
Newly created node joins the network (bootstrapping)
Sends find_node(filehash) to k-closest nodes
Let's say hash is generated by hashing file binary named file1.txt
Received nodes each finds the queried filehash in its different hash table
Say, a hash map that has a list of files(File Hash, file location)
Step 4,5 repeated until the node is found (meanwhile all associated nodes are updating the buckets)
Does this flow look all right?
Additionally, bootstrapping method of Kademlia too confuses me.
When the node gets created (user executes the program), It seems like it uses bootstrapping node to fill up the buckets. But then what's bootstrapping node? Is it another process that's always running? What if the bootstrapping node gets turned off?
Can someone help me better understand the concept?
Thanks for the help in advance.
Does this flow look all right?
It seems roughly correct, but your wording is not very precise.
Each node has a routing table by which it organizes the neighbors it knows about and another table in which it organizes the data it is asked to store by others. Nodes have a quasi-random ID that determines their position in the routing keyspace. The hashes of keys for stored data don't precisely match any particular node ID, so the data is stored on the nodes whose ID is closest to the hash, as determined by the distance metric. That's how node IDs and key hashes are used for both.
When you perform a lookup for data (i.e. find_value) you ask the remote nodes for the k-closest neighbor set they have in their routing table, which will allow you to home in on the k-closest set for a particular target key. The same query also asks the remote node to return any data they have matching that target ID.
When you perform a find_node on the other hand you're only asking them for the closest neighbors but not for data. This is primarily used for routing table maintenance where you're not looking for any data.
Those are the abstract operations, if needed an actual implementation could separate the lookup from the data retrieval, i.e. first perform a find_node and then use the result set to perform one or more separate get operations that don't involve additional neighbor lookups (similar to the store operation).
Since kademlia is UDP-based you can't really serve arbitrary files because those could easily exceed reasonable UDP packet sizes. So in practice kademlia usually just serves as a hash table for small binary values (e.g. contact information, public keys and such). Bulk operations are either performed by other protocols bootstrapped off those values or by additional operations beyond those mentioned in the kademlia paper.
What the paper describes is only the basic functionality for a routing algorithm and most basic key value storage. It is a spherical cow in a vacuum. Actual implementations usually need additional features or work around security and reliability problems faced on the public internet.
But then what's bootstrapping node? Is it another process that's always running? What if the bootstrapping node gets turned off?
That's covered in this question (by example of the bittorrent DHT)
I am studying back-end programming, specifically with Node.js and ExpressJS and currently it baffles me how does the "keys" prop of cookie-session library help us? What is the point of it? Have been reading a lot of different materials related to authentication, sessions and etc, but the answer to this particular question remains to be ambiguous to me.
Could someone give me an in-depth explanation, preferably both ways: in simple terms and using programming lexicon, regarding this topic?
to explain it in simple terms:
it's essentially using different keys (rotating the keys) every certain time period to encrypt the data; so that the data breach from one key can be contained/limited; or let's say a key can be cracked in x months, then rotating the key - using a different key every x-1 months to reduce the probability of data being compromised.
This question actually belongs to crypto stack exchange and is kinda hard to describe and out of the scope to include in the docs. The search also doesn't return any accurate and results unless you search specifically for methods/algorithms of key rotation.
visit these to get some conceptual overview and in-depth examples:
What's the purpose of key-rotation?
(recommended)
Key Rotation for Authenticated Encryption
And these for more in-depth technical and mathematical reference:
Fully Key-Homomorphic Encryption, Arithmetic Circuit ABE and Compact Garbled Circuits?
Fast and Secure Updatable Encryption
Whatever values (inside the array) are provided for keys prop are used to encrypt and decrypt the user.id / sessions.id that we store in the cookie of our browser.
I'm trying to understand how Kademlia works in regards to finding a resource. There is pretty good description of now to build a node tree which is closest to the self node, how to find the distance between nodes, how to initiate the process etc. What I don't understand is how the file infohash fits into this picture. All descriptions tell us how to get into the play and build your own part of the distributed hash table but it is not it. We are doing this to actually find a resource, a file with a certain infohash. How it is stored in this node tree or there is a separate one? How is it works to find nodes which have this infohash, consequently having the file.
There is brief mentioning of the fact that the node id and infohash having the same 20 bytes length codes and something that node id XOR infohash is the distance between the node and the resource but I cannot imagine how is that and how it helps to find the resource? After all a node id actually having the resource can have the greatest XOR distance to the resource.
Thank you,
Alex
I recommend that you don't just read the bittorrent DHT specification but also the original kademlia paper, since the former is fairly concise and only mentions some things in passing.
Bittorrent's get_peers lookup is equivalent to the find_value operation described in the paper.
In short: just like you can do an iterative lookup to find the K-closest-node-set - closest based on xor-distance relative to the target key - for your own node's ID you can do so for any other ID.
For get_peers you simply use the infohash as target key.
The K-closest-node-set for a particular infohash is the set of nodes considered responsible to store the data for said infohash. Although due to inaccuracies of implementations and node churn more than K nodes around the target key may be storing data of interest.
Should I be generating the id of the documents in a CouchDB or should I depend on CouchDB to generate it? What are the advantages or disadvantages in these approaches? Is there any performance implications on any of these options?
There is no difference as far as CouchDB is concerned. Frederick is right that sequential ids are slightly faster. If you query /_uuids?count=10 you will notice that the UUIDs are sequential (by default).
However, even with random IDs, once you run compaction, they will all be in the "right" order internally in the .couch file and at that point there is no difference. So in the long run, I don't usually worry about it.
The main thing is that you should use mostly sequential ids. As this article and this bit of the couchdb book explain, using random ids results in a much less efficient structure internally, both speed wise and in terms of space used on disc.
Self generated ids are almost impossible to deal with if you have two or more separated instances of your app. Because the synchronisation between the different instances is not instantaneous. A solution for this can be to have one server dedicated to generate (or check the availability of) the ids, for example using a SQL database, and acting as a gate for document creation.
On the other hand, if you have only one server and will never need more, there is one advantage I find interesting to self generated uids: since they have to be unique, you can use them in urls. For instance take the slug of the title of a blog post as the _id.
Performance-wise, the CouchDB's generated ids are pretty long so if your own ids are shorter, you will save significant disk space (assuming you have a looot of documents).
Both answers above tell about PROS of sequential IDs.
Here is a major problem arose by sequential IDs.
Predictability of other IDs in documents using a single ID.
Due to this we can't use sequential IDs in application URLs as identifiers due to other IDs being predictable using one ID, and using as url authentication is also not possible.( As done by file sharing services).