I am assuming that after finding peers using that Infohash you send handshake messages until one peer establishes a connection. But I can't seem to find any packet in Wireshark that has the metadata in it. If possible can you show me how a metadata query looks like?
The only thing I've found about metadata was bep 9 but it was kinda hard for me to understand and I didn't find anything in Wireshark similar
You already found the relevant BEP 9. To implement it you first have to implement BEP 10 and then check the extension header whether the peer supports BEP 9 and if it does send a series of extension messages with the appropriate negotiated IDs carrying a metadata request message.
You should be seeing the extension handshake message in most bittorrent connections since many clients support it. You're less likely to see metadata request/data messages (unless you explicitly trigger them in a client) because it's simply a less common operation than downloading the torrent payload.
In http://www.bittorrent.org/beps/bep_0005.html, the announce_peer query needs a token which has the require ,"the "token" received in response to a previous get_peers query".
Does it mean that if node A has never send a get_peer query to node B, then node A would never receive a peer_announce query from node B ?
Does it mean that if node A has never send a get_peer query to node B, then node A would never receive a peer_announce query from node B ?
No, it's the other way around:
Node A wants to announce that it is downloading a certain torrent.
Node A makes a get_peers query to Node B.
Node B sends a response to Node A for the get_peers query that it has just sent. Node B's response includes the token.
Node A, now, can send an announce_peer query to Node B using the token it has just received from Node B's response.
Why all this fuss?
The return value for a query for peers includes an opaque value known as the "token." For a node to announce that its controlling peer is downloading a torrent, it must present the token received from the same queried node in a recent query for peers. When a node attempts to "announce" a torrent, the queried node checks the token against the querying node's IP address. This is to prevent malicious hosts from signing up other hosts for torrents.
http://www.bittorrent.org/beps/bep_0005.html
The token represents a capability to announce that one is joining a swarm. It's like first you ask about the swarm -- "Hey, what's up with this swarm?" -- and the peer you ask tells you what it knows about that swarm, and it also gives you a ticket you can return, or not, if you want to join that swarm too. Fine-grained capability security for the win!
Yes that's right. It ensures that someone announcing a peer is actually interested in the corresponding torrent, and has demonstrated control over the announced address.
I've been trying to understand the torrent-magnet technology, but I can't seem to figure out how you get connected to the first peer when opening a magnet link.
When you get a magnet link like below, it contains no initial peer - only the BitTorrent Info Hash (btih) and the file name.
magnet:?xt=urn:btih:bbb6db69965af769f664b6636e7914f8735141b3&dn=ubuntu-12.04-desktop-i386.iso
According to BitTorrent & Magnets: How Do They Work? (MakeUseOf)
If you click a magnet link that does not specify a tracker (tr) the first peer will be found using DHT. Once you’ve got a peer, peer exchange kicks in too.
The DHT article on Wikipedia does not specify how to find a peer, but in the Kademlia article (upon which BitTorrent DHT is based), it says
A node that would like to join the net must first go through a bootstrap process. In this phase, the joining node needs to know the IP address and port of another node—a bootstrap node (obtained from the user, or from a stored list)—that is already participating in the Kademlia network.
But where does it know that node from? I don't see an address or anything present in the magnet link. Since it's decentralized (trackerless), I wouldn't expect it to know the node in advance. Or is the DHT in fact not decentralized?
For the most part, when you start a bittorrent client, bootstrap off of:
nodes from your last session, that were saved to disk
other peers that you have on any of the swarms you're on
There are a few well-known bootstrap nodes which clients can use if they have no other means of finding any. Essentially the only case this happens is when you install a client for the first time, and the first torrent you download is a magnet link without a tracker.
You can then hit router.utorrent.com:6881. I believe transmission, azureus and bitcomet run similar routers, and possibly other clients as well.
By "router", I mean a node that appear to behave like any other node in the DHT, but probably has a different mechanism for determining which nodes to hand out, and probably is optimized specifically for the use case of just introducing dht nodes to each other.
UPDATE: you can run your own DHT bootstrap machine, here's the source code.
Im trying to understand how the DHT protocol works, specially on the file-sharing-torrent world. I read many articles, but im still confused with the filename-value hash generation.
My thoughs how the dht works are the following:
Lets say im joining a p2p network and i want to share some files. For these files, hashmap keys are generated and "traveled" through the network until the nodes who are responsible for these generated keys are accessed. Then each of these nodes add in his list a record that says "the guy with the x IP address has the file that is related with the specified key.
When i search for a file, the hashmap key is generated for this file and travels the network until the node responsible for this key is found. Then this node is communicates with me and send me the IP addresses for the nodes that hosts the real data
Are my thoughs above correct??
Your thoughts are correct. This is the general idea behind DHT's.
For the first time I used a magnet link. Curious about how it works, I looked up the specs and didn't find any answers. The wiki says xt means "exact topic" and is followed by the format (btih in this case) with a SHA1 hash. I saw base32 mentioned, knowing it's 5 bits per character and 32 characters, I found it holds exactly 160bits, which is exactly the size of the SHA1.
There's no room for an IP address or anything, it's just a SHA1. So how does the BitTorrent client find the actual file? I turned on URL Snooper to see if it visits a page (using TCP) or does a lookup or the like, but nothing happened. I have no idea how the client finds peers. How does this work?
Also, what is the hash of? Is it a hash of an array of all the file hashes together? Maybe it's a hash of the actual torrent file required (stripping certain information)?
In a VM, I tried a magnet link with uTorrent (which was freshly installed) and it managed to find peers. Where did the first peer come from? It was fresh and there were no other torrents.
A BitTorrent magnet link identifies a torrent using1 a SHA-1 or truncated SHA-256 hash value known as the "infohash". This is the same value that peers (clients) use to identify torrents when communicating with trackers or other peers. A traditional .torrent file contains a data structure with two top-level keys: announce, identifying the tracker(s) to use for the download, and info, containing the filenames and hashes for the torrent. The "infohash" is the hash of the encoded info data.
Some magnet links include trackers or web seeds, but they often don't. Your client may know nothing about the torrent except for its infohash. The first thing it needs to is find other peers who are downloading the torrent. It does this using a separate peer-to-peer network2 operating a "distributed hash table" (DHT). A DHT is a big distributed index which maps torrents (identified by infohashes) to lists of peers (identified by IP address and ports) who are participating in a swarm for that torrent (uploading/downloading data or metadata).
The first time a client joins the DHT network it generates a random 160-bit ID from the same space as infohashes. It then bootstraps its connection to the DHT network using either hard-coded addresses of clients controlled by the client developer, or DHT-supporting clients previously encountered in a torrent swarm. When it wants to participate in a swarm for a given torrent, it searches the DHT network for several other clients whose IDs are as close3 as possible to the infohash. It notifies these clients that it would like to participate in the swarm, and asks them for the connection information of any peers they already know of who are participating in the swarm.
When peers are uploading/downloading a particular torrent, they try to tell each other about all of the other peers they know of that are participating in the same torrent swarm. This lets peers know of each other quickly, without subjecting a tracker or DHT to constant requests. Once you've learned of a few peers from the DHT, your client will be able to ask those peers for the connection information of yet more peers in the torrent swarm, until you have all of the peers you need.
Finally, we can ask these peers for the torrent's info metadata, containing the filenames and hash list. Once we've downloaded this information and verified that it's correct using the known infohash, we're in practically the same position as a client that started with a regular .torrent file and got a list of peers from the included tracker.
The download may begin.
1 The infohash is typically hex-encoded, but some old clients used base 32 instead. v1 (urn:btih:) uses the SHA-1 digest directly, while v2 (urn:bimh:) adds a multihash prefix to identify the hash algorithm and digest length.
2 There are two primary DHT networks: the simpler "mainline" DHT, and a more complicated protocol used by Azureus.
3 The distance is measured by XOR.
Further Reading
BEP-3: The BitTorrent Protocol Specification
BEP-52: The BitTorrent Protocol Specification v2
BEP-5: DHT Protocol
BEP-9: Extension for Peers to Send Metadata Files
BEP-10: Extension Protocol
BEP-11: Peer Exchange (PEX)
Azureus DHT Description
Peer discovery and resource discovery (files in your case) are two different things.
I am more familiar with JXTA but all peer to peer networks work on the same basic principles.
The first thing that needs to happen is peer discovery.
Peer Discovery
Most p2p networks are "seeded" networks: when first starting a peer will connect to a well-known (hard-coded) address to retrieve a list of running peers. It can be direct seeding like connecting to dht.transmissionbt.com as mentioned in another post or indirect seeding as usually done with JXTA where the peer connects to an address that only delivers a plain text list of other peers network addresses.
Once connection is established with the first (few) peer(s), the connecting peer performs a discovery of other peers (by sending requests out) and maintains a table of them. Since the number of other peers can be huge, the connecting peer only maintains part of a Distributed Hash Table (DHT) of the peers. The algorithm to determine which part of the table the connecting peer should maintain varies depending on Network. BitTorrent uses Kademlia with 160 bit identifiers/keys.
Resource Discovery
Once a few peers have been discovered by the connecting peer, the latter sends a few requests out for discovery of resources to them. Magnet links identifies those resources and are built in such a way that they are a "signature" for a resource and guarantee that they uniquely identify the requested content among all the peers.
The connecting peer will then send a discovery request for the magnet link/resource to peers around it. The DHT is built in such a way that it helps determine which peers should be asked first for the resource (read on Kademlia in Wikipedia for more).
If the requested peer does not hold the requested resource it will usually "pass on" the query to additional peers fetched from its own DHT.
The number of "hops" the query can be passed on is usually limited; 4 is an usual number with JXTA type networks.
When a peer holds the resource, it replies with its full details. The connecting peer can then connect to the peer holding the resource (directly or via a relay - I won't go into details here) and start fetching it.
Resources/Services in P2P networks are not directly attached to network addresses: they are distributed and that is the beauty of these highly scalable networks.
I was curious by the same question myself. Reading the code for transmission, I found the following in libtrnasmission/tr-dht.c:
3248: bootstrap_from_name( "dht.transmissionbt.com", 6881,
bootstrap_af(session) );
It tries that 6 times, waiting 40(!) seconds between tries. I guess you can test it by deleting the config files (~/.config/transmission on unix), and blocking all communication to dht.transmissionbt.com, and see what happens (wait 240 seconds at least).
So it appears the client has a bootstrap node built in to start with. Of course, once it has gotten into the network, it doesn't need that bootstrap node anymore.
I finally found specification. For the first time google didnt help. (wiki linked to bittorrent.com which is the main site. I Clicked the developers link, notice the bittorrent.org tab on the right then it was easy from there. Its hard finding links when you have no idea what they are labeled and many clicks away).
It seems like all torrents have a network of peers. You find peers from trackers and you keep them between sessions. The network allows you to find peers and other things. I havent read how its used with magnet links but it seems like it is undefined how a fresh client find peers. Perhaps some is baked in, or they use their home server or known trackers embeded into the client to get the first peer in the network.
When I started answering your question, I didn't realize you were asking how the magnet scheme works. Just thought you wanted to know how the parts relevant to the bittorrent protocol were generated.
The hash listed in the magnet uri is the torrent's info hash encoded in base32. The info hash is the sha1 hash of the bencoded info block of the torrent.
This python code demonstrates how it can be calculated.
I wrote a (very naive) C# implementation to test this out since I didn't have a bencoder on hand and it matches what is expected from the client.
static string CalculateInfoHash(string path)
{
// assumes info block is last entry in dictionary
var infokey = "e4:info";
var offset = File.ReadAllText(path).IndexOf(infokey) + infokey.Length;
byte[] fileHash = File.ReadAllBytes(path).Skip(offset).ToArray();
byte[] bytes;
using (SHA1 sha1 = SHA1.Create())
bytes = sha1.ComputeHash(fileHash, 0, fileHash.Length - 1); // need to remove last 'e' to compensate for bencoding
return String.Join("", bytes.Select(b => b.ToString("X2")));
}
As I understand it, this hash does not include any information on how to locate the tracker, the client needs to find this out through other means (the announce url provided). This is just what distinguishes one torrent from another on the tracker.
Everything related to the bittorrent protocol still revolves around the tracker. It is still the primary means of communication among the swarm. The magnet uri scheme was not designed specifically for use by bittorrent. It's used by any P2P protocols as an alternative form of communicating. Bittorrent clients adapted to accept magnet links as another way to identify torrents that way you don't need to download .torrent files anymore. The magnet uri still needs to specify the tracker in order to locate it so the client may participate. It can contain information about other protocols but is irrelevant to the bittorrent protocol. The bittorrent protocol ultimately will not work without the trackers.
the list of peers are probably populated from the torrent that upgrades the client (e.g. there's a torrent for utorrent that upgrades it). as long as everyone's using the same client, it should be good because you have no choice but to share the upgrade.