An etcd instance can have multiple peers. According to the CoreOS doc I can configure a parameter ETCD_PEER_KEY_FILE=/path/to/peers.key. If I interpret this parameter according to its name, I can put the private keys of a peer (or more peer_s_) in that file. What's the rational behind this? Can someone explain, what this parameter really means (didn't find any documentation).
Got it. Documentation is here
--cert-file=<path>: Certificate used for SSL/TLS connections to etcd.
--peer-cert-file=<path>: Certificate used for SSL/TLS connections between peers. This will be used both for listening on the peer address as well as sending requests to other peers.
Related
I use mitmproxy to gather intel from outbound AS2 (HTTP) requests leaving our network. The schema goes like this:
Mendelson AS2 ➡ mitmproxy ➡ partner AS2 server
↘
redis
Where possible, I need to verify all SSL certs to make sure the business connection is safe. Some partners use less known CAs which I then add to a truststore used by the ssl_verify_upstream_trusted_ca option.
Some partners though don't really care about security and I need to trust their certificates no matter what. How do I do that in mitmproxy? Disabling the verification entirely isn't an option.
Thank you.
It's been a while since I've tried to solve this using a custom addon and it seems to work fine so I'll share it here:
https://gist.github.com/jsmucr/24cf0859dd7c9bba8eb2817d7b0bf4b6
This approach has a bit of disadvantage and that's the fact that it doesn't check if the peer certificate changes.
According to the specification of Bittorrent tracker (https://wiki.theory.org/BitTorrentSpecification) peer is identified by their peer_id and ip. peer_id is passed to tracker in announce request, ip - from request headers, but optionally can be passed as a query parameter. How tracker knows that this request was actually sent from this peer (not a fraud) without any key?
Note that the official specification lists that parameter as optional, meaning that it is not part of the identity of the client. Rather it should be considered as a hint to the tracker that the tracker should hand out that IP to other clients instead of the source of the announce connection. This hint should be ignored if the tracker doesn't trust the client.
Trust can be established if a client authenticates itself to the tracker (e.g. via HTTP basic auth or a secret token in the announce URL).
Instead of trusting the client the tracker could also verify it, e.g. by opening a bittorrent connect to the IP and port that was announced and then send a bittorrent handshake for the infohash that was announced to check if the remote can accept bittorrent connections. That also has the advantage of rejecting announces that aren't reachable due to firewalls or NAT limitations.
For open, public trackers neither approach is practical, so they usually ignore the &ip=... parameter. Trackers deployed in more restricted environments do make use of the trust and/or verify approaches.
As of now I have used confix tx generator to generate config material of muliple peers and have changed the createPeerAdmin card to include the IP address of all Peer nodes so that connection.json file produced contains the IP address of all the peer nodes in my network , Is there a way in which i can add 1 more peer node and it could be connected with my already running network.
Yes, there is.
You should read about service discovery feature.
There are client side APIs in the SDKs you can use to utilize this feature and have the peers be selected dynamically.
In BitTorrent, client connects to the tracker specified in .torrent file. Tracker is a kind of centralized server and it is the starting point. So BitTorrent is not pure p2p.
If we want to develop pure p2p system, we should design routing overlay network. All nodes will have routing table like routers do. But even in routing overlay network, each node should know at least one existing node(GUID, IP address) initially. So how can we determine this? Should we keep 'one existing node to connect initially' forever like fixed centralized server? If so, I think this is not fully decentralized method.
The solution you describe (defining a central peer wit well-know ip address) is not the only one.
The other solution is to post an html page (or json file) in a well-know URL on the net, and make sure this item contains an updated list of peers this peer can initialy connect. This list can be updated if a peer goes down. Eventually, you can use multiple URLs.
Pure P2P system is a theoretical concept which cannot be implemented fully in reality.
You could use an anycast. So the first other client will answer and may send such an initial "client list". Where your client can connect to them to get more lists.
Classically I would implement a multicast to a adress and wait for an answer of other clients.
Firstly, a true peer to peer network is not necessarily decentralized. Secondly, decentralization does not necessarily mean that the network doesn't make use of secondary services which may themselves be centralized.
The main question in both of these issues is wheather the primary resources of the network solution are distributed through correlated peers.
For example, peer-to-peer video conferencing may use a central contacts service but still be pure peer-to-peer as long as the peers resolve such issues before entering a true peer-to-peer scope. This would also be decentralized.
What it comes down to is what your trying to solve by using peer-to-peer. A video conference is a video conference - it starts with a video being recorded on one peer and ends with the video being viewed on another. As long as each byte of this data is transferred directly between the peers (even if there's hundreds of peers in the conference, and regardless of how these peers found each other) it is a true peer-to-peer video conference.
Note that the video peers will still be in your typical ring, and that the contacts lists may still use the node key for location information rather than an IP. This will still be a network overlay as it will still be built over IP, replacing it's addressing scheme on the peer level to facilitate true peer-to-peer networking.
What it realy comes down to is the concepts of a network connection. IP just pushes packets to unspecified routers until it gets to a specific address. 'connections' between each endpoint only exist within the higher software levels (including when dealing with TCP/IP). A connection is just the data used within software to understand who is who and how each point can handle data etc. Peer-to-peer network overlays effectively distributes this data, eliminating the need for each peer to create massive amounts of connections to communicate on a massive scope. Decentralization is not required for this (as long as peer to peer communication is not centralized), and a secondary service within the system wont necessarily limit a network's scope or otherwise centralize actual peer-to-peer networking.
So to answer your question, it doesn't matter where it initially connects in order to be considered peer-to-peer, and different peer-to-peer services will handle this based on their service design.
Ok so I am thinking about writing a p2p protocol for a number of different services for an AI project, and I thought I'd come here to see if i could get some ideas on how to get the initial connection.
I have come across several ways to establish the initial connection:
1) You have a static ip address on the internet that distributes information on other peers. This isn't good, because:
a) it's a single point of failure, the service could go offline, preventing any new connections from creating an initial connection to peers,
b) the IP address could change. This could be mitigated by using a domain name which is maintained to point to the current ip address of the location of a service which provides data on peers, however this can be subverted in theory by hackers by spoofing or arp poisoning, dns attacks etc.
2) You could force the user to provide an initial ip address or hostname for another peer, and it's up to the user to find the hostname / ip address / port number. This is good, but if someone posts disinformation or they cannot find a peer on google or some other search site then obviously it's fallable.
3) You could leave it to the peer to publish their own existence in a central location - for example a group of IRC channels or a group of websites. Again, unless it's going through a central trusted domain, it's hard to determine the authenticity of the peer.
4) You could use some kind of nmap style discovery algorithm that searches through subnets for appropriate protocols. The problem with this approach is it's slow and it's likely to attract attention from things like firewalls etc.
5) This is a variation of 3) you could allow the peers to advertise their own information on a website, then instead of having to look for the information in a suitable location (a specific website, or group of websites), you can let google's search algorithm find it, and do the discovery for you, however you could imagine that this may take a few days for google to cache the website data with information on the peers. Also again, it would be upto you to provide some way to verify the authenticity of the advertised data.
6) If you are interested in an exclusive p2p network that locks out certain people (for example, you might want to have a file sharing network and you don't want law enforcement to be able to access it, or MPIAA), then you could use 2) and then have a referral system where you require that the initial connection provide the referrer's ip address, and then the service could connect to the referrers ip address and ask the referrer if they did indeed refer the referee.
That's all I can think of currently, but if anyone comes up with any other ways to do this, i would be very interested.
If I have a torrent w/o any trackers in it, and I just started a bittorent client so I have no peers yet...how do I know who to first connect with in the DHT? It seems like I would have to know at least ONE node in the DHT to get started....
The mainline DHT bootstrap nodes are router.utorrent.com and a CNAME to it, router.bittorrent.com. Port 6881.
When a BitTorrent client connects to DHT, there is an initial place that it goes to find peers. With the original BitTorrent client, there was a url to bitorrent.com that would help get things started. I tried looking up the reference but I couldn't find it. Once you've established connections with other clients, then you can do an announce on the DHT network to find peers for the torrent you're looking for.
Here's a link to the BitTorrent specs that discuss DHT.
A trackerless torrent dictionary does
not have an "announce" key. Instead, a
trackerless torrent has a "nodes" key.
This key should be set to the K
closest nodes in the torrent
generating client's routing table.
Alternatively, the key could be set to
a known good node such as one operated
by the person generating the torrent.
Please do not automatically add
"router.bittorrent.com" to torrent
files or automatically add this node
to clients routing tables.
the graph at the bottom of this DHT monitoring project site shows
dht.transmissionbt.com
router.utorrent.com
router.bittorrent.com
as bootstrapping peers
In BiTTorrent, you have three main options:
Torrent File: some torrent files can embed nodes for you to link into the DHT with (in fact, it's recommended when making a torrent file)
Hardcoding: Some torrent clients hard code a few bootstrap nodes (like the ones mentioned by stk). These are usually run by companies and organizations with long-running servers.
PEX / Peer Conversations: You can usually ask for DHT nodes from the people you are downloading other torrents from (if your clients understand eachothers language. ie some versions are incompatible).
Transmission uses a hardcoded bootstrap node for dht if there is no other way to get peers:
bootstrap_from_name( "dht.transmissionbt.com", 6881, bootstrap_af(session) );
I guess each torrent client uses their own bootstrap node.
For the record, Deluge also uses hardcoded boostrap nodes:
dht_bootstraps = set(
lt_bootstraps.split(',')
+ [
'router.bittorrent.com:6881',
'router.utorrent.com:6881',
'router.bitcomet.com:6881',
'dht.transmissionbt.com:6881',
'dht.aelitis.com:6881',
]
)
A client can learn about other DHT-capable peers through it's interactions with them. A peer's support for DHT is advertised in it's Handshake. Once a client discovers at least one good, well-connected DHT peer, it can navigate the DHT to find more and closer DHT peers. It will remember these peers, called nodes in DHT-speak, between restarts of the software and maintain/update the list continuously while it is running. In the worse case where a client knows of no good DHT-capable peers, it will require you to download a tracker-based torrent so it can hopefully contact a few good DHT-capable peers it learns about through the tracker.
Update:
For it's initial list of DHT peers, as #Seppo points out, a torrent client can use one or more hard-coded DNS names to find the addresses for well-known peers, and it may also include a hard-coded list of peers as a final fallback as well. One limitation of DNS, however, it no port information is provided so a default port of 6881 is generally assumed whereas other means support peers operating on different ports.
Here are the primary nodes I've come across.
dht.transmissionbt.com 6881
router.bittorrent.com 6881
router.bitcomet.com 6881
dht.aelitis.com 6881
bootstrap.jami.net 4222