i have a basic idea about DHT and P2P. I need to explain how DHT is used in a P2P network with an example. Thank you
Well there are structured, and unstructured p2p overlays, basically this means, that structured networks have rules that dictate where objects stored are going to be located, and how the network topology will be organized, unstructured overlays don't.
DHTs are the most popular examples of structured, overlays what it basically does is dictate how queries should be routed (ie finding who has object x), and how they should be placed, using varying methods, ie chord has a ring topology and uses consistent hashing + finger table, kademlia uses xor and a tree topology.
Related
I have a use case for Twilio compositions (two video/audio streams) which relies heavily on the timing/syncing accuracy of the participant streams.
We produce a composition of both videos, and also compositions of each individual audio streams. The word timings within each stream are extracted and compared which relies heavily on the syncing accuracy between the two streams.
In particular we require accurate syncing (to the best of the available data) even when there were network issues, disconnects/reconnects and so on.
Can any comment on how accurate and robust the timing/syncing of Twilio's compositions are for this use case?
Have you had similar use cases and can you comment on your experience with accuracy down to say ~100ms?
Sorry - I know this is a general and vague question - just not sure where else to ask.
Posting an answer here for everyone's future reference.
Having done hundreds of compositions in a wide variety of test scenarios (joining at different times, reconnections, dodgy internet, closing browser, etc) I can happily report that Twilio compositions system generates extremely reilable and accurate compositions in terms of timing and syncing.
However it seems necessary to include both participants in the composition to ensure the start times are consistent.
My solution to creating an individual participant's composition (fully synced with the other participants was:
Create a composition with the video from both participants
Make other participant's video z-ordered behind the main participant, and placed 16x16 in the corner
Only include the audio streams from the participant of interest
Doing this in turn for each participant produces individual participant compositions which are completely aligned with each other.
I have a use case for Twilio compositions (two video/audio streams) which relies heavily on the timing/syncing accuracy of the participant streams.
We produce a composition of both videos, and also compositions of each individual audio streams. The word timings within each stream are extracted and compared which relies heavily on the syncing accuracy between the two streams.
In particular we require accurate syncing (to the best of the available data) even when there were network issues, disconnects/reconnects and so on.
Can any comment on how accurate and robust the timing/syncing of Twilio's compositions are for this use case?
Have you had similar use cases and can you comment on your experience with accuracy down to say ~100ms?
Sorry - I know this is a general and vague question - just not sure where else to ask.
Posting an answer here for everyone's future reference.
Having done hundreds of compositions in a wide variety of test scenarios (joining at different times, reconnections, dodgy internet, closing browser, etc) I can happily report that Twilio compositions system generates extremely reilable and accurate compositions in terms of timing and syncing.
However it seems necessary to include both participants in the composition to ensure the start times are consistent.
My solution to creating an individual participant's composition (fully synced with the other participants was:
Create a composition with the video from both participants
Make other participant's video z-ordered behind the main participant, and placed 16x16 in the corner
Only include the audio streams from the participant of interest
Doing this in turn for each participant produces individual participant compositions which are completely aligned with each other.
I need a tool (preferably online, but at that stage I don't care anymore) which allows me to modelize relationships (similar to BPMN) which can be represented as compound graphs (though most of the times, it will be a simpler nested graph). I can't seem to achieve my goal using BPMN formalism (swimlanes are the closest but they do not allow arbitrary nesting). I am thinking to reuse UML modelization tools but I don't know enough about UML to know if UML covers my use case.
Imagine an integrated circuit which is made of chips connected with wires. So three chips : A,B,C and Chip A -> Chip B, Chip C. Each chip has input ports and output ports, and some of the output ports of A are connected to the input ports of B, some are connected to those of C. Now Chip A itself can be considered as an integrated circuit, and as such can be decomposed in another graph : Chip A :: Chip A.1, Chip A.3 -> Chip A.2. Each Chip A.1 can also be decomposed etc. This is what is called a compound graph. Each node of a compound graph can have children which are themselves graphs.
I do know about the Harel's and UML's statecharts formalism, which allows for hierarchical graphs to describe hierarchical nested state machines (hence any nested graph can potentially be expressed with statechart formalism), though I do not know any online tool which allow to draw statechart. The question is whether UML has a mechanism or tools which allow to specify a compound graph, and if so, what is the nesting mechanism?
Level 1
-------
---|ChipA|---ChipB---
| |---ChipC---
|-----|
Zoom on Chip A
-----ChipA.1---ChipA.2---- (...linked to ChipB input port
|--ChipA.3-| |-- (...linked to ChipC input port
An example of compound graph from the litterature :
In response to your comment,
The question is whether UML has a mechanism or tools which allow to specify a compound graph, and if so, what is the nesting mechanism?
At the infrastructure level, it uses the Element::ownedElement association to represent containment. The Namespace::ownedMember associate subsets this association, as do the association between classes and their members, packages and nested packages, components and their contained items, and so on. All of these are conceptually compound graphs, but are not rendered in a similar way to you diagrams.
In terms of you 'Chip made up of chips', the CompositeStructures package in the UML superstructure ( section 9 of the 2.4.1 superstructure ) uses these association to create composites, or for a domain specific example built on top of UML then SysML is the UML dialect created for systems engineering.
A systems engineering block diagram consists of blocks created with ports and connectors; blocks can have internal structure of further network of blocks. The connectors carry flows, whic can be information, energy or matter.
An example block diagram for some plant shows the custom rendering used to show owned elements in the same diagram as their containers:
See http://www.omgsysml.org/ or http://sysmlforum.com/sysml-faq/ for further information.
Tools for sysml are listed on the http://sysml.tools/ site.
Start with the component diagram in UML.
Your chips can be treated as components, ports will be... well, ports ;-) but also you should define some sort of interfaces (functions offered/needed by those ports). As I understand it the arrow direction depicts some sort of dependency (so the chip at the end with arrowhead provides something and thus will have a provided interface while the other end will have a required interface and a proper connection between them).
If a specific port is realised with internal structure you'll use a delegation connector for that.
If you create a new application which uses a distributed hash table (DHT), you need to bootstrap the p2p network. I had the idea that you could join an existing DHT (e.g. the Bittorrent DHT).
Is this feasable? Of course, we assume the same technology. Combining Chord with Kademlia is obviously not feasable.
If yes, would this be considered parasitic or symbiotic? Parasitic meaning that it conflicts with the original use somehow. Symbiotic, if it is good for both applications as they support each other.
In general: Kademlia and Chord are just abstract designs, while implementations provide varying functionality.
If its feature-set is too narrow you won't be able to map your application logic onto it. If it's overly broad for your needs it might be a pain to re-implement if no open source library is available.
For bittorrent: The bittorrent DHT provides 20byte key -> List[IP,Port] lookups as its primary feature, where the IP is determined by the sender IP and thus cannot be used to store arbitrary data. There are some secondary features like bloom filter statistics over those lists but they're probably even less useful for other applications.
It does not provide general key-value storage, at least not as part of the core specification. There is an extension proposal for that
Although implementations provide some basic forward-compatibility for unknown message types by treating them like node lookup requests instead of just ignoring them that is only of limited usefulness if your application supplies a small fraction of the nodes, since you're unlikely to encounter other nodes implementing that functionality during a lookup.
If yes, would this be considered parasitic or symbiotic?
That largely depends on whether you are a "good citizen" in the network.
Does your implementation follow the spec, including commonly used extensions?
Does your general use-case stay within an order of magnitude compared to other nodes when it comes to the traffic it causes?
Is the application lifecycle long enough to not lie outside the expected churn rates of the target DHT?
Can someone please provide links to any paper/reference that talks about disconnected components in P2P networks?
I have found this paper. It deals with various P2P networks including kademlia which is the basis of bittorent DHT. It defines a probabilistic metric called routability rather than talking about connectivity but I guess the two things are related. (With high routability the graph is probably connected.) From the paper:
... we consider the measure of
routability, which is deļ¬ned as the
expected number of routable node pairs
divided by the number of possible node
pairs among the surviving nodes. ...
(source: imagehost.org)
One paper calls it the islanding problem, another calls it isolated overlays.