How kafka consensus works in hyperledger fabric.?
If i have three orderers which one will create block.?
How messages will be delivered to all the peers.? are they fetched from kafka or orderer is responsible for delivering them.?
Consensus in blockchain involves nodes agreeing on the same order of transactions.
Ordering nodes send to Kafka transactions, and receive from Kafka transactions in the same order, since Kafka presents an abstraction of a shared queue.
All orderers create blocks when they read enough messages or enough data from kafka. Also - if a transaction was sent but no block was created, and enough time (timeout) passed - the orderer node would send a special message to Kafka that would signal all ordering nodes to cut a block. This ensures that all orderers cut blocks based on timeout, but also that they cut the same blocks.
Each leader peer connects to a random orderer and then sends a request, saying from what block index it wants to receive blocks. Then - the orderer reads the blocks from its file system and sends them to the peer. When the peer receives the blocks, it also sends them to other peers via the gossip component inside the peer, which ensures peers stay in sync.
Related
We have 2 servers with a peer, orderer and kafka each. They are connected in the same channel, both have a chaincode installed and instantiated and the policy is one organization or the other.
Imagine that the internet goes down and they disconnect:
Would both work individually?
Can the write new transactions to the ledger?
What would happen with the new submited blocks in the ledger when the internet goes up and running? How do this new blocks synchronize?
Thanks
EDIT1:
See image for clarification:
How would the network syncrhonize If during the disconnection both write to the ledger, how are those new generated blocks react? One gets invalidated? Or both are valid?
The peers once disconnected won't receive keep alive from the channel peers and will keep throwing the same if you have debug logging enabled.
The peer won't lose any config even though it got disconnected from network. The discovery service in fabric takes care of finding the peers configured in the channel. So, Once the connection resumes it will automatically re-synchronize with the peers with gossip messages.
The peers can then write and read from ledger as usual.
There are multiple things to consider here:
1) When you use a Kafka-based orderer, you will have to cluster the Kafka brokers if you expect them to be part of the same ordering service. Kafka is used to distribute the messages to the ordering nodes. If your Kafka brokers are not in a cluster, then you will have separate ordering services. Recall that Kafka also requires Zookeeper as well. Zookeeper has a 2f+1 fault tolerance model, so if you want to tolerate failure of a single node (failure includes communication issues), you will need at least 3 Zookeeper nodes and they should be deployed on separate hosts. For Kafka, you will want at least 2 brokers and would need to set the minimum ISRs (in sync replicas) to 2. Ideally you'd have 4 Kafka brokers.
2) In order for transactions to be processed, enough peers to satisfy the endorsement policy as well as the ordering service must be available / accessible. Peers which cannot connect to the ordering service will catch up once they can reestablish connectivity.
Does the gossip protocol is used only to spread the committed blocks via all peers on the "order" stage? Does it work on the “execute” stage, when the endorsing peers execute and sign the transactions?
The gossip component is the peer's communication layer, and it contributes to the block dissemination via having peers distribute blocks among themselves instead of having all of them pull them directly from the orderer nodes.
It works roughly like this:
When the orderer cuts a new block, it appends it to its local file system.
In every organization, there is usually a peer that is selected among the peers in its organization (for that channel) to pull blocks from the orderer, and forward them to other peers in its organziation. The other peers also forward the blocks even further to other peers, etc. etc. and this all happens via a "gossip" style protocol.
I recommend that you read the gossip section in the documentation.
I am using orderer in kafka mode. Now while invoking chaincode, I need to supply orderer name. But then whats the use of kafka to select orderer if I need to supply the orderer name by my own.
I'll note that the client can initialize a channel in memory that has record of multiple orderers, and the SDK should provide the option of sending your transaction via a random orderer. While one organization's client may communicate with one orderer, another organization might prefer to have its client set up to use a different orderer (or group of orderers, and perhaps these are running on the organizations own servers).
Where kafka comes in is that it's a way to provide crash-fault tolerance to channels with high throughput and a set up of multiple orderers by helping keep track of transactions and thus allowing proper sequencing of blocks. Specifically, when the client sends the transaction to an orderer, the orderer then relays to a partition that the kafka cluster maintains, and then orderers then consume/read from this partition to package transactions into blocks (orderers are both producers and consumers in this set up). kafka keeps all the orderers in sync by maintaining a stream of transactions that's used by all of them.
The full technical solution is outlined in https://docs.google.com/document/d/19JihmW-8blTzN99lAubOfseLUZqdrB6sBR0HsRgCAnY/edit, the below image is from page 11.
From the readthedocs page (https://hyperledger-fabric.readthedocs.io/en/release-1.2/kafka.html):
Each channel maps to a separate single-partition topic in Kafka. When an OSN receives transactions via the Broadcast RPC, it checks to make sure that the broadcasting client has permissions to write on the channel, then relays (i.e. produces) those transactions to the appropriate partition in Kafka. This partition is also consumed by the OSN which groups the received transactions into blocks locally, persists them in its local ledger, and serves them to receiving clients via the Deliver RPC. For low-level details, refer to the document that describes how we came to this design — Figure 8 is a schematic representation of the process described above.
What parameter is used for orderer to order transactions?
by timestamp issued in an endorser
by sequence number issued in a client or an endorser
received by an orderer in a sequential order
depends on configuration
other
In general, the orderer makes no guarantees about the order in which messages will be delivered just that messages will be delivered in the same order to all peer nodes.
In practice, the following generally holds true for the current orderer implementations:
Solo - messages should be delivered in the order in which they were received
Kafka - messages should be delivered in the order in which they were received by each orderer node and generally even in the order they are received across multiple ordering nodes.
But as mentioned, neither of the above is guaranteed by any ordering implementation
All,
As I know, in Hyperledger Fabric environment, an orderer delivers messages to peers. If there is an off-line peer. How is the message delivered to the peer when it recovers to ON-LINE? How does the orderer know the peer recovers to ON-LINE?
Regards,
Count
When the peer comes back online it will get blocks as follows:
If the peer is a gossip leader in the org, then it requests a stream
of blocks from an orderer via deliver API, starting at the peers
current block height.
If the peer is not a gossip leader and is
current or just a little behind (within a small threshold), then it
gets blocks via gossip from the org's leader peer (or potentially
another peer in the org).
If the peer is not a gossip leader and is
way behind (beyond the threshold), then it gets blocks via block
transfer from another peer (can be cross-org).
For more details see the gossip data dissemination layer documentation.
I'd like to add more details on the flow of state replication. First of all ordering service provides following API:
service AtomicBroadcast {
rpc Broadcast(stream common.Envelope) returns (stream BroadcastResponse) {}
rpc Deliver(stream common.Envelope) returns (stream DeliverResponse) {}
}
Where Broadcast used to send transactions for ordering to the ordering service and Deliver used by peers to request blocks from the ordering service starting from certain position. Possible options for Deliver are
message SeekPosition {
oneof Type {
SeekNewest newest = 1;
SeekOldest oldest = 2;
SeekSpecified specified = 3;
}
}
There are two possible mode of operations for the peer considering status synchronization and replication of the ledger. Generally speaking ledger blocks distributed across the network by using gossip algorithm. In order to prevent all peers connecting to the ordering service there is a notion of leader election, e.g. for each organization there is one peer selected to open connection to the ordering service and start pulling blocks and forward them to the gossip layer to make them distributed between peers in the network. Before start leader check what is the ledger height and ask to deliver new blocks starting from it, all peers monitor availability of the leader and initiate new leader election if needed.
Now, there is additional background process which collects state messages from the peers and if it discovers there is a gap between local ledger height and height of others peers in the network it will initiate replication of missing blocks, via "state transfer" process as was mentioned by #Dave.
Hence, to conclude if peers comes online and for some reason elected to be a leader it will replicate ledger blocks directly from ordering service, otherwise he will get either via gossip layer or state transfer in case of significant gap in ledger height.
You can refer to this image for a brief overview