We have 2 servers with a peer, orderer and kafka each. They are connected in the same channel, both have a chaincode installed and instantiated and the policy is one organization or the other.
Imagine that the internet goes down and they disconnect:
Would both work individually?
Can the write new transactions to the ledger?
What would happen with the new submited blocks in the ledger when the internet goes up and running? How do this new blocks synchronize?
Thanks
EDIT1:
See image for clarification:
How would the network syncrhonize If during the disconnection both write to the ledger, how are those new generated blocks react? One gets invalidated? Or both are valid?
The peers once disconnected won't receive keep alive from the channel peers and will keep throwing the same if you have debug logging enabled.
The peer won't lose any config even though it got disconnected from network. The discovery service in fabric takes care of finding the peers configured in the channel. So, Once the connection resumes it will automatically re-synchronize with the peers with gossip messages.
The peers can then write and read from ledger as usual.
There are multiple things to consider here:
1) When you use a Kafka-based orderer, you will have to cluster the Kafka brokers if you expect them to be part of the same ordering service. Kafka is used to distribute the messages to the ordering nodes. If your Kafka brokers are not in a cluster, then you will have separate ordering services. Recall that Kafka also requires Zookeeper as well. Zookeeper has a 2f+1 fault tolerance model, so if you want to tolerate failure of a single node (failure includes communication issues), you will need at least 3 Zookeeper nodes and they should be deployed on separate hosts. For Kafka, you will want at least 2 brokers and would need to set the minimum ISRs (in sync replicas) to 2. Ideally you'd have 4 Kafka brokers.
2) In order for transactions to be processed, enough peers to satisfy the endorsement policy as well as the ordering service must be available / accessible. Peers which cannot connect to the ordering service will catch up once they can reestablish connectivity.
Related
So I do understand the concept of the Raft Protocol.
But I struggle with it in the use case of a Hyperledger Fabric Ordering Service.
What I understand so far:
When a transaction is proposed and endorsed by peers the application forwards it to the ordering service.
The Raft Leader-Node receives this transaction and writes it to his Log then sends it to his Follower-Nodes that also update their Logs.
What I don't understand:
Where and how are the blocks created?
Do all Nodes create a Block and check if a majority has the same?
I read somewhere that a Blockcutter-method is invoked by the Leader and the resulting Block is then proposed to the local Raft Finite State Machine (what is this?).
It's too wasteful to do a consensus round for each transaction. Instead, the Raft leader aggregates several transactions into a batch, and then creates a block from that batch, using a block cutter object.
Then, the Raft leader initiates a consensus round to make the block be replicated to all followers via the Raft protocol.
How kafka consensus works in hyperledger fabric.?
If i have three orderers which one will create block.?
How messages will be delivered to all the peers.? are they fetched from kafka or orderer is responsible for delivering them.?
Consensus in blockchain involves nodes agreeing on the same order of transactions.
Ordering nodes send to Kafka transactions, and receive from Kafka transactions in the same order, since Kafka presents an abstraction of a shared queue.
All orderers create blocks when they read enough messages or enough data from kafka. Also - if a transaction was sent but no block was created, and enough time (timeout) passed - the orderer node would send a special message to Kafka that would signal all ordering nodes to cut a block. This ensures that all orderers cut blocks based on timeout, but also that they cut the same blocks.
Each leader peer connects to a random orderer and then sends a request, saying from what block index it wants to receive blocks. Then - the orderer reads the blocks from its file system and sends them to the peer. When the peer receives the blocks, it also sends them to other peers via the gossip component inside the peer, which ensures peers stay in sync.
I am using orderer in kafka mode. Now while invoking chaincode, I need to supply orderer name. But then whats the use of kafka to select orderer if I need to supply the orderer name by my own.
I'll note that the client can initialize a channel in memory that has record of multiple orderers, and the SDK should provide the option of sending your transaction via a random orderer. While one organization's client may communicate with one orderer, another organization might prefer to have its client set up to use a different orderer (or group of orderers, and perhaps these are running on the organizations own servers).
Where kafka comes in is that it's a way to provide crash-fault tolerance to channels with high throughput and a set up of multiple orderers by helping keep track of transactions and thus allowing proper sequencing of blocks. Specifically, when the client sends the transaction to an orderer, the orderer then relays to a partition that the kafka cluster maintains, and then orderers then consume/read from this partition to package transactions into blocks (orderers are both producers and consumers in this set up). kafka keeps all the orderers in sync by maintaining a stream of transactions that's used by all of them.
The full technical solution is outlined in https://docs.google.com/document/d/19JihmW-8blTzN99lAubOfseLUZqdrB6sBR0HsRgCAnY/edit, the below image is from page 11.
From the readthedocs page (https://hyperledger-fabric.readthedocs.io/en/release-1.2/kafka.html):
Each channel maps to a separate single-partition topic in Kafka. When an OSN receives transactions via the Broadcast RPC, it checks to make sure that the broadcasting client has permissions to write on the channel, then relays (i.e. produces) those transactions to the appropriate partition in Kafka. This partition is also consumed by the OSN which groups the received transactions into blocks locally, persists them in its local ledger, and serves them to receiving clients via the Deliver RPC. For low-level details, refer to the document that describes how we came to this design — Figure 8 is a schematic representation of the process described above.
My question is simple - once the Orderer Service Network is created and is up and running, is it possible to extend it with new nodes acting as Orderers, Kafkas and Zookeepers?
The scenario I am thinking is when each participant in the network owns a stake at both peer and OSN layer. Therefore, a new participant should join the network with both peer(s), orderer(s), kafka node(s) and zookeeper node(s).
I am aware of recommended limits related to the size of Kafka/Zookeeper cluster; however, I am curious about technical possibility of successfully executing the above scenario.
Yes it's possible:
If you add an orderer node you need it to wait to synchronize via fetching the transactions from kafka.
If you add a zookeeper node, well - consult the zookeeper manual, but - it's possible.
I haven't tried but i guess it should be the same for kafka too.
All,
As I know, in Hyperledger Fabric environment, an orderer delivers messages to peers. If there is an off-line peer. How is the message delivered to the peer when it recovers to ON-LINE? How does the orderer know the peer recovers to ON-LINE?
Regards,
Count
When the peer comes back online it will get blocks as follows:
If the peer is a gossip leader in the org, then it requests a stream
of blocks from an orderer via deliver API, starting at the peers
current block height.
If the peer is not a gossip leader and is
current or just a little behind (within a small threshold), then it
gets blocks via gossip from the org's leader peer (or potentially
another peer in the org).
If the peer is not a gossip leader and is
way behind (beyond the threshold), then it gets blocks via block
transfer from another peer (can be cross-org).
For more details see the gossip data dissemination layer documentation.
I'd like to add more details on the flow of state replication. First of all ordering service provides following API:
service AtomicBroadcast {
rpc Broadcast(stream common.Envelope) returns (stream BroadcastResponse) {}
rpc Deliver(stream common.Envelope) returns (stream DeliverResponse) {}
}
Where Broadcast used to send transactions for ordering to the ordering service and Deliver used by peers to request blocks from the ordering service starting from certain position. Possible options for Deliver are
message SeekPosition {
oneof Type {
SeekNewest newest = 1;
SeekOldest oldest = 2;
SeekSpecified specified = 3;
}
}
There are two possible mode of operations for the peer considering status synchronization and replication of the ledger. Generally speaking ledger blocks distributed across the network by using gossip algorithm. In order to prevent all peers connecting to the ordering service there is a notion of leader election, e.g. for each organization there is one peer selected to open connection to the ordering service and start pulling blocks and forward them to the gossip layer to make them distributed between peers in the network. Before start leader check what is the ledger height and ask to deliver new blocks starting from it, all peers monitor availability of the leader and initiate new leader election if needed.
Now, there is additional background process which collects state messages from the peers and if it discovers there is a gap between local ledger height and height of others peers in the network it will initiate replication of missing blocks, via "state transfer" process as was mentioned by #Dave.
Hence, to conclude if peers comes online and for some reason elected to be a leader it will replicate ledger blocks directly from ordering service, otherwise he will get either via gossip layer or state transfer in case of significant gap in ledger height.
You can refer to this image for a brief overview