So I do understand the concept of the Raft Protocol.
But I struggle with it in the use case of a Hyperledger Fabric Ordering Service.
What I understand so far:
When a transaction is proposed and endorsed by peers the application forwards it to the ordering service.
The Raft Leader-Node receives this transaction and writes it to his Log then sends it to his Follower-Nodes that also update their Logs.
What I don't understand:
Where and how are the blocks created?
Do all Nodes create a Block and check if a majority has the same?
I read somewhere that a Blockcutter-method is invoked by the Leader and the resulting Block is then proposed to the local Raft Finite State Machine (what is this?).
It's too wasteful to do a consensus round for each transaction. Instead, the Raft leader aggregates several transactions into a batch, and then creates a block from that batch, using a block cutter object.
Then, the Raft leader initiates a consensus round to make the block be replicated to all followers via the Raft protocol.
Related
In Hyperledger Fabric, what is the expected behavior of peer when all orderer nodes are down.
Should peer also down, or stop serving request from client, or continue to serve query request?
In our test, after orderers are stopped, the peer keeps writing "failed to create connection to orderer" log. When we query a key by calling chaincode the value is returned.
Can you help clarify if this is expected behavior. Thank you.
I am working on a distributed hyperledger fabric network. I would recommend the Orderer Raft Consensus https://hyperledger-fabric.readthedocs.io/en/release-2.2/orderer/ordering_service.html#ordering-service-implementations.
I have solved this in such a way that in my case I have three orderers that run independently on different environments.
If I crash all these orderers, the peer containers will continue to run on the other participants of the network. As you said, they cannot make any transactions.
If one of my orderers crashes it is not so bad after the raft consensus, the containers keep running. If another one fails, no transactions can be made. In this case I let the peers continue and check if the orderers are available again.
The behaviour you described I would put down to the fact that the peer requests the value from his ledger, he doesn't need an orderer for that. https://hyperledger.github.io/fabric-chaincode-node/master/api/fabric-shim.ChaincodeStub.html#getState
Have a read of this: https://github.com/hyperledger/fabric/blob/master/docs/source/peers/peers.md This is the best documentation for how the system works I've found and there's more in the docs directory on the repo for orderers, etc.
My understanding is: The peers are there to sign (endorse) transaction proposals. The orderer exists to order, validate, package and distribute transactions to peers. The peers can also distribute their knowledge of validated transactions via the gossip channel.
If all orderers go down, the transactions will not be validated/packaged/distributed so the blockchain will be out of action until the orderers are restored.
When we query a key by calling chaincode the value is returned.
Peers will still remain up and ready to sign/endorse transaction proposals, and querying the blockchain held at the peers will still work. Chaincodes are hosted by the peers. Orderers do not host chaincode.
Also see here https://github.com/hyperledger/fabric/blob/master/docs/source/orderer/ordering_service.md#ordering-service-implementations for the various modes the orderer can be run in: Raft mode, Kafka ordering, Solo ordering.
I think the current observerd behavour is expected and in my view it is just fine.
Let's check the purpose of orderer?
Order the transactions
Cut the block and distribute the block amongst the orgs when the criteria is met ( min txn/size or time).
This also means, orderer is needed when your Fabric network is processing those transactions which intend to write data into the ledger, isnt it? And Query is not a transaction which writes into the ledger. So it doesn't need the orderer. For query, it will pick up the data from the peer's local database.
So I think what could be done is, to send out an alert to the production support when your application detect orderer node down ( with some health check ? ). And your application displays a dimnished capacity/limited operations message while work on bringing up the orderer network, the system can still serve the search queries.
From my view, its just fantastic. But its finally upto you. Cheers!!
Our DLT network requires high TPS and splitting ordering service into 3 different locations will have a negative impact on the performance as per raft protocol all ordering nodes(deployed on different locations) will take part in consensus. So we have below questions related to this:
Can we specify the consenter set(contains orderer node only from the main location) in raft ordering service so that ordering service will take a consensus from only the orderer node present in the consenter set?
Can we have an orderer leader is chosen from the consenter set(contains orderer node only from the main location)?
Do other orderer nodes than the consenter set do not take active participation in consensus?
Definition for consenter set is referred from https://hyperledger-fabric.readthedocs.io/en/latest/orderer/ordering_service.html
Is there any other way of doing this?
You can define any set of consenters for each channel, but every consenter must be a part of the system channel.
You cannot control the leader of the consenter set, it is chosen by the protocol only.
Yes, if you have, say, n orderer nodes in the network, you can define that only k<n of them will be used for a given channel.
Having said that, Raft supports pipelining of blocks, so even if you deploy the nodes in different parts of the world, you would still have decent throughput.
We have 2 servers with a peer, orderer and kafka each. They are connected in the same channel, both have a chaincode installed and instantiated and the policy is one organization or the other.
Imagine that the internet goes down and they disconnect:
Would both work individually?
Can the write new transactions to the ledger?
What would happen with the new submited blocks in the ledger when the internet goes up and running? How do this new blocks synchronize?
Thanks
EDIT1:
See image for clarification:
How would the network syncrhonize If during the disconnection both write to the ledger, how are those new generated blocks react? One gets invalidated? Or both are valid?
The peers once disconnected won't receive keep alive from the channel peers and will keep throwing the same if you have debug logging enabled.
The peer won't lose any config even though it got disconnected from network. The discovery service in fabric takes care of finding the peers configured in the channel. So, Once the connection resumes it will automatically re-synchronize with the peers with gossip messages.
The peers can then write and read from ledger as usual.
There are multiple things to consider here:
1) When you use a Kafka-based orderer, you will have to cluster the Kafka brokers if you expect them to be part of the same ordering service. Kafka is used to distribute the messages to the ordering nodes. If your Kafka brokers are not in a cluster, then you will have separate ordering services. Recall that Kafka also requires Zookeeper as well. Zookeeper has a 2f+1 fault tolerance model, so if you want to tolerate failure of a single node (failure includes communication issues), you will need at least 3 Zookeeper nodes and they should be deployed on separate hosts. For Kafka, you will want at least 2 brokers and would need to set the minimum ISRs (in sync replicas) to 2. Ideally you'd have 4 Kafka brokers.
2) In order for transactions to be processed, enough peers to satisfy the endorsement policy as well as the ordering service must be available / accessible. Peers which cannot connect to the ordering service will catch up once they can reestablish connectivity.
I am using orderer in kafka mode. Now while invoking chaincode, I need to supply orderer name. But then whats the use of kafka to select orderer if I need to supply the orderer name by my own.
I'll note that the client can initialize a channel in memory that has record of multiple orderers, and the SDK should provide the option of sending your transaction via a random orderer. While one organization's client may communicate with one orderer, another organization might prefer to have its client set up to use a different orderer (or group of orderers, and perhaps these are running on the organizations own servers).
Where kafka comes in is that it's a way to provide crash-fault tolerance to channels with high throughput and a set up of multiple orderers by helping keep track of transactions and thus allowing proper sequencing of blocks. Specifically, when the client sends the transaction to an orderer, the orderer then relays to a partition that the kafka cluster maintains, and then orderers then consume/read from this partition to package transactions into blocks (orderers are both producers and consumers in this set up). kafka keeps all the orderers in sync by maintaining a stream of transactions that's used by all of them.
The full technical solution is outlined in https://docs.google.com/document/d/19JihmW-8blTzN99lAubOfseLUZqdrB6sBR0HsRgCAnY/edit, the below image is from page 11.
From the readthedocs page (https://hyperledger-fabric.readthedocs.io/en/release-1.2/kafka.html):
Each channel maps to a separate single-partition topic in Kafka. When an OSN receives transactions via the Broadcast RPC, it checks to make sure that the broadcasting client has permissions to write on the channel, then relays (i.e. produces) those transactions to the appropriate partition in Kafka. This partition is also consumed by the OSN which groups the received transactions into blocks locally, persists them in its local ledger, and serves them to receiving clients via the Deliver RPC. For low-level details, refer to the document that describes how we came to this design — Figure 8 is a schematic representation of the process described above.
All,
As I know, in Hyperledger Fabric environment, an orderer delivers messages to peers. If there is an off-line peer. How is the message delivered to the peer when it recovers to ON-LINE? How does the orderer know the peer recovers to ON-LINE?
Regards,
Count
When the peer comes back online it will get blocks as follows:
If the peer is a gossip leader in the org, then it requests a stream
of blocks from an orderer via deliver API, starting at the peers
current block height.
If the peer is not a gossip leader and is
current or just a little behind (within a small threshold), then it
gets blocks via gossip from the org's leader peer (or potentially
another peer in the org).
If the peer is not a gossip leader and is
way behind (beyond the threshold), then it gets blocks via block
transfer from another peer (can be cross-org).
For more details see the gossip data dissemination layer documentation.
I'd like to add more details on the flow of state replication. First of all ordering service provides following API:
service AtomicBroadcast {
rpc Broadcast(stream common.Envelope) returns (stream BroadcastResponse) {}
rpc Deliver(stream common.Envelope) returns (stream DeliverResponse) {}
}
Where Broadcast used to send transactions for ordering to the ordering service and Deliver used by peers to request blocks from the ordering service starting from certain position. Possible options for Deliver are
message SeekPosition {
oneof Type {
SeekNewest newest = 1;
SeekOldest oldest = 2;
SeekSpecified specified = 3;
}
}
There are two possible mode of operations for the peer considering status synchronization and replication of the ledger. Generally speaking ledger blocks distributed across the network by using gossip algorithm. In order to prevent all peers connecting to the ordering service there is a notion of leader election, e.g. for each organization there is one peer selected to open connection to the ordering service and start pulling blocks and forward them to the gossip layer to make them distributed between peers in the network. Before start leader check what is the ledger height and ask to deliver new blocks starting from it, all peers monitor availability of the leader and initiate new leader election if needed.
Now, there is additional background process which collects state messages from the peers and if it discovers there is a gap between local ledger height and height of others peers in the network it will initiate replication of missing blocks, via "state transfer" process as was mentioned by #Dave.
Hence, to conclude if peers comes online and for some reason elected to be a leader it will replicate ledger blocks directly from ordering service, otherwise he will get either via gossip layer or state transfer in case of significant gap in ledger height.
You can refer to this image for a brief overview