Our DLT network requires high TPS and splitting ordering service into 3 different locations will have a negative impact on the performance as per raft protocol all ordering nodes(deployed on different locations) will take part in consensus. So we have below questions related to this:
Can we specify the consenter set(contains orderer node only from the main location) in raft ordering service so that ordering service will take a consensus from only the orderer node present in the consenter set?
Can we have an orderer leader is chosen from the consenter set(contains orderer node only from the main location)?
Do other orderer nodes than the consenter set do not take active participation in consensus?
Definition for consenter set is referred from https://hyperledger-fabric.readthedocs.io/en/latest/orderer/ordering_service.html
Is there any other way of doing this?
You can define any set of consenters for each channel, but every consenter must be a part of the system channel.
You cannot control the leader of the consenter set, it is chosen by the protocol only.
Yes, if you have, say, n orderer nodes in the network, you can define that only k<n of them will be used for a given channel.
Having said that, Raft supports pipelining of blocks, so even if you deploy the nodes in different parts of the world, you would still have decent throughput.
Related
Here is the log of Hyperledger Fabric test network. There are 2 peer node(peer0.org1.example.com & peer0.org2.example.com) and 2 orderer node(orderer.example.com & orderer2.example.com).
My question is what 「writeBlock」,「StoreBlock」，「Validate」「commit」stage actually does? I know that at 「propose」stage, a chosen orderer(in this case, orderer.example.com) will create a block and then send the created block to other orderer such as orderer2.example.com. However, I don't understand what orderers does at「writeBlock」stage. Do they sign the created block at 「writeBlock」stage? Next, peer nodes received the block from who? orderer.example.com or orderer2.example.com? In my guess, I think it's from orderer.example.com but I'm not sure.
I don't understand what orderers does at「writeBlock」stage.
I'm not sure about those stages (writeBlock looks self-explanatory). I don't know about "StoreBlock", it isn't mentioned in the official docs. I'm wondering which Fabric version you are using.
Next, peer nodes received the block from who?
In a RAFT network you have a leader/follower model. You can think of a leader peer as an entry point/doorman for all communication coming from the ordering nodes. An ordering node sends blocks to the leading peer(s) of each organization; the leading peer is then responsible for distributing those blocks to other peers belonging to the same organization. Distribution is done using gossip protocol.
Hyperledger Fabric is a complex system, with many components and a lot of moving parts.
The log that you are referring to is the log of the orderer node - orderer.example.com. It's a part of the Fabric test network. You can see what's happening in detail by looking at the logs of other Fabric components, such as peer node logs.
The Fabric network has a set of orderer nodes, which are responsible for ordering the transactions and creating blocks in the blockchain.
The orderer nodes are configured to use a consensus algorithm (such as Solo or Kafka) to achieve a consensus about the order of the transactions.
A Fabric network has a set of peer nodes, which are responsible for executing the chaincode (smart contracts) and catching and validating the transactions.
The Fabric test network consists of one orderer node, which is configured to use Solo consensus algorithm. This orderer node is orderer.example.com.
The Fabric test network consists of two peer nodes, which are peer0.org1.example.com and peer0.org2.example.com.
The log that you are referring to is the log of the orderer node - ord
So I do understand the concept of the Raft Protocol.
But I struggle with it in the use case of a Hyperledger Fabric Ordering Service.
What I understand so far:
When a transaction is proposed and endorsed by peers the application forwards it to the ordering service.
The Raft Leader-Node receives this transaction and writes it to his Log then sends it to his Follower-Nodes that also update their Logs.
What I don't understand:
Where and how are the blocks created?
Do all Nodes create a Block and check if a majority has the same?
I read somewhere that a Blockcutter-method is invoked by the Leader and the resulting Block is then proposed to the local Raft Finite State Machine (what is this?).
It's too wasteful to do a consensus round for each transaction. Instead, the Raft leader aggregates several transactions into a batch, and then creates a block from that batch, using a block cutter object.
Then, the Raft leader initiates a consensus round to make the block be replicated to all followers via the Raft protocol.
In Hyperledger Fabric, chaincode can contain non-deterministic funcitons, and necessary consensus percentage can be customized.
What happens if chain code is non-deterministic and the output is different in some nodes, but consensus was passed.
How syncronizing is archived ?
Can new node joined this channel have same transaction history?
Since the endorsement passed it will be commited then syncronized by gossip with all nodes.
The nodes on committing(VSCC) only verify the signature and if the minimun necessary set of endorsers was reached.
For example if you require 2 out of 3 Orgs to agree, when you have the minimum necessary(2) agreeing it will be commited without the need of the third org.
I am using orderer in kafka mode. Now while invoking chaincode, I need to supply orderer name. But then whats the use of kafka to select orderer if I need to supply the orderer name by my own.
I'll note that the client can initialize a channel in memory that has record of multiple orderers, and the SDK should provide the option of sending your transaction via a random orderer. While one organization's client may communicate with one orderer, another organization might prefer to have its client set up to use a different orderer (or group of orderers, and perhaps these are running on the organizations own servers).
Where kafka comes in is that it's a way to provide crash-fault tolerance to channels with high throughput and a set up of multiple orderers by helping keep track of transactions and thus allowing proper sequencing of blocks. Specifically, when the client sends the transaction to an orderer, the orderer then relays to a partition that the kafka cluster maintains, and then orderers then consume/read from this partition to package transactions into blocks (orderers are both producers and consumers in this set up). kafka keeps all the orderers in sync by maintaining a stream of transactions that's used by all of them.
The full technical solution is outlined in https://docs.google.com/document/d/19JihmW-8blTzN99lAubOfseLUZqdrB6sBR0HsRgCAnY/edit, the below image is from page 11.
From the readthedocs page (https://hyperledger-fabric.readthedocs.io/en/release-1.2/kafka.html):
Each channel maps to a separate single-partition topic in Kafka. When an OSN receives transactions via the Broadcast RPC, it checks to make sure that the broadcasting client has permissions to write on the channel, then relays (i.e. produces) those transactions to the appropriate partition in Kafka. This partition is also consumed by the OSN which groups the received transactions into blocks locally, persists them in its local ledger, and serves them to receiving clients via the Deliver RPC. For low-level details, refer to the document that describes how we came to this design — Figure 8 is a schematic representation of the process described above.
All,
As I know, in Hyperledger Fabric environment, an orderer delivers messages to peers. If there is an off-line peer. How is the message delivered to the peer when it recovers to ON-LINE? How does the orderer know the peer recovers to ON-LINE?
Regards,
Count
When the peer comes back online it will get blocks as follows:
If the peer is a gossip leader in the org, then it requests a stream
of blocks from an orderer via deliver API, starting at the peers
current block height.
If the peer is not a gossip leader and is
current or just a little behind (within a small threshold), then it
gets blocks via gossip from the org's leader peer (or potentially
another peer in the org).
If the peer is not a gossip leader and is
way behind (beyond the threshold), then it gets blocks via block
transfer from another peer (can be cross-org).
For more details see the gossip data dissemination layer documentation.
I'd like to add more details on the flow of state replication. First of all ordering service provides following API:
service AtomicBroadcast {
rpc Broadcast(stream common.Envelope) returns (stream BroadcastResponse) {}
rpc Deliver(stream common.Envelope) returns (stream DeliverResponse) {}
}
Where Broadcast used to send transactions for ordering to the ordering service and Deliver used by peers to request blocks from the ordering service starting from certain position. Possible options for Deliver are
message SeekPosition {
oneof Type {
SeekNewest newest = 1;
SeekOldest oldest = 2;
SeekSpecified specified = 3;
}
}
There are two possible mode of operations for the peer considering status synchronization and replication of the ledger. Generally speaking ledger blocks distributed across the network by using gossip algorithm. In order to prevent all peers connecting to the ordering service there is a notion of leader election, e.g. for each organization there is one peer selected to open connection to the ordering service and start pulling blocks and forward them to the gossip layer to make them distributed between peers in the network. Before start leader check what is the ledger height and ask to deliver new blocks starting from it, all peers monitor availability of the leader and initiate new leader election if needed.
Now, there is additional background process which collects state messages from the peers and if it discovers there is a gap between local ledger height and height of others peers in the network it will initiate replication of missing blocks, via "state transfer" process as was mentioned by #Dave.
Hence, to conclude if peers comes online and for some reason elected to be a leader it will replicate ledger blocks directly from ordering service, otherwise he will get either via gossip layer or state transfer in case of significant gap in ledger height.
You can refer to this image for a brief overview