I built Fabric network using Kafka-based orderer service.
(2 peer, 1 orderer and 4 kafka)
When I invoke a chaincode, below message appears in peer's log.
[gossip/state] queueNewMessage -> WARN 7a24 Failed adding payload: Ledger height is at 5685, cannot enqueue block with sequence of 938
What is this message?
Can I ignore this warning?
This is probably because the second peer's height is much less than the first peer's height, and it (the second peer) pulls from the orderer blocks that the first peer doesn't need, and forwards to it as a token of good will.
The peer logs that he got a block he doesn't need.
Related
we have done a migration of a Fabric network, from one tenant to another Azure Cloud. The nodes are on three IaaS and we have done a snpashot of them, the network did not stop . So, as a result when we use this snpashots on the new tenant, it looks like the peers have more blocks than the orderers. Orderers have RAFT consensus without problems and update their blocks. But they can't get the latest blocks from the peers.
I can run chaincode, and read values, but cannot insert new transanctions.
There is this error in the peers :
"2023-01-17 13:20:56.933 UTC [peer.blocksprovider] func1 -> WARN 269 Encountered an error reading from deliver stream: EOF channel=test orderer-address=orderer5.example.com:7050
2023-01-17 13:20:56.933 UTC [peer.blocksprovider] DeliverBlocks -> WARN 26a Got error while attempting to receive blocks: received bad status NOT_FOUND from orderer channel=test"
.
and on the orderer: " ERRO 05b [channel: test] Error reading from channel, cause was: NOT_FOUND".
It seems that the peer to request blocks the orderer does not know the channel, but when it starts it detects it and inserts the blocks it had pending from other orderers.
Can the orderers read the missing blocks from the peers, or the only way to start a network from snpashots from different IaaS is to stop the network first?
Here is the log of Hyperledger Fabric test network. There are 2 peer node(peer0.org1.example.com & peer0.org2.example.com) and 2 orderer node(orderer.example.com & orderer2.example.com).
My question is what 「writeBlock」,「StoreBlock」，「Validate」「commit」stage actually does? I know that at 「propose」stage, a chosen orderer(in this case, orderer.example.com) will create a block and then send the created block to other orderer such as orderer2.example.com. However, I don't understand what orderers does at「writeBlock」stage. Do they sign the created block at 「writeBlock」stage? Next, peer nodes received the block from who? orderer.example.com or orderer2.example.com? In my guess, I think it's from orderer.example.com but I'm not sure.
I don't understand what orderers does at「writeBlock」stage.
I'm not sure about those stages (writeBlock looks self-explanatory). I don't know about "StoreBlock", it isn't mentioned in the official docs. I'm wondering which Fabric version you are using.
Next, peer nodes received the block from who?
In a RAFT network you have a leader/follower model. You can think of a leader peer as an entry point/doorman for all communication coming from the ordering nodes. An ordering node sends blocks to the leading peer(s) of each organization; the leading peer is then responsible for distributing those blocks to other peers belonging to the same organization. Distribution is done using gossip protocol.
Hyperledger Fabric is a complex system, with many components and a lot of moving parts.
The log that you are referring to is the log of the orderer node - orderer.example.com. It's a part of the Fabric test network. You can see what's happening in detail by looking at the logs of other Fabric components, such as peer node logs.
The Fabric network has a set of orderer nodes, which are responsible for ordering the transactions and creating blocks in the blockchain.
The orderer nodes are configured to use a consensus algorithm (such as Solo or Kafka) to achieve a consensus about the order of the transactions.
A Fabric network has a set of peer nodes, which are responsible for executing the chaincode (smart contracts) and catching and validating the transactions.
The Fabric test network consists of one orderer node, which is configured to use Solo consensus algorithm. This orderer node is orderer.example.com.
The Fabric test network consists of two peer nodes, which are peer0.org1.example.com and peer0.org2.example.com.
The log that you are referring to is the log of the orderer node - ord
I have query, i am reading the hyperledger fabric article, In article its mention that, User User1 sends a transaction Tx1 to Peer P1, Peer P1 endose E1, the transaction Tx1, the transaction received back to user1 and he pass it to order or1, And order will validate the Tx1 and create the block and it will commit.
My Question is :
1. when order is committing the block, why it will send to peer P1 back.
2. In article they mention Peer P1 is also validate Tx1, How Peer will
verify the Tx1.
3. If Peer P1 verify the Tx1, does Peer P1 also keep the record of all the Peers and Orderer details.
4. If Peer also can Validate the Tx1, best thing is to apply the
block/Batch creation mechanism to Peer directly, so no need of orderer. i have doubt please suggest me.
The application “broadcasts” the transaction proposal and response
within a “transaction message” to the ordering service. The
transaction will contain the read/write sets, the endorsing peers
signatures and the Channel ID. The ordering service does not need to
inspect the entire content of a transaction in order to perform its
operation, it simply receives transactions from all channels in the
network, orders them chronologically by channel, and creates blocks of
transactions per channel.
Read this doc: https://hyperledger-fabric.readthedocs.io/en/release-1.4/txflow.html
However, I will answer to your questions:
when order is committing the block, why it will send to peer P1 back.
A) Orderer will just receive pool of transactions and form blocks and ship to committing peers, it will not do any validation and it creates blocks so orderer also will have a copy of the ledger
In the article they mention Peer P1 is also validate Tx1, How Peer will
verify the Tx1.
A) If Peer1 is an endorsing peer then it will simulate the ledger by making the transaction commit and it will record the behavior it will send back to the user.
If Peer P1 verify the Tx1, does Peer P1 also keep the record of all the Peers and Orderer details.
A) By default all endorsing peers are commiting peers, so yes it will keep
If Peer also can Validate the Tx1, best thing is to apply the
block/Batch creation mechanism to Peer directly, so no need of orderer. i have doubt please suggest me.
A) Here if you come from ethereum background orderer is like a miner, here he will not do any calculations simply form batches and ship to peers like a postman job.
While adding a new organization to the hyper ledger fabric network, the peer logs shows the below error.
[gossip/service] updateAnchors -> ERRO 327 Tried joining channel
mychannel but our org( org4), isn't among the orgs of the channel:
[org1 org2 org3] , aborting.
And a warning like below.
[gossip/gossip] Gossip -> WARN 3d9 Failed obtaining gossipChannel of
[99 104 45 99 111 109 109 111 110] aborting.
Hyper ledger Fabric version : v1.1.0
Dynamic leader election properties added into peer-base.
I was following the instructions here.
Because the peer is being joined with the genesis block (which knows about the original members of the channel, but not the new members), these log statements are emitted.
Usually, these statements will be benign; so long as the peer is able to retrieve and commit blocks, you may ignore these messages. Eventually, the peer will receive the config block which adds its org, and gossip networking will begin working properly again.
This is a known issue, and in the future, we hope to allow peers to be joined to a channel using the latest configuration block, and not only the genesis block.
All,
As I know, in Hyperledger Fabric environment, an orderer delivers messages to peers. If there is an off-line peer. How is the message delivered to the peer when it recovers to ON-LINE? How does the orderer know the peer recovers to ON-LINE?
Regards,
Count
When the peer comes back online it will get blocks as follows:
If the peer is a gossip leader in the org, then it requests a stream
of blocks from an orderer via deliver API, starting at the peers
current block height.
If the peer is not a gossip leader and is
current or just a little behind (within a small threshold), then it
gets blocks via gossip from the org's leader peer (or potentially
another peer in the org).
If the peer is not a gossip leader and is
way behind (beyond the threshold), then it gets blocks via block
transfer from another peer (can be cross-org).
For more details see the gossip data dissemination layer documentation.
I'd like to add more details on the flow of state replication. First of all ordering service provides following API:
service AtomicBroadcast {
rpc Broadcast(stream common.Envelope) returns (stream BroadcastResponse) {}
rpc Deliver(stream common.Envelope) returns (stream DeliverResponse) {}
}
Where Broadcast used to send transactions for ordering to the ordering service and Deliver used by peers to request blocks from the ordering service starting from certain position. Possible options for Deliver are
message SeekPosition {
oneof Type {
SeekNewest newest = 1;
SeekOldest oldest = 2;
SeekSpecified specified = 3;
}
}
There are two possible mode of operations for the peer considering status synchronization and replication of the ledger. Generally speaking ledger blocks distributed across the network by using gossip algorithm. In order to prevent all peers connecting to the ordering service there is a notion of leader election, e.g. for each organization there is one peer selected to open connection to the ordering service and start pulling blocks and forward them to the gossip layer to make them distributed between peers in the network. Before start leader check what is the ledger height and ask to deliver new blocks starting from it, all peers monitor availability of the leader and initiate new leader election if needed.
Now, there is additional background process which collects state messages from the peers and if it discovers there is a gap between local ledger height and height of others peers in the network it will initiate replication of missing blocks, via "state transfer" process as was mentioned by #Dave.
Hence, to conclude if peers comes online and for some reason elected to be a leader it will replicate ledger blocks directly from ordering service, otherwise he will get either via gossip layer or state transfer in case of significant gap in ledger height.
You can refer to this image for a brief overview