NB: I am seeking to understand how endorsements works in general. This will help me determine how to design applications when using Hyperledger Composer.
When I read the links here and here, I came across this statement: "Transactions have to be “endorsed” and only endorsed transactions may be committed and have an effect on the state". The statement is clear. However, let's consider the composer developer tutorial here. We have a commodity that is currently owned by an owner(Trader1) who could sell it to somebody else(Trader 2). Currently, how many endorsements are needed for the transaction to be put on the blockchain? Because, when running the application, I only submit a transaction Trade and I get results. I only deal with one function, and I get results. The following things are transparent to me as a programmer:
Creation of a transaction proposal,
When the transaction proposal is endorsed and by whom,
Whether an endorsement is performed explicitly by a human on the other end or it's programmatically done by code
That there is a proposal response from the endorser and how many they are,
When the application verifies the endorsing peer signatures
When the application creates a transaction message from the transaction proposal and response
etc.
All I do is submit one transaction and get a result.
So it becomes hard for me to assess the value of endorsement policies besides the theory in the text. And thus, the difficulty in designing a program to utilize the same. For example, consider two scenarios which we could use to handle a Trade:
We need 2 endorsements from the seller and the buyer before a transaction is commited. This would effectively be one transaction (This is what is transparent to me)
We need 2 authorizations from seller and the buyer before a transaction is commited. These authorizations could update states in the commodity such that we capture the approval from both the seller and the buyer. This could be 2 transactions i.e. sellerTradeRequest, buyerTradeApproval. The sellerTradeRequest could update commodity.sellerApproval=true while the buyerTradeApproval could update commodity.buyerApproval=true. Then, a final trade transaction that checks that the states on the commodity are OK i.e. commodity.sellerApproval=true and commodity.buyerApproval=true before commiting the transaction.
If I get a clear distinction between 1 and 2, especially how composer enables 1 above. Then maybe I will start understanding how to use endorsements.
Could anyone help?
The endorsement process is described in the docs. That said, in simple terms, the process of endorsement involves an endorsing peer signing the read/write set of a transaction proposal with its certificate. This basically says: the peer identified by the signing certificate asserts that these are the read/write sets of the proposed transaction simulation.
A client that invokes a proposed transaction will need knowledge of the endorsement policy for that channel/chaincode so that it can transmit the proposed transaction to each of the endorsing peers (or to a sufficient subset of endorsing peers to satisfy the policy). When it has received the responses from each endorsing peer, it will check that the transaction is valid and then broadcast the endorsed transaction to the ordering service, which will add it to a block and broadcast to the validating peers for that channel.
Validating peers will then validate the endorsement policy of the channel/chaincode against the transactions in a block to ensure that:
all endorsements are valid (i.e. they are valid signatures from valid certificates over the expected message)
there are an appropriate number of endorsements
endorsements come from the expected source(s)
If the endorsement policy is satisfied, then the transaction is committed and the world state updated with the read/write set.
At present, Composer cannot manage the endorsement policy, but this is definitely on the development roadmap.
The endorsement policy enables you the chance to add an extra verification layer to your Blockchain. You define it when you create the channel.
When a (Initial) Peer sends a proposal, firts of all the proposal is send to all of the Peers that you have defined in your Endorsement policy.
Then, each Endorser Peer executes the proposal against its ledger. The Endorser Peer will send the result signed to the Initial Peer.
The Initial Peer will receive more than one respone. Then, the Initial Peer will verify all the signatures and will compare the result of each response. The result sent by each Endorse Peer have to be the same.
In that case, the Initial Peer will send the transaction to the Orderers. That transaction will wraper all the responses all of the Endorses Peers.
So, you should define your Endorsement policy according to your requirements
Related
An endorsement is the process where endorsing peers execute a transaction and agree on the result. How is it different from the consensus in Fabric?
One key difference between Hyperledger Fabric and many other blockchain platforms is the lifecycle of a transaction.
In other platforms, the lifecycle of a transaction is usually Order-execute in which:
Order: transactions are added to the ledger in some order and disseminated to all peers.
Execute: transactions are sequentially executed (e.g. using smart contract code) on all peers.
While in Hyperledger Fabric, the lifecycle of a transaction is different as it is a Execute-order-validate model:
Execute: Transactions are executed (using chaincode) in any order, possibly even in parallel.
Order: When enough peers agree on the results of a transaction, it’s added to the ledger and disseminated to all peers. This step is where the transactions are first given an ordering — until transactions are added to the ledger.
Validate: Each peer validates and applies the ledger’s transactions in sequence. Now that the transactions have an ordering, the peers can check whether a later transaction was invalidated by an earlier transaction. For example, this prevents one item from being sold two times (called double-spending).
Endorsement in Hyperledger Fabric basically allows users to define policies around the execution of chaincode. These endorsement policies define which peers need to agree on the results of a transaction before it can be added to the ledger.
Now let's see how endorsement works:
Fabric starts with a transaction proposal. It’s a bundle of information used to trigger a specific chaincode. The transaction proposal is sent to some peers for endorsement. An endorsing peer executes the chaincode, which (if it succeeds) yields an actual transaction for the ledger. The endorsing peer then signs the transaction and returns it to the proposer. This is the Execute step in execute-order-validate.
Once the creator of the proposal receives enough signatures to satisfy the endorsement policy, it can submit the transaction (and the signatures) to be added to the ledger. This is the Order step.
Consensus or Validation is the last step in which all the peers verify if there are any changes in the world state in between this whole transaction process and then validates only those transactions that are valid and marks other transactions as invalid.
In case of Bitcoin (and Proof of Work in general) there is a simple rule that provides automatic conflict resolution when there are 2 differing copies of the ledger for any reason. And that rule is the longer chain wins.
But how are conflicts resolved in case of Fabric? Saying that conflicts never occur in Fabric is not an answer because if conflicts never occur, then what is the purpose of getting endorsement from more than one peer?
Stating alternatively, if your answer is that conflicts never occur in Fabric, then please explain why would someone want to get endorsement from more than one peer?
Yet another way to frame this question: let's say the copy of the ledger hosted on the peer of another organization got hacked unbeknownst to them. Now your organization and the other organization have different records. How will the conflict be reconciled? And don't forget the havoc this will cause - all transactions submitted by users will now fail to get endorsement until the conflict is resolved. A hack on another organization disrupted your business even though your ledger was not compromised.
In Proof of work:
Client will submit a transaction and it will be in pool, any miner can take and validate the transaction and then do mining, if he get solved quickly then he will publish to other miners. Here many nodes involved in Orderer to create a block
In Hyperledger Fabric [HighLevel]:
Client send transaction to endorsing peers(more than one), endorsing peers will sent back R/W and signatures to client, if endorsement failed means data in consistency then it will mark as failed and sent back to client.
Client will send entire payload to Orderer. Orderer simply creates a block and ship to committing peers, committing peers simply very endorsements (more than one ) and commit to the ledger
No tell me by comparing both, do u think still conflicts will come in hyperledger fabric?
Purpose of getting endorsement: In order to authentic more endorsements more accurate.
LowLevel:
Stage 1: [CLient Initiate Tx]
Client A is sending a request to update the ledger.
This request targets peerA and peerB, who are respectively
representative of Client A and Client B. The endorsement policy states
that both peers must endorse any transaction, therefore the request
goes to peerA and peerB.
Stage 2: Endorsing peers verify signature & execute the transaction
The endorsing peers verify (1) that the transaction proposal is well
formed, (2) it has not been submitted already in the past
(replay-attack protection), (3) the signature is valid (using the
MSP), and (4) that the submitter (Client A, in the example) is
properly authorized to perform the proposed operation on that channel
(namely, each endorsing peer ensures that the submitter satisfies the
channel’s Writers policy). The endorsing peers take the transaction
proposal inputs as arguments to the invoked chaincode’s function. The
chaincode is then executed against the current state database to
produce transaction results including a response value, read set, and
write set (i.e. key/value pairs representing an asset to create or
update). No updates are made to the ledger at this point. The set of
these values, along with the endorsing peer’s signature is passed back
as a “proposal response” to the SDK which parses the payload for the
application to consume.
Stage 3: Client Proposal responses are inspected
The application verifies the endorsing peer signatures and compares
the proposal responses to determine if the proposal responses are the
same. If the chaincode is only queried the ledger, the application
would inspect the query response and would typically not submit the
transaction to the ordering service. If the client application intends
to submit the transaction to the ordering service to update the
ledger, the application determines if the specified endorsement policy
has been fulfilled before submitting (i.e. did peerA and peerB both
endorse). The architecture is such that even if an application chooses
not to inspect responses or otherwise forwards an unendorsed
transaction, the endorsement policy will still be enforced by peers
and upheld at the commit validation phase.
Stage 4 : Client assembles endorsements into a transaction and broadcast
The application “broadcasts” the transaction proposal and response
within a “transaction message” to the ordering service. The
transaction will contain the read/write sets, the endorsing peers
signatures and the Channel ID. The ordering service does not need to
inspect the entire content of a transaction in order to perform its
operation, it simply receives transactions from all channels in the
network, orders them chronologically by channel, and creates blocks of
transactions per channel.
Stage 5: Transaction is validated and committed
The blocks of transactions are “delivered” to all peers on the
channel. The transactions within the block are validated to ensure
endorsement policy is fulfilled and to ensure that there have been no
changes to ledger state for read set variables since the read set was
generated by the transaction execution. Transactions in the block are
tagged as being valid or invalid.
Stage 6: Ledger updated
Each peer appends the block to the channel’s chain, and for each valid
transaction the write sets are committed to current state database. An
event is emitted, to notify the client application that the
transaction (invocation) has been immutably appended to the chain, as
well as notification of whether the transaction was validated or
invalidated.
I don't know some details about Fabric in this document.
Document: The application verifies the endorsing peer signatures and compares the proposal responses.
Is it necessary to have all the responses the same when verifying， or most of them？
If it is part of them, what is the ratio?
Whether it can be controlled by the Fabric SDK?
Document: The ordering service does not need to inspect the entire content of a transaction in order to perform its operation, it simply receives transactions from all channels in the network, orders them chronologically by channel, and creates blocks of transactions per channel.
Raft and Kafka are only used to guarantee availability, not for voting. Is this correct?
Document: Transaction is validated and committed.
When committing, is there a process of voting verification?
The application does not actually need to check anything (it's
optional but a good way to avoid sending invalid transactions).
When you instantiate chaincode on a channel, you set the endorsement policy for that chaincode. The endorsement policy specifies how many organization's peers must sign the response. This is actually enforced during validation on the peers (after the peer receives blocks/transaction from the orderer but prior to committing the data). If there are not enough signatures to satisfy the policy, transactions are marked invalid and state is not committed.
With Fabric v1.2 and later, clients can use the discovery service to obtain the minimum list of peers required to meet the endorsement policy for chaincode on each channel.
Kafka / Raft are used to ensure that all ordering service nodes process transactions and deliver blocks in the same order. With Fabric v1.4.1 you will have the option to use Raft consensus rather than Kafka.
See comment in 2. about enforcing endorsement policies during validation.
Can someone explain how the immutability is implemented in Hyperledger Fabric? If we have private channel with little amount of peers, how it can be guaranteed, that one side hasn't changed data in it's ledger?
In order to guarantee that no party in the channel has tampered data in its own favor you need to present sophisticated endorsement policy to include all required parties and make sure they adequately represented within endorsement policy. Hence making it obligatory for client which issues new transaction to get endorsement from all interested parties, hence ensuring that all have same consistent state. For example if you have two organizations Org1 and Org2 and they do not trust each other, you would like to create endorsement policy:
AND(Org1.memmber, Org2.member)
Therefore client will have to collect endorsements from peers of both organization to consider the transaction valid those endorsement have to sign same bytes, which won't be the case if data was forged. You can read more about endorsements in official documentation. There is also a recent publication of Fabric architecture which explains it in more details.
I am curious how does Fabric choose among one of selected Organizations in an "OR"-type Endorsement Policy. Is it a random choice or does it follow a predetermined logic?
For instance, let's say that I have a following policy :
OR('Org1.member', 'Org2.member', 'Org3.member')
Now, let's say that the Endorsing Peer which is supposed to process an incoming transaction proposal belongs to Org1.
Because of uncertainty about network connectivity and availability of other organizations, Org1 would be a preferred entity elected for endorsement (because it happens locally on that very same peer).
However, is this the case in Hyperledger Fabric?
Any help understanding the above will be greatly appreciated.
Fabric clients should be aware of the endorsement policies and it's up to them to decide on endorsing peers. So in your example with:
OR('Org1.member', 'Org2.member', 'Org3.member')
client should know that in order to get valid transaction it has to be endorsed by either someone from org1 or org2 or org3. So client could send transaction proposal to some peer into org1 and wait until get response. An alternative strategy would be to send transaction proposal to 3 peers one from each organization.
Once client collects enough endorsement it will submit transaction to the ordering service and prior to commit peer will ensure that endorsement policy being satisfied. Now please note that endorsement policy doesn't specifies exact endorsement peers, but just saying that it has to be someone from that org with certificate approved by org root CA.
So right now client has to know endorsement policies and being aware of the membership to being able sent transaction proposals, however there work in progress FAB-5451, to provide service discovery based capabilities so client will be able to dynamically learn policies and will be able to query for set of endorsing peers.
wouldn't it be better to step back from talking directly to peers and orderers? use channel instead. always speaking only to specific elements is SO fragile.