I executed Hyperledger Fabric test network and it had no errors. When I benchmarked its performance with Hyperledger Caliper tool, it was successful. But when the transactions count is increased to 800 or 1000 or above it throws the below error,
2023.02.13-11:49:56.059 error [caliper] [connectors/peer-gateway/PeerGateway] Failed to perform submit transaction [Admin_createPatient] using arguments [Client3_CAR557,Toyota,S,brown,Shotaro,29,123456890], with error: EndorseError: 2 UNKNOWN: too many requests for /gateway.Gateway, exceeding concurrency limit (500)
To rectify the error I changed the value of gatewayService to 2500 from its default value 500, but still the error not rectified.
limits:
concurrency:
# endorserService limits concurrent requests to endorser service that handles chaincode deployment, query and invocation,
# including both user chaincodes and system chaincodes.
endorserService: 2500
# deliverService limits concurrent event listeners registered to deliver service for blocks and transaction events.
deliverService: 2500
# gatewayService limits concurrent requests to gateway service that handles the submission and evaluation of transactions.
gatewayService: 2500
In the error message I assume that it was updated to say 2500 ? If not then the change wasn't picked up. If it was picked up then you will need to increase the value further. The number corresponds to in flight transactions, so you can only have that many transactions submitted and not committed being processed. This should track with the unfinished transaction count that caliper outputs during a benchmark run.
Related
Running the Fabric test network with a very basic asset based chaincode.
Using the Java SDK to submit batches of assets. All works fine until I've submitted about 75,000 transactions. At this point, the following exception starts to occur every insert. It does however appear to successfully add assets.
org.hyperledger.fabric.gateway.ContractException: Commit strategy failed
Looking at the ContractException, the proposalResponse for each peer appears to have the ChaincodeResponseStatus set to SUCCESS, and indeed the hashes of the payload responses match. There seems to be nothing within the exception to indicate something has gone wrong (apart from the presence of the exception itself).
A restart of the Java app using the SDK seems to resolve this until 75k or so inserts. No errors appear in the logs of the peers (or indeed the chaincode containers). If I add debug output to the CC, no adverse output is generated.
The steps in the submit flow at the client end are:
Send transaction proposal to peers for endorsement.
Send endorsed transaction to the orderer.
Listen for block events from peers, and look through each block for the submitted transaction ID.
The way the block events are handled depends on which commit strategy is selected, but the default behaviour is:
If an unsuccessful validation code is observed from any peer, the transaction failed validation at the peer, the ledger was not updated, and you get an exception to indicate that.
If a timeout period is reached before a result (either successful validation or connection failure) is received for all org peers, a TimeoutException occurs.
Once a successful transaction validation code has been received or a connection failure has occurred for all org peers:
If at least one successful validation code is observed, you're good.
If the block eventing connection to all peers fails without the transaction being observed, you get the "Commit strategy failed" message.
So this message is received after the transaction has been successfully endorsed and sent to the orderer, but then the block eventing connection to all peers failed and the client does not know whether or not the transaction was successful. It might have committed successfully. It might have failed. It might not have been committed at all yet.
The block eventing connections should retry and reestablish connectivity, but I have seen some cases where something in the network path is black-holing traffic for some reason. Since it seems to happen consistently for you after a certain number of transactions, it's just possible there is some issue in the client that is causing a failure over time, like a memory leak. It's worth doing some investigation.
In general, if you do have a transaction in this indeterminate state, a good approach can be to resubmit the transaction with the same transaction ID to the orderer again until you can observe the commit status and know for sure whether the transaction was successful or failed. Using the same transaction ID is important as it avoids the possibility of double-spend since Fabric will reject a transaction at validation time if that transaction ID has previously been committed, either successfully or unsuccessfully.
I have a single peer v1.4 Fabric setup. I'm doing 5 parallel transactions on my chaincode with a simple for loop. My input data size is around 1 MB for every transation. Only one of the transactions is getting succeeded, all others are failing with the error MVCC_READ_CONFLICT which is an expected error.
The issue however is, the peer container memory usage goes extremely high ~3 GB during these transactions, and even when all of them return the response, the memory usage of the peer container remains the same. When I restart the peer, only then does the memory usage resets to its normal value i.e. ~200 MB.
Could anyone please help me with this issue? I ran the peer container in debug mode but couldn't find anything concrete. I don't even know where to start from.
Let me know if you want to check the logs or any configuration.
I' m using Hyperledger Fabric 1.4.3 with the following conditions.
Ubuntu 18.04
CouchDB
Raft
TLS enabled
Using discovery service
Endorsement policy: "AND ('Org0MSP.peer')"
When I send 100 transactions asynchronously from Node.js SDK, all transactions are processed normally.
But, I send 2000 transactions asynchronously, the following error occurs.
[Node.js SDK log]
[DiscoveyEndorsementHandler]: _build_endorse_group_member >> G0:0 - endorsement failed - Error: Failed to connect before the deadline URL::grpcs//peer0.org0:7051
[Peer log]
[core.com] SerdrHandshake -> ERRO TLS handshake failed with error read tcp {org0 peer ip address:port} -> {Node.js SDK server ip address:port}: i/o timeout server=PeerServer remoteaddress={Node.js SDK server ip address:port}
So, some transactions failed.
Why does this error occur? Is there any way to solve this error?
When you say 'asynchronously' I assume you mean 'concurrently'. The error being returned is an i/o timeout, so this would indicate to me that most likely, your server hardware is not fast enough to handle the volume of concurrent requests you are attempting.
If you are concerned about denial of service, you can employ standard techniques for limiting the number of concurrent connections to your peer such as via network proxies. If you are simply trying to test the throughput bounds of Fabric, I would try slowly ramping up your transaction rate until you begin to see errors such as this timeout.
I successfully deployed my network file(.bna). Then I started a REST API using command composer-rest-server. I submit a single transaction using my front end Laravel application. When I try using for loop for submitting multiple transactions, I get an error in some time that MVCC_READ_CONFLICT. I decrease my network's bachtimeout. But the error continues. Please answer anyone if you have any idea about this issue.
Fabric vertion: 1.1.0
Composer : .19.16
Node :8.12
OS: Ubuntu 16.04
Well, MVCC_READ_CONFLICT means you are doing concurrent modification for some key in two different transactions, hence after transaction being ordered into block, whatever transaction gets in first committed while second one or subsequent transaction which works on same key marked invalid with MVCC_READ_CONFLICT.
To understand better the reason behind this status it's probably worth noting the transaction flow in fabric:
Client submit transaction proposal for endorsement sending it to endorsing peers
Endorsing peers executes simulation of chaincode where execution results are captured into Read-Write Set
Client collects endorsements and composes transaction, submitting it for ordering
Ordering service batches transactions into block employing total order of transactions
Block distributed between peers
Peer conducts validation to attest conformance with endorsement policy for each transaction
After that there is multi value concurrency control (MVCC), which checks for concurrent modifications, in fact validating keys version of RWSet and if concurrent modification detected tx invalidated with status MVCC_READ_CONFLICT
You can find more details in documentation "Transaction Flow".
lower the latency of the block creation so that blocks will be created more frequently and thus peers would be updated faster, for example, max_message_count=1 .but that may lead to some performance issue
On Hyperledger Fabric v1, I am trying to throw multiple requests to the same channel at the same time.
But the following error came out.
Error: EventHub has been shutdown
Is there a limit on the number of simultaneous execution of chaincodes?
If there is a limit, is there a parameter to increase it?
Kindly let me know why this happens.
To give more clarity on this, The event hub stops after a given period of time and the endorsement takes more time for processing and once the endorsements completed, it'll throw an event. But before that the eventhub will timeout and wont be able to catch it. The best way to fix it is by increasing the timeout time in your connection profile