I' m using Hyperledger Fabric 1.4.3 with the following conditions.
Ubuntu 18.04
CouchDB
Raft
TLS enabled
Using discovery service
Endorsement policy: "AND ('Org0MSP.peer')"
When I send 100 transactions asynchronously from Node.js SDK, all transactions are processed normally.
But, I send 2000 transactions asynchronously, the following error occurs.
[Node.js SDK log]
[DiscoveyEndorsementHandler]: _build_endorse_group_member >> G0:0 - endorsement failed - Error: Failed to connect before the deadline URL::grpcs//peer0.org0:7051
[Peer log]
[core.com] SerdrHandshake -> ERRO TLS handshake failed with error read tcp {org0 peer ip address:port} -> {Node.js SDK server ip address:port}: i/o timeout server=PeerServer remoteaddress={Node.js SDK server ip address:port}
So, some transactions failed.
Why does this error occur? Is there any way to solve this error?
When you say 'asynchronously' I assume you mean 'concurrently'. The error being returned is an i/o timeout, so this would indicate to me that most likely, your server hardware is not fast enough to handle the volume of concurrent requests you are attempting.
If you are concerned about denial of service, you can employ standard techniques for limiting the number of concurrent connections to your peer such as via network proxies. If you are simply trying to test the throughput bounds of Fabric, I would try slowly ramping up your transaction rate until you begin to see errors such as this timeout.
Related
Running the Fabric test network with a very basic asset based chaincode.
Using the Java SDK to submit batches of assets. All works fine until I've submitted about 75,000 transactions. At this point, the following exception starts to occur every insert. It does however appear to successfully add assets.
org.hyperledger.fabric.gateway.ContractException: Commit strategy failed
Looking at the ContractException, the proposalResponse for each peer appears to have the ChaincodeResponseStatus set to SUCCESS, and indeed the hashes of the payload responses match. There seems to be nothing within the exception to indicate something has gone wrong (apart from the presence of the exception itself).
A restart of the Java app using the SDK seems to resolve this until 75k or so inserts. No errors appear in the logs of the peers (or indeed the chaincode containers). If I add debug output to the CC, no adverse output is generated.
The steps in the submit flow at the client end are:
Send transaction proposal to peers for endorsement.
Send endorsed transaction to the orderer.
Listen for block events from peers, and look through each block for the submitted transaction ID.
The way the block events are handled depends on which commit strategy is selected, but the default behaviour is:
If an unsuccessful validation code is observed from any peer, the transaction failed validation at the peer, the ledger was not updated, and you get an exception to indicate that.
If a timeout period is reached before a result (either successful validation or connection failure) is received for all org peers, a TimeoutException occurs.
Once a successful transaction validation code has been received or a connection failure has occurred for all org peers:
If at least one successful validation code is observed, you're good.
If the block eventing connection to all peers fails without the transaction being observed, you get the "Commit strategy failed" message.
So this message is received after the transaction has been successfully endorsed and sent to the orderer, but then the block eventing connection to all peers failed and the client does not know whether or not the transaction was successful. It might have committed successfully. It might have failed. It might not have been committed at all yet.
The block eventing connections should retry and reestablish connectivity, but I have seen some cases where something in the network path is black-holing traffic for some reason. Since it seems to happen consistently for you after a certain number of transactions, it's just possible there is some issue in the client that is causing a failure over time, like a memory leak. It's worth doing some investigation.
In general, if you do have a transaction in this indeterminate state, a good approach can be to resubmit the transaction with the same transaction ID to the orderer again until you can observe the commit status and know for sure whether the transaction was successful or failed. Using the same transaction ID is important as it avoids the possibility of double-spend since Fabric will reject a transaction at validation time if that transaction ID has previously been committed, either successfully or unsuccessfully.
I'm trying to setup hyperledger fabric 1.2 on different hosts.
But when creating channel via: ./peer.sh channel create -o orderer0.trade.com:7050 -c tradechannel -f ../tradechannel.tx --cafile tlsca.trade.com-cert.pem -t 150s.
I got this error in CLI: got unexpected status: SERVICE_UNAVAILABLE -- will not enqueue, consenter for this channel hasn't started yet
and here is log from orderer:
[channel: tradechannel] Rejecting broadcast of message from 192.168.167.149:60655 with SERVICE_UNAVAILABLE: rejected by Consenter: will not enqueue, consenter for this channel hasn't started yet.
Closing Broadcast stream
transport: http2Server.HandleStreams failed to read frame: read tcp 192.168.171.33:7050->192.168.167.149:60655: read: connection reset by peer
It seems I have problem with grpc but totally have no idea about this.
the CLI is in Macbook and Orderer runs in RedHat.
I also faced the same issue when I was spawning the hyperledger fabric on kubernetes using kafka as ordering service.
The error is coming because kafka need some to sync with zookeepers. So you can wait for some time after the blockchain network creation. Then create the channel. In my case I wait for 10 minutes and I am able to create a channel.
But when I am spawning the fabric network using docker and kafka as ordering service this error never occurs and in this case kafka synchronise with zookeepers very fast, I don't why no error "rejected by Consenter: will not enqueueenter image description here".
I successfully deployed my network file(.bna). Then I started a REST API using command composer-rest-server. I submit a single transaction using my front end Laravel application. When I try using for loop for submitting multiple transactions, I get an error in some time that MVCC_READ_CONFLICT. I decrease my network's bachtimeout. But the error continues. Please answer anyone if you have any idea about this issue.
Fabric vertion: 1.1.0
Composer : .19.16
Node :8.12
OS: Ubuntu 16.04
Well, MVCC_READ_CONFLICT means you are doing concurrent modification for some key in two different transactions, hence after transaction being ordered into block, whatever transaction gets in first committed while second one or subsequent transaction which works on same key marked invalid with MVCC_READ_CONFLICT.
To understand better the reason behind this status it's probably worth noting the transaction flow in fabric:
Client submit transaction proposal for endorsement sending it to endorsing peers
Endorsing peers executes simulation of chaincode where execution results are captured into Read-Write Set
Client collects endorsements and composes transaction, submitting it for ordering
Ordering service batches transactions into block employing total order of transactions
Block distributed between peers
Peer conducts validation to attest conformance with endorsement policy for each transaction
After that there is multi value concurrency control (MVCC), which checks for concurrent modifications, in fact validating keys version of RWSet and if concurrent modification detected tx invalidated with status MVCC_READ_CONFLICT
You can find more details in documentation "Transaction Flow".
lower the latency of the block creation so that blocks will be created more frequently and thus peers would be updated faster, for example, max_message_count=1 .but that may lead to some performance issue
I am trying to close the established connections when using the Hyperledger Fabric NodeJS SDK. Just running netstat -tlpan | grep 'ESTABLISHED.*[PID]' where [PID] is the node process ID shows the number of established connections to peers and the event hub increasing continuously even after close is called on the channel.
This eventually leads to the OS hitting the max number of file descriptors. It is a network leak. Anyone with experience in cleaning this up? I think this is related to the GRPC stream.
The close channel and close peer, orderer does not kill the connections.
Here are the things I've tried:
Getting the peer objects and deleting them
Getting the stream to the server, calling end, cancel, deleting it, nullifying it
Setting Channel to null
Deleting the file descriptor from the shell -> Causes next transaction to fail
Getting the client duplex stream object for the Event Hub and setting allowHalfOpen to false.
All from the client side, none of these sorts it out. Calling channel.close() eventually limits the connections to 7050,7051 but not to 7053.
Any thoughts on this?
I have 2 different machines in cloud.
Containers on first machine:
orderer.mydomain.com
peer0.org1.mydomain.com
db.peer0.org1.mydomain.com
ca.org1.mydomain.com
Containers on second machine:
peer0.org2.mydomain.com
db.peer0.org2.mydomain.com
ca.org2.mydomain.com
I start them both. I can make them both join the same channel. I deploy a BNA exported from hyperledger composer to both peers. I send transactions to peer0.org1.mydomain.com and query and get same results from peer0.org2.mydomain.com.
Everything works perfectly so far.
However after 5 - 10 minutes peer on second machine (peer0.org2) gets disconnected from the orderer. When I send transactions to org1 I can query them from org1 and I see the results. But org2 gets detached. Doesn't accept new tx. (orderer connection gone) I can query org2 and see old results.
I added CORE_CHAINCODE_KEEPALIVE=30 to my peer env variables. I see keep alive actions in org2 peers logs. But didn't solve my problem.
I should note: Containers are in a docker network called "basic". This network was used in my local computer. However it still works in cloud.
In orderer logs:
Error sending to stream: rpc error: code = Internal desc = transport is closing
This happens every time I try. But when I run these containers in my local machine they keep connected without problems.
EDIT1: After checking the logs: peer0.org2 receives all tx and sends them to orderer. Orderer receives requests from peer but can't update peers. I can connect to both requestUrl or eventUrl on the problematic peer. There is no network problem.
I guess I found the problem. It is about MS Azure networking. After 4 minutes azure cuts idle connections:
https://discuss.pivotal.io/hc/en-us/articles/115005583008-Azure-Networking-Connection-idle-for-more-than-4-minutes
EDIT1:
Yes the problem was about MS Azure.. If there is anyone out there trying to run hyperledger on Azure keep in mind that if peer stays idle for more than 4 minutes azure times out tcp connections. You can configure it to timeout in 30 mins. It is not a bug but it was annoying for us not being able to understand why it wasn't working after 4 mins.
So you can use your own server or other cloud solution or use azure by adapting to their rules.