Hyperledger Fabric orderers throwing faulty node error status: NOT_FOUND - hyperledger-fabric

I have 5 ordering nodes (version 2.4) running on my cluster. I tried to rotate their tls certs and made some mistake. The network is a dev network and now all the ordering nodes are throwing below mentioned error:
2022-03-25 10:18:13.947 UTC 014c WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer1.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.947 UTC 014d WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer1.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.948 UTC 014e WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer4.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.948 UTC 014f WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer4.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.948 UTC 0150 WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer2.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.949 UTC 0151 WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer2.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.949 UTC 0152 WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer0.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.949 UTC 0153 WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer0.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.953 UTC 0154 WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer3.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.953 UTC 0155 WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer3.org1.com:7050 channel=assetschannel
Since it is a dev network, I can reset the entire network but I want to consider this scenario as if it occurred on a production network and because we can not reset a production network, I would like to get some suggestions on how we can recover the ordering nodes from this error.
Any suggestions would be appreciated

As per my knowledge, If we have backup of the ledger and WAL, we can restore them & bring up the network. Otherwise, we need to reset the network.

Related

Hyperledger Fabric ProcessProposal -> Failed to invoke chaincode channel= chaincode=_lifecycle

I am running Hyperledger Fabric and while trying to install the chaincode on peer0.org1, "peer lifecycle chaincode install basic.tar.gz", I run into some errors and can not install it -- Error: chaincode install failed with status: 500 - error in simulation: failed to execute transaction ee4a6ae0672861fa153e7fd5cd797e59b3eeebf1d2950ccd9e13e8231de484a9: error sending: timeout expired while executing transaction.
Below I will share the logs. Can anyone tell me what should I do? I have tried to reinstall it but with no success.
----logs---
2022-08-28 16:10:46.246 UTC 004a INFO [chaincode.externalbuilder.ccaas_builder] waitForExit -> ::Error: chaincode type not supported: node command=detect
2022-08-28 16:14:33.640 UTC 004b WARN [peer.blocksprovider] func1 -> Encountered an error reading from deliver stream: rpc error: code = Unavailable desc = transport is closing channel=mychannel orderer-address=orderer.example.com:7050
2022-08-28 16:14:33.640 UTC 004c WARN [peer.blocksprovider] DeliverBlocks -> Orderer hung up without sending status channel=mychannel orderer-address=orderer.example.com:7050
2022-08-28 16:15:46.750 UTC 004d INFO [endorser] callChaincode -> finished chaincode: _lifecycle duration: 300000ms channel= txID=ee4a6ae0
2022-08-28 16:15:46.778 UTC 004e ERRO [endorser] simulateProposal -> failed to invoke chaincode _lifecycle, error: timeout expired while executing transaction
github.com/hyperledger/fabric/core/chaincode.(*Handler).Execute
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571 channel= txID=ee4a6ae0
2022-08-28 16:15:46.778 UTC 004f WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel= chaincode=_lifecycle error="error in simulation: failed to execute transaction ee4a6ae0672861fa153e7fd5cd797e59b3eeebf1d2950ccd9e13e8231de484a9: error sending: timeout expired while executing transaction"
2022-08-28 16:15:46.778 UTC 0050 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.21.0.1:51150 grpc.code=OK grpc.call_duration=5m0.029919322s
2022-08-28 16:16:03.168 UTC 0051 INFO [lifecycle] InstallChaincode -> Successfully installed chaincode with package ID 'basic_1.0:9821071a953f452bda87fe89637d1e377dac825da4649ba998464a3c85d60c50'
2022-08-28 16:20:33.471 UTC 0052 WARN [peer.blocksprovider] func1 -> Encountered an error reading from deliver stream: rpc error: code = Unavailable desc = transport is closing channel=mychannel orderer-address=orderer.example.com:7050
2022-08-28 16:20:33.471 UTC 0053 WARN [peer.blocksprovider] DeliverBlocks -> Orderer hung up without sending status channel=mychannel orderer-address=orderer.example.com:7050

Hyperledger Fabric 2.4 approve chaincode error

I'm setting up an hyperledger fabric network with docker compose.
All the containers are up and running but i'm having problems with the appprovemyorg command.
These are the logs of the peer after joining the channel:
2022-05-24 07:41:54.316 UTC 0023 INFO [ledgermgmt] CreateLedger -> Creating ledger [mychannel] with genesis block
2022-05-24 07:41:54.319 UTC 0024 INFO [blkstorage] newBlockfileMgr -> Getting block information from block storage
2022-05-24 07:41:54.328 UTC 0025 INFO [kvledger] commit -> [mychannel] Committed block [0] with 1 transaction(s) in 6ms (state_validation=0ms block_and_pvtdata_commit=3ms state_commit=1ms) commitHash=[]
2022-05-24 07:41:54.328 UTC 0026 INFO [kvledger] updateLedgerStatus -> Updating ledger [mychannel] status to [ACTIVE]
2022-05-24 07:41:54.329 UTC 0027 INFO [ledgermgmt] CreateLedger -> Created ledger [mychannel] with genesis block
2022-05-24 07:41:54.332 UTC 0028 INFO [gossip.gossip] JoinChan -> Joining gossip network of channel mychannel with 1 organizations
2022-05-24 07:41:54.332 UTC 0029 INFO [gossip.gossip] learnAnchorPeers -> No configured anchor peers of org1MSP for channel mychannel to learn about
2022-05-24 07:41:54.333 UTC 002a INFO [gateway] configUpdate -> Updating orderer config channel=mychannel
2022-05-24 07:41:54.333 UTC 002b INFO [gossip.state] NewGossipStateProvider -> Updating metadata information for channel mychannel, current ledger sequence is at = 0, next expected block is = 1
2022-05-24 07:41:54.333 UTC 002c INFO [endorser] callChaincode -> finished chaincode: cscc duration: 21ms channel= txID=a704dc84
2022-05-24 07:41:54.333 UTC 002d INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.22.0.9:59204 grpc.code=OK grpc.call_duration=22.209824ms
2022-05-24 07:42:00.335 UTC 002e INFO [gossip.election] beLeader -> 49ad88f230e73d14bb6f9b8a891c0f5bc47a35cd793ff62222e489f762864a33 : Becoming a leader
2022-05-24 07:42:00.335 UTC 002f INFO [gossip.service] func1 -> Elected as a leader, starting delivery service for channel mychannel
2022-05-24 07:42:00.335 UTC 0030 INFO [deliveryClient] StartDeliverForChannel -> This peer will retrieve blocks from ordering service and disseminate to other peers in the organization for channel mychannel
2022-05-24 07:30:08.206 UTC 0031 WARN [peer.blocksprovider] DeliverBlocks -> Could not connect to ordering service: could not dial endpoint 'orderer2:8050': failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 0.0.0.0:8050: connect: connection refused" channel=mychannel
2022-05-24 07:30:08.307 UTC 0032 WARN [peer.blocksprovider] DeliverBlocks -> Could not connect to ordering service: could not dial endpoint 'orderer3:9050': failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 0.0.0.0:9050: connect: connection refused" channel=mychannel
2022-05-24 07:30:08.428 UTC 0033 WARN [peer.blocksprovider] DeliverBlocks -> Could not connect to ordering service: could not dial endpoint 'orderer1:7050': failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 0.0.0.0:7050: connect: connection refused" channel=mychannel
As you can see, i have some warnings but I can safely join the channel and I have no problems at all when i install the chaincode (the warnings disappear too).
These are the logs on the peer after installing the chaincode:
2022-05-24 07:45:17.017 UTC 0052 INFO [chaincode.externalbuilder.ccaas_builder] waitForExit -> ::Detect command=detect
2022-05-24 07:45:17.017 UTC 0053 INFO [chaincode.externalbuilder.ccaas_builder] waitForExit -> ::Error: chaincode type not supported: java command=detect
2022-05-24 07:45:17.023 UTC 0054 INFO [lifecycle] InstallChaincode -> Successfully installed chaincode with package ID 'testchaincode_1:de8cddcb2deda13a7a3c1e9f6e37764e6d2c1bb6f89bac1f0d2c6396587b35f7'
2022-05-24 07:45:17.023 UTC 0055 INFO [endorser] callChaincode -> finished chaincode: _lifecycle duration: 14ms channel= txID=694acf7b
2022-05-24 07:45:17.023 UTC 0056 INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.22.0.9:59206 grpc.code=OK grpc.call_duration=15.8101ms
When i try to approve the chaincode with this command:
peer lifecycle chaincode approveformyorg --tls --cafile /tmp/hyperledger/org1/peer1/tls-msp/tlscacerts/tls-tls-ca-7052.pem -o orderer1:7050 --ordererTLSHostnameOverride orderer1 --channelID mychannel --name mycc --version 1 --sequence 1 --waitForEvent --init-required --package-id testchaincode_1:de8cddcb2deda13a7a3c1e9f6e37764e6d2c1bb6f89bac1f0d2c6396587b35f7
I have this error:
Error: timed out waiting for txid on all peers
With these logs on the peer:
2022-05-24 07:46:54.303 UTC 0059 INFO [lifecycle] ApproveChaincodeDefinitionForOrg -> Successfully endorsed chaincode approval with name 'mycc', package ID 'testchaincode_1:de8cddcb2deda13a7a3c1e9f6e37764e6d2c1bb6f89bac1f0d2c6396587b35f7', on channel 'mychannel' with definition {sequence: 1, endorsement info: (version: '1', plugin: 'escc', init required: true), validation info: (plugin: 'vscc', policy: '12202f4368616e6e656c2f4170706c69636174696f6e2f456e646f7273656d656e74'), collections: ()}
2022-05-24 07:46:54.303 UTC 005a INFO [endorser] callChaincode -> finished chaincode: _lifecycle duration: 0ms channel=mychannel txID=d56b21b6
2022-05-24 07:46:54.308 UTC 005b INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.22.0.9:59210 grpc.code=OK grpc.call_duration=6.78249ms
2022-05-24 07:46:55.194 UTC 005c WARN [peer.blocksprovider] DeliverBlocks -> Could not connect to ordering service: could not dial endpoint 'orderer1:7050': failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 0.0.0.0:7050: connect: connection refused" channel=mychannel
2022-05-24 07:47:24.311 UTC 005d INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.request_deadline=2022-05-24T07:47:24.31Z grpc.peer_address=172.22.0.9:59212 error="context finished before block retrieved: context deadline exceeded" grpc.code=Unknown grpc.call_duration=30.0006887s
2022-05-24 07:47:54.260 UTC 005e WARN [peer.blocksprovider] DeliverBlocks -> Could not connect to ordering service: could not dial endpoint 'orderer2:8050': failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 0.0.0.0:8050: connect: connection refused" channel=mychannel
So i can't approve the chaincode and advance in the process.
Can anyone help me solve this problem?
Thanks

hyperledger fabric 2.2 ,Failed to add an orderer to an existing network

I'm having a strange problem with adding an orderer.Modify system-channel config,commit transaction successfully(called Mode 1).However, the warning log appears when the orderer is started.
The running orderer logs are as follows:
orderer1-Bank | 2021-01-25 09:22:07.942 UTC [common.deliver] deliverBlocks -> WARN 086 [channel: system-channel] Client 192.168.1.112:49642 is not authorized: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied: permission denied
The new orderer logs are as follows:
2021-01-25 09:22:07.935 UTC [orderer.common.cluster.replication] fetchLastBlockSeq -> WARN 00d Received status:FORBIDDEN from orderer1-ProCoop:7050: forbidden pulling the channel channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:07.935 UTC [orderer.common.cluster.replication] fetchLastBlockSeq -> WARN 00e Received status:FORBIDDEN from orderer1-Bank:7050: forbidden pulling the channel channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:07.935 UTC [orderer.common.cluster.replication] func1 -> WARN 00f Received error of type 'forbidden pulling the channel' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-Bank:7050"} channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:07.936 UTC [orderer.common.cluster.replication] func1 -> WARN 010 Received error of type 'forbidden pulling the channel' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-ProCoop:7050"} channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:14.924 UTC [orderer.common.cluster.replication] probeEndpoint -> WARN 011 Failed connecting to {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-CBRC:7050"}: failed to create new connection: context deadline exceeded channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:14.924 UTC [orderer.common.cluster.replication] func1 -> WARN 012 Received error of type 'failed to create new connection: context deadline exceeded' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-CBRC:7050"} channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:14.925 UTC [orderer.common.cluster.replication] connectToSomeEndpoint -> WARN 013 Could not connect to any endpoint of [{"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-CBRC:7050"} {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-ProCoop:7050"} {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-Bank:7050"}] channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:19.938 UTC [orderer.common.cluster.replication] fetchLastBlockSeq -> WARN 014 Received status:FORBIDDEN from orderer1-Bank:7050: forbidden pulling the channel channel=system-channel
orderer1-CBRC | 2021-01-25 09:22:19.939 UTC [orderer.common.cluster.replication] func1 -> WARN 015 Received error of type 'forbidden pulling the channel' from {"CAs":[{"Expired":false,"Issuer":"self","Subject":"CN=ca-tls,OU=Fabric,O=Hyperledger,ST=North Carolina,C=US"}],"Endpoint":"orderer1-Bank:7050"} channel=system-channel
If I set the same three orderers (same MSP files) as above when initializing the creation block(called Mode 2).the orderer will start normally.I have compared the most recent config block after the successful modification in mode 1 with the config block in mode 2, and they are identical except for the last_update property.
Adding an orderer takes a lot of steps. I have written one article on the same you can follow this guide
Edit1: steps you need to follow
Create crypto-material for the new ordered.
then you have to Add TLS to System Channel
Fetch the Latest Configuration Block
Bring the new orderer container up.
Adding new Orderers End Point to System Channel
Add TLS to Application Channel
Adding new Orderer Endpoint to Application Channel
That's all you have to do.

create mychannel Rejecting deliver request for because of consenter error

Error:
2020-10-23 09:36:01.590 UTC [common.deliver] deliverBlocks -> WARN 045 [channel: mychannel] Rejecting deliver request for ip:port because of consenter error

How to fix "context finished before block retrieved: context canceled" occurred while instantiating chaincode?

The instantiation command completes successfully, but when analyzing peer logs, you may notice this:
2019-04-17 17:25:52.581 UTC [gossip.state] commitBlock -> DEBU 48c [canal-contrato] Committed block [1] with 1 transaction(s)
2019-04-17 17:25:52.581 UTC [common.deliver] deliverBlocks -> DEBU 48d [channel: canal-contrato] Delivering block for (0xc00023f9c0) for 192.168.16.1:48230
2019-04-17 17:25:52.581 UTC [fsblkstorage] waitForBlock -> DEBU 48e Going to wait for newer blocks. maxAvailaBlockNumber=[1], waitForBlockNum=[2]
2019-04-17 17:25:52.586 UTC [common.deliver] deliverBlocks -> DEBU 48f Context canceled, aborting wait for next block
2019-04-17 17:25:52.586 UTC [common.deliverevents] func1 -> DEBU 490 Closing Deliver stream
2019-04-17 17:25:52.586 UTC [comm.grpc.server] 1 -> INFO 491 streaming call completed {"grpc.start_time": "2019-04-17T17:25:50.441Z", "grpc.service": "protos.Deliver", "grpc.method": "DeliverFiltered", "grpc.peer_address": "192.168.16.1:48230", "error": "context finished before block retrieved: context canceled", "grpc.code": "Unknown", "grpc.call_duration": "2.144399922s"}
Can anyone orient me what I'm possibly doing wrong and what are the consequences of this error?
Notes:
The orderer logs don't present any type of error
All containers are running correctly
I'm using node version 8.9.0 (with npm 5.5.1)
I have 1 organization with 1 peer, 1 CA and 1 ordered (just to test)
I'm using hyperlegder fabric version 1.4
This is not an error. You are using an SDK that connects to the peer and waits for the instantiate to finish. The block is received by the peer, and when it does - the SDK closes the gRPC stream because it doesn't need it anymore, and the peer logs this to notify you why it closed the stream from the server side.

Resources