Docker swarm deployment of Hyperledger fabric - hyperledger-fabric

Question

I have got an issue that occurs when i deploy my project using docker swarm, the creation of a new object in world state database happens fine but while updating that object sometimes a mvcc error occurs. When i checked the logs it shows that the database does not exist, I feel the issue might be because of peer communication and batch timeout (this is my guess). I am using 1.4.6 version of fabric. The same scenario, i had tried to recreate while deploying using docker compose but it doesn't
Configtx.yaml
Organizations:
- &OrdererOrg
Name: OrdererMSP
ID: OrdererMSP
MSPDir: crypto-config/ordererOrganizations/xxxx.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Writers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Admins:
Type: Signature
Rule: "OR('OrdererMSP.admin')"
- &Org1
Name: xxxxMSP
ID: xxxxMSP
MSPDir: crypto-config/peerOrganizations/xxxx.xxxx.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Writers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Admins:
Type: Signature
Rule: "OR('xxxxMSP.admin')"
AnchorPeers:
- Host: peer0.xxxx.xxxx.com
Port: 7051
- &Org2
Name: xxxxMSP
ID: xxxxMSP
MSPDir: crypto-config/peerOrganizations/xxxx.xxxx.com/msp
Policies:
Readers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Writers:
Type: Signature
Rule: "OR('xxxxMSP.admin', 'xxxxMSP.member')"
Admins:
Type: Signature
Rule: "OR('xxxxMSP.admin')"
AnchorPeers:
- Host: peer0.xxxx.xxxx.com
Port: 9051
Capabilities:
Channel: &ChannelCapabilities
V1_4_3: true
V1_3: false
V1_1: false
Orderer: &OrdererCapabilities
V1_4_2: true
V1_1: false
Application: &ApplicationCapabilities
V1_4_2: true
V1_3: false
V1_2: false
V1_1: false
Application: &ApplicationDefaults
Organizations:
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
Capabilities:
<<: *ApplicationCapabilities
Orderer: &OrdererDefaults
OrdererType: etcdraft
Addresses:
- orderer0.xxxx.com:7050
- orderer1.xxxx.com:7050
- orderer2.xxxx.com:7050
BatchTimeout: 2s
BatchSize:
MaxMessageCount: 10
AbsoluteMaxBytes: 99 MB
PreferredMaxBytes: 512 KB
EtcdRaft:
Consenters:
- Host: orderer0.xxxx.com
Port: 7050
ClientTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer0.xxxx.com/tls/server.crt
ServerTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer0.xxxx.com/tls/server.crt
- Host: orderer1.xxxx.com
Port: 7050
ClientTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer1.xxxx.com/tls/server.crt
ServerTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer1.xxxx.com/tls/server.crt
- Host: orderer2.xxxx.com
Port: 7050
ClientTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer2.xxxx.com/tls/server.crt
ServerTLSCert: crypto-config/ordererOrganizations/xxxx.com/orderers/orderer2.xxxx.com/tls/server.crt
Organizations:
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
BlockValidation:
Type: ImplicitMeta
Rule: "ANY Writers"
Channel: &ChannelDefaults
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
Capabilities:
<<: *ChannelCapabilities
Profiles:
xxxxGenesis:
<<: *ChannelDefaults
Orderer:
<<: *OrdererDefaults
Organizations:
- *OrdererOrg
Capabilities:
<<: *OrdererCapabilities
Consortiums:
SampleConsortium:
Organizations:
- *Org1
- *Org2
CV_Channel:
Consortium: SampleConsortium
<<: *ChannelDefaults
Application:
<<: *ApplicationDefaults
Organizations:
- *Org1
- *Org2
Capabilities:
<<: *ApplicationCapabilities
Peer container logs
{"log":"\u001b[34m2020-09-15 08:03:37.496 UTC [couchdb] CreateDatabaseIfNotExist -\u003e INFO 02b\u001b[0m Created state database cvchannel_\n","stream":"stderr","time":"2020-09-15T08:03:37.497361704Z"}
{"log":"\u001b[31m2020-09-15 08:03:37.674 UTC [couchdb] runWarmIndexAllIndexes -\u003e ERRO 02c\u001b[0m Error detected during WarmIndexAllIndexes(): error handling CouchDB request. Error:not_found, Status Code:404, Reason:Database does not exist.\n","stream":"stderr","time":"2020-09-15T08:03:37.674619911Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchInstance).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:37.674649981Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1847\n","stream":"stderr","time":"2020-09-15T08:03:37.674656059Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:37.674661185Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1650\n","stream":"stderr","time":"2020-09-15T08:03:37.674666013Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).ListIndex\n","stream":"stderr","time":"2020-09-15T08:03:37.674670852Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1151\n","stream":"stderr","time":"2020-09-15T08:03:37.674675596Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).WarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:37.674680273Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1324\n","stream":"stderr","time":"2020-09-15T08:03:37.674684939Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).runWarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:37.674689595Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1311\n","stream":"stderr","time":"2020-09-15T08:03:37.674708559Z"}
{"log":"runtime.goexit\n","stream":"stderr","time":"2020-09-15T08:03:37.674713109Z"}
{"log":"\u0009/opt/go/src/runtime/asm_amd64.s:1337\n","stream":"stderr","time":"2020-09-15T08:03:37.674717108Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.692 UTC [kvledger] CommitWithPvtData -\u003e INFO 02d\u001b[0m [cvchannel] Committed block [0] with 1 transaction(s) in 172ms (state_validation=1ms block_and_pvtdata_commit=64ms state_commit=94ms) commitHash=[]\n","stream":"stderr","time":"2020-09-15T08:03:37.693007188Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.695 UTC [ledgermgmt] CreateLedger -\u003e INFO 02e\u001b[0m Created ledger [cvchannel] with genesis block\n","stream":"stderr","time":"2020-09-15T08:03:37.695884567Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.712 UTC [gossip.gossip] JoinChan -\u003e INFO 02f\u001b[0m Joining gossip network of channel cvchannel with 2 organizations\n","stream":"stderr","time":"2020-09-15T08:03:37.712518745Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.712 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 030\u001b[0m No configured anchor peers of xxxxMSP for channel cvchannel to learn about\n","stream":"stderr","time":"2020-09-15T08:03:37.712607056Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.712 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 031\u001b[0m No configured anchor peers of xxxxMSP for channel cvchannel to learn about\n","stream":"stderr","time":"2020-09-15T08:03:37.712680067Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.740 UTC [gossip.state] NewGossipStateProvider -\u003e INFO 032\u001b[0m Updating metadata information for channel cvchannel, current ledger sequence is at = 0, next expected block is = 1\n","stream":"stderr","time":"2020-09-15T08:03:37.741210227Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.742 UTC [sccapi] deploySysCC -\u003e INFO 033\u001b[0m system chaincode lscc/cvchannel(github.com/hyperledger/fabric/core/scc/lscc) deployed\n","stream":"stderr","time":"2020-09-15T08:03:37.743086696Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [cscc] Init -\u003e INFO 034\u001b[0m Init CSCC\n","stream":"stderr","time":"2020-09-15T08:03:37.74339807Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [sccapi] deploySysCC -\u003e INFO 035\u001b[0m system chaincode cscc/cvchannel(github.com/hyperledger/fabric/core/scc/cscc) deployed\n","stream":"stderr","time":"2020-09-15T08:03:37.743424102Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [qscc] Init -\u003e INFO 036\u001b[0m Init QSCC\n","stream":"stderr","time":"2020-09-15T08:03:37.743544721Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [sccapi] deploySysCC -\u003e INFO 037\u001b[0m system chaincode qscc/cvchannel(github.com/hyperledger/fabric/core/scc/qscc) deployed\n","stream":"stderr","time":"2020-09-15T08:03:37.743558369Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [sccapi] deploySysCC -\u003e INFO 038\u001b[0m system chaincode (+lifecycle,github.com/hyperledger/fabric/core/chaincode/lifecycle) disabled\n","stream":"stderr","time":"2020-09-15T08:03:37.743790508Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [endorser] callChaincode -\u003e INFO 039\u001b[0m [][a3cc0bd2] Exit chaincode: name:"cscc" (431ms)\n","stream":"stderr","time":"2020-09-15T08:03:37.744038661Z"}
{"log":"\u001b[34m2020-09-15 08:03:37.743 UTC [comm.grpc.server] 1 -\u003e INFO 03a\u001b[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.0.4.4:33316 grpc.code=OK grpc.call_duration=435.114623ms\n","stream":"stderr","time":"2020-09-15T08:03:37.744207412Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.750 UTC [gossip.election] beLeader -\u003e INFO 03b\u001b[0m 5fc3148715eaea54bba5cae87f301cbeca436e2d8842d7e5b9469c5b8398d314 : Becoming a leader\n","stream":"stderr","time":"2020-09-15T08:03:43.751403096Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.751 UTC [gossip.service] func1 -\u003e INFO 03c\u001b[0m Elected as a leader, starting delivery service for channel cvchannel\n","stream":"stderr","time":"2020-09-15T08:03:43.751480343Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.752 UTC [deliveryClient] StartDeliverForChannel -\u003e INFO 03d\u001b[0m This peer will retrieve blocks from ordering service and disseminate to other peers in the organization for channel cvchannel\n","stream":"stderr","time":"2020-09-15T08:03:43.752972274Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.765 UTC [deliveryClient] RequestBlocks -\u003e INFO 03e\u001b[0m Starting deliver with block [1] for channel cvchannel\n","stream":"stderr","time":"2020-09-15T08:03:43.765848602Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.774 UTC [gossip.privdata] StoreBlock -\u003e INFO 03f\u001b[0m [cvchannel] Received block [1] from buffer\n","stream":"stderr","time":"2020-09-15T08:03:43.774744553Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] JoinChan -\u003e INFO 040\u001b[0m Joining gossip network of channel cvchannel with 2 organizations\n","stream":"stderr","time":"2020-09-15T08:03:43.782642453Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 041\u001b[0m No configured anchor peers of xxxxMSP for channel cvchannel to learn about\n","stream":"stderr","time":"2020-09-15T08:03:43.782658448Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 042\u001b[0m Learning about the configured anchor peers of xxxxMSP for channel cvchannel : [{peer0.xxxx.xxxx.com 7051}]\n","stream":"stderr","time":"2020-09-15T08:03:43.782662723Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.782 UTC [gossip.gossip] learnAnchorPeers -\u003e INFO 043\u001b[0m Anchor peer with same endpoint, skipping connecting to myself\n","stream":"stderr","time":"2020-09-15T08:03:43.782666538Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.783 UTC [committer.txvalidator] Validate -\u003e INFO 044\u001b[0m [cvchannel] Validated block [1] in 9ms\n","stream":"stderr","time":"2020-09-15T08:03:43.784169783Z"}
{"log":"\u001b[31m2020-09-15 08:03:43.845 UTC [couchdb] runWarmIndexAllIndexes -\u003e ERRO 045\u001b[0m Error detected during WarmIndexAllIndexes(): error handling CouchDB request. Error:not_found, Status Code:404, Reason:Database does not exist.\n","stream":"stderr","time":"2020-09-15T08:03:43.845975731Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchInstance).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:43.846007175Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1847\n","stream":"stderr","time":"2020-09-15T08:03:43.846012195Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).handleRequest\n","stream":"stderr","time":"2020-09-15T08:03:43.846016704Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1650\n","stream":"stderr","time":"2020-09-15T08:03:43.846020817Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).ListIndex\n","stream":"stderr","time":"2020-09-15T08:03:43.846024956Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1151\n","stream":"stderr","time":"2020-09-15T08:03:43.846028939Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).WarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:43.846033069Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1324\n","stream":"stderr","time":"2020-09-15T08:03:43.846037135Z"}
{"log":"github.com/hyperledger/fabric/core/ledger/util/couchdb.(*CouchDatabase).runWarmIndexAllIndexes\n","stream":"stderr","time":"2020-09-15T08:03:43.846041182Z"}
{"log":"\u0009/opt/gopath/src/github.com/hyperledger/fabric/core/ledger/util/couchdb/couchdb.go:1311\n","stream":"stderr","time":"2020-09-15T08:03:43.846045266Z"}
{"log":"runtime.goexit\n","stream":"stderr","time":"2020-09-15T08:03:43.846049541Z"}
{"log":"\u0009/opt/go/src/runtime/asm_amd64.s:1337\n","stream":"stderr","time":"2020-09-15T08:03:43.846053454Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.865 UTC [kvledger] CommitWithPvtData -\u003e INFO 046\u001b[0m [cvchannel] Committed block [1] with 1 transaction(s) in 81ms (state_validation=0ms block_and_pvtdata_commit=29ms state_commit=39ms) commitHash=[47dc540c94ceb704a23875c11273e16bb0b8a87aed84de911f2133568115f254]\n","stream":"stderr","time":"2020-09-15T08:03:43.865566497Z"}
{"log":"\u001b[34m2020-09-15 08:03:43.946 UTC [endorser] callChaincode -\u003e INFO 047\u001b[0m [][29451530] Entry chaincode: name:"lscc" \n","stream":"stderr","time":"2020-09-15T08:03:43.94617674Z"}
{"log":"\u001b[34m2020-09-15 08:03:44.017 UTC [couchdb] CreateDatabaseIfNotExist -\u003e INFO 048\u001b[0m Created state database cvchannel_lscc\n","stream":"stderr","time":"2020-09-15T08:03:44.018004477Z"}
{"log":"\u001b[34m2020-09-15 08:03:44.020 UTC [lscc] executeInstall -\u003e INFO 049\u001b[0m Installed Chaincode [certcc] Version [1.0] to peer\n","stream":"stderr","time":"2020-09-15T08:03:44.020291974Z"}
{"log":"\u001b[34m2020-09-15 08:03:44.020 UTC [endorser] callChaincode -\u003e INFO 04a\u001b[0m [][29451530] Exit chaincode: name:"lscc" (74ms)\n","stream":"stderr","time":"2020-09-15T08:03:44.020511347Z"}
{"log":"\u001b[34m2020-09-15 08:03:44.020 UTC [comm.grpc.server] 1 -\u003e INFO 04b\u001b[0m unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.0.4.4:33328 grpc.code=OK grpc.call_duration=75.164566ms\n","stream":"stderr","time":"2020-09-15T08:03:44.020765429Z"}
{"log":"\u001b[34m2020-09-15 08:04:00.231 UTC [comm.grpc.server] 1 -\u003e INFO 04c\u001b[0m unary call completed grpc.service=gossip.Gossip grpc.method=Ping grpc.request_deadline=2020-09-15T08:04:02.231Z grpc.peer_address=10.0.4.4:57698 grpc.peer_subject="CN=peer1.xxxx.xxxx.com,OU=peer+OU=xxxx,O=Hyperledger,ST=North Carolina,C=US" grpc.code=OK grpc.call_duration=184.503µs\n","stream":"stderr","time":"2020-09-15T08:04:00.231875809Z"}
{"log":"\u001b[34m2020-09-15 08:04:00.243 UTC [comm.grpc.server] 1 -\u003e INFO 04d\u001b[0m streaming call completed grpc.service=gossip.Gossip grpc.method=GossipStream grpc.request_deadline=2020-09-15T08:04:10.233Z grpc.peer_address=10.0.4.4:57698 grpc.peer_subject="CN=peer1.xxxx.xxxx.com,OU=peer+OU=xxxx,O=Hyperledger,ST=North Carolina,C=US" error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=9.822291ms\n","stream":"stderr","time":"2020-09-15T08:04:00.243456691Z"}