Getting "Rejecting deliver request for IP:port because of consenter" error with raft - hyperledger-fabric

Question

I'm running a multi-org setup on cloud consists of 2 Orgs, 4 Peers(2 peers per Org) and 3 ordering nodes. All the peer nodes and the node orderer0 are running on DigitalOcean droplet. orderer2 and orderer3 nodes are running on AWS and GCP respectively. For ordering service I'm using Raft and orderer2 was selected as a Leader. For creating the channel, installing/instantiating/querying the chaincode, I'm executing a scripts.sh(present in byfn under scripts directory) on the peer. The script executed successfully i.e channel was created(using orderer0 node), joined by all the peers and installation/instantiation/query of chaincode executed successfully. But when I checked the orderer0 logs I found below errors.
2019-11-15 13:33:08.814 UTC [common.deliver] deliverBlocks -> WARN 04a [channel: mychannel] Rejecting deliver request for 139.59.7.201:59304 because of consenter error
2019-11-15 13:33:08.815 UTC [comm.grpc.server] 1 -> INFO 04b streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=139.59.7.201:59304 grpc.code=OK grpc.call_duration=201.373401ms
After a few seconds
2019-11-15 13:33:09.654 UTC [orderer.consensus.etcdraft] run -> INFO 058 raft.node: 1 elected leader 2 at term 2 channel=mychannel node=1
2019-11-15 13:33:09.657 UTC [orderer.consensus.etcdraft] serveRequest -> INFO 059 Raft leader changed: 0 -> 2 channel=mychannel node=1
2019-11-15 13:33:09.865 UTC [common.deliver] Handle -> WARN 05a Error reading from 139.59.7.201:59314: rpc error: code = Canceled desc = context canceled
docker-compose-orderer.yaml
version: '2'
networks:
byfn:
services:
orderer.example.com:
container_name: orderer.example.com
image: hyperledger/fabric-orderer:1.4.3
restart: always
environment:
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_byfn
- ORDERER_HOST=orderer.example.com
- ORDERER_GENERAL_LOGLEVEL=info
- FABRIC_LOGGING_SPEC=debug
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=7050
- ORDERER_GENERAL_GENESISMETHOD=file
- ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
- ORDERER_GENERAL_GENESISPROFILE=OrdererOrg
- CONFIGTX_ORDERER_ADDRESSES=[127.0.0.1:7050]
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
#- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
#- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
- CORE_CHAINCODE_LOGGING_SHIM=DEBUG
- ORDERER_TLS_CLIENTROOTCAS_FILES=/var/hyperledger/users/Admin#example.com/tls/ca.crt
- ORDERER_TLS_CLIENTCERT_FILE=/var/hyperledger/users/Admin#example.com/tls/client.crt
- ORDERER_TLS_CLIENTKEY_FILE=/var/hyperledger/users/Admin#example.com/tls/client.key
- GODEBUG=netdns=go
extra_hosts:
- "peer0.org1.example.com:139.59.13.3"
- "peer1.org1.example.com:139.59.13.119"
- "peer0.org2.example.com:139.59.7.201"
- "peer1.org2.example.com:139.59.24.225"
- "orderer2.example.com:3.14.67.48"
- "orderer3.example.com:34.69.118.13"
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ./channel-artifacts/:/var/hyperledger/configs
- ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp:/var/hyperledger/orderer/msp
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/:/var/hyperledger/orderer/tls
- ./crypto-config/ordererOrganizations/example.com/users:/var/hyperledger/users
#- orderer.example.com:/var/hyperledger/production/orderer
ports:
- 7050:7050
networks:
- byfn
docker-compose-orderer2.yaml
version: '2'
networks:
byfn:
services:
orderer2.example.com:
container_name: orderer2.example.com
image: hyperledger/fabric-orderer:1.4.3
restart: always
environment:
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_byfn
- ORDERER_HOST=orderer2.example.com
- ORDERER_GENERAL_LOGLEVEL=info
- FABRIC_LOGGING_SPEC=debug
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=7050
- ORDERER_GENERAL_GENESISMETHOD=file
- ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
- ORDERER_GENERAL_GENESISPROFILE=OrdererOrg
- CONFIGTX_ORDERER_ADDRESSES=[127.0.0.1:7050]
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
#- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
#- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
- CORE_CHAINCODE_LOGGING_SHIM=DEBUG
- ORDERER_TLS_CLIENTROOTCAS_FILES=/var/hyperledger/users/Admin#example.com/tls/ca.crt
- ORDERER_TLS_CLIENTCERT_FILE=/var/hyperledger/users/Admin#example.com/tls/client.crt
- ORDERER_TLS_CLIENTKEY_FILE=/var/hyperledger/users/Admin#example.com/tls/client.key
- GODEBUG=netdns=go
extra_hosts:
- "peer0.org1.example.com:139.59.13.3"
- "peer1.org1.example.com:139.59.13.119"
- "peer0.org2.example.com:139.59.7.201"
- "peer1.org2.example.com:139.59.24.225"
- "orderer.example.com:139.59.1.164"
- "orderer2.example.com:3.14.67.48"
- "orderer3.example.com:34.69.118.13"
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ./channel-artifacts/:/var/hyperledger/configs
- ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/msp:/var/hyperledger/orderer/msp
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/:/var/hyperledger/orderer/tls
- ./crypto-config/ordererOrganizations/example.com/users:/var/hyperledger/users
#- orderer.example.com:/var/hyperledger/production/orderer
ports:
- 7050:7050
networks:
- byfn
docker-compose-orderer3.yaml
version: '2'
networks:
byfn:
services:
orderer3.example.com:
container_name: orderer3.example.com
image: hyperledger/fabric-orderer:1.4.3
restart: always
environment:
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_byfn
- ORDERER_HOST=orderer3.example.com
- ORDERER_GENERAL_LOGLEVEL=info
- FABRIC_LOGGING_SPEC=debug
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=7050
- ORDERER_GENERAL_GENESISMETHOD=file
- ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
- ORDERER_GENERAL_GENESISPROFILE=OrdererOrg
- CONFIGTX_ORDERER_ADDRESSES=[127.0.0.1:7050]
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
#- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
#- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
- CORE_CHAINCODE_LOGGING_SHIM=DEBUG
- ORDERER_TLS_CLIENTROOTCAS_FILES=/var/hyperledger/users/Admin#example.com/tls/ca.crt
- ORDERER_TLS_CLIENTCERT_FILE=/var/hyperledger/users/Admin#example.com/tls/client.crt
- ORDERER_TLS_CLIENTKEY_FILE=/var/hyperledger/users/Admin#example.com/tls/client.key
- GODEBUG=netdns=go
extra_hosts:
- "peer0.org1.example.com:139.59.13.3"
- "peer1.org1.example.com:139.59.13.119"
- "peer0.org2.example.com:139.59.7.201"
- "peer1.org2.example.com:139.59.24.225"
- "orderer.example.com:139.59.1.164"
- "orderer2.example.com:3.14.67.48"
- "orderer3.example.com:34.69.118.13"
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ./channel-artifacts/:/var/hyperledger/configs
- ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer3.example.com/msp:/var/hyperledger/orderer/msp
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer3.example.com/tls/:/var/hyperledger/orderer/tls
- ./crypto-config/ordererOrganizations/example.com/users:/var/hyperledger/users
#- orderer.example.com:/var/hyperledger/production/orderer
ports:
- 7050:7050
networks:
- byfn
docker-compose-orderer.yaml
version: '2'
networks:
byfn:
services:
orderer.example.com:
container_name: orderer.example.com
image: hyperledger/fabric-orderer:1.4.3
restart: always
environment:
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_byfn
- ORDERER_HOST=orderer.example.com
- ORDERER_GENERAL_LOGLEVEL=info
- FABRIC_LOGGING_SPEC=info
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=7050
- ORDERER_GENERAL_GENESISMETHOD=file
- ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
- ORDERER_GENERAL_GENESISPROFILE=OrdererOrg
- CONFIGTX_ORDERER_ADDRESSES=[127.0.0.1:7050]
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
#- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
#- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
- CORE_CHAINCODE_LOGGING_SHIM=DEBUG
- ORDERER_TLS_CLIENTROOTCAS_FILES=/var/hyperledger/users/Admin#example.com/tls/ca.crt
- ORDERER_TLS_CLIENTCERT_FILE=/var/hyperledger/users/Admin#example.com/tls/client.crt
- ORDERER_TLS_CLIENTKEY_FILE=/var/hyperledger/users/Admin#example.com/tls/client.key
- GODEBUG=netdns=go
extra_hosts:
- "peer0.org1.example.com:139.59.13.3"
- "peer1.org1.example.com:139.59.13.119"
- "peer0.org2.example.com:139.59.7.201"
- "peer1.org2.example.com:139.59.24.225"
- "orderer2.example.com:3.14.67.48"
- "orderer3.example.com:34.69.118.13"
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ./channel-artifacts/:/var/hyperledger/configs
- ./channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp:/var/hyperledger/orderer/msp
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/:/var/hyperledger/orderer/tls
- ./crypto-config/ordererOrganizations/example.com/users:/var/hyperledger/users
#- orderer.example.com:/var/hyperledger/production/orderer
ports:
- 7050:7050
networks:
- byfn
orderer(orderer0) node logs
https://justpaste.it/49a1n
Orderer2 node logs ae huge hence sharing the link
https://justpaste.it/6ro0v
orderer3 logs
https://justpaste.it/5e4j8
peer0org1 logs
https://justpaste.it/33rm5
peer1org1 logs
https://justpaste.it/1s2uz
peer0org2 logs
https://justpaste.it/6emlk
peer1org2 logs
https://justpaste.it/53fna