I'm running a Hyperledger Fabric 1.4.1 using etcd/raft and I'm running on this error on the orderers of the network whenever I send a transaction to the orderer.
We have observed this 12 times in a period of 9 days.
What might be the causes of it?
2019-08-07 17:06:07.241 UTC [orderer.consensus.etcdraft] 2 -> DEBU 7fc91 Proposed block [15] to raft consensus channel=pocccssinciensaled node=2
2019-08-07 17:06:07.242 UTC [orderer.consensus.etcdraft] run -> PANI 7fc92 Failed to persist etcd/raft data: input/output error channel=pocccssinciensaled node=2
panic: Failed to persist etcd/raft data: input/output error
goroutine 68 [running]:
github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000ede40, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x515
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc00000e258, 0x104, 0x1033fcb, 0x24, 0xc0002f9bc8, 0x1, 0x1, 0x0, 0x0, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(0xc00000e258, 0x1033fcb, 0x24, 0xc0002f9bc8, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159 +0x79
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(0xc00000e260, 0x1033fcb, 0x24, 0xc0002f9bc8, 0x1, 0x1)
/opt/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:74 +0x60
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*node).run(0xc000586500, 0x0)
/opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/node.go:118 +0x43c
created by github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*node).start
/opt/gopath/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/node.go:80 +0x218
Related
Help me please!
I try to create app channel with the next command:
./bin/peer channel create -o localhost:7050 -c mychannel -f ./channel-artifacts/mychannel.tx --outputBlock ./channel-artifacts/mychannel.block --tls --cafile "$CORE_PEER_TLS_ROOTCERT_FILE"
But I always receive this error:
2022-09-01 14:08:46.482 MSK 0001 INFO [channelCmd] InitCmdFactory -> Endorser and orderer connections initialized
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'mychannel': error authorizing update: error validating ReadSet: existing config does not contain element for [Value] /Channel/Consortium but was in the read set
I receive these logs in orderer container terminal in debug mode:
orderer-test.ru | 2022-09-01 11:49:05.803 UTC 0453 WARN [orderer.common.broadcast] ProcessMessage -> [channel: mychannel] Rejecting broadcast of config message from 192.168.32.1:39190 because of error: error applying config update to existing channel 'mychannel': error authorizing update: error validating ReadSet: existing config does not contain element for [Value] /Channel/Consortium but was in the read set
orderer-test.ru | 2022-09-01 11:49:05.803 UTC 0454 DEBU [orderer.common.server] func1 -> Closing Broadcast stream
orderer-test.ru | 2022-09-01 11:49:05.803 UTC 0455 INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Broadcast grpc.peer_address=192.168.32.1:39190 grpc.code=OK grpc.call_duration=1.267632ms
orderer-test.ru | 2022-09-01 11:49:05.807 UTC 0456 WARN [common.deliver] Handle -> Error reading from 192.168.32.1:39186: rpc error: code = Canceled desc = context canceled
orderer-test.ru | 2022-09-01 11:49:05.807 UTC 0457 DEBU [orderer.common.server] func1 -> Closing Deliver stream
orderer-test.ru | 2022-09-01 11:49:05.807 UTC 0458 INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=192.168.32.1:39186 error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=11.688059ms
orderer-test.ru | 2022-09-01 11:49:05.807 UTC [grpc] InfoDepth -> DEBU 003 [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"
orderer-test.ru | 2022-09-01 11:49:05.807 UTC [grpc] InfoDepth -> DEBU 004 [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"
When i am running the code peer channel fetch config config_block.pb -o orderer.example.com:7050 --ordererTLSHostnameOverride orderer.example.com --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -c mychannel
I am getting this
2021-07-26 12:42:59.856 UTC [cli.common] readBlock -> INFO 002 Expect
block, but got status: &{SERVICE_UNAVAILABLE}
I have started getting this error after I have added more orderers. I am using Raft for orderering service. And fabric-ca for generating certificates
configtx.yaml:
Orderer: &OrdererDefaults # Orderer Type: The orderer implementation to start
OrdererType: etcdraft
# Addresses used to be the list of orderer addresses that clients and peers
# could connect to. However, this does not allow clients to associate orderer
# addresses and orderer organizations which can be useful for things such
# as TLS validation. The preferred way to specify orderer addresses is now
# to include the OrdererEndpoints item in your org definition
Addresses:
- orderer.example.com:7050
- orderer2.example.com:8050
- orderer3.example.com:9050
EtcdRaft:
Consenters:
- Host: orderer.example.com
Port: 7050
ClientTLSCert: ../organizations/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt
ServerTLSCert: ../organizations/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt
- Host: orderer2.example.com
Port: 8050
ClientTLSCert: ../organizations/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
ServerTLSCert: ../organizations/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
- Host: orderer3.example.com
Port: 9050
ClientTLSCert: ../organizations/ordererOrganizations/example.com/orderers/orderer3.example.com/tls/server.crt
ServerTLSCert: ../organizations/ordererOrganizations/example.com/orderers/orderer3.example.com/tls/server.crt
# Options to be specified for all the etcd/raft nodes. The values here
# are the defaults for all new channels and can be modified on a
# per-channel basis via configuration updates.
Options:
# TickInterval is the time interval between two Node.Tick invocations.
TickInterval: 500ms
# ElectionTick is the number of Node.Tick invocations that must pass
# between elections. That is, if a follower does not receive any
# message from the leader of current term before ElectionTick has
# elapsed, it will become candidate and start an election.
# ElectionTick must be greater than HeartbeatTick.
ElectionTick: 10
# HeartbeatTick is the number of Node.Tick invocations that must
# pass between heartbeats. That is, a leader sends heartbeat
# messages to maintain its leadership every HeartbeatTick ticks.
HeartbeatTick: 1
# MaxInflightBlocks limits the max number of in-flight append messages
# during optimistic replication phase.
MaxInflightBlocks: 5
# SnapshotIntervalSize defines number of bytes per which a snapshot is taken
SnapshotIntervalSize: 16 MB
# Batch Timeout: The amount of time to wait before creating a batch
BatchTimeout: 2s
# Batch Size: Controls the number of messages batched into a block
BatchSize:
# Max Message Count: The maximum number of messages to permit in a batch
MaxMessageCount: 10
# Absolute Max Bytes: The absolute maximum number of bytes allowed for
# the serialized messages in a batch.
AbsoluteMaxBytes: 99 MB
# Preferred Max Bytes: The preferred maximum number of bytes allowed for
# the serialized messages in a batch. A message larger than the preferred
# max bytes will result in a batch larger than preferred max bytes.
PreferredMaxBytes: 512 KB
# Organizations is the list of orgs which are defined as participants on
# the orderer side of the network
Organizations:
# Policies defines the set of policies at this level of the config tree
# For Orderer policies, their canonical path is
# /Channel/Orderer/<PolicyName>
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
# BlockValidation specifies what signatures must be included in the block
# from the orderer for the peer to validate it.
BlockValidation:
Type: ImplicitMeta
Rule: "ANY Writers"
docker-compose-file:
services:
orderer.example.com:
container_name: orderer.example.com
image: hyperledger/fabric-orderer:latest
labels:
service: hyperledger-fabric
environment:
# - FABRIC_LOGGING_SPEC=INFO
- FABRIC_LOGGING_SPEC=DEBUG
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=7050
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_GENERAL_BOOTSTRAPMETHOD=none
- ORDERER_CHANNELPARTICIPATION_ENABLED=true
- ORDERER_ADMIN_TLS_ENABLED=true
- ORDERER_ADMIN_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_ADMIN_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_ADMIN_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_TLS_CLIENTROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_LISTENADDRESS=0.0.0.0:7053
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ../system-genesis-block/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ../organizations/ordererOrganizations/example.com/orderers/orderer.example.com/msp:/var/hyperledger/orderer/msp
- ../organizations/ordererOrganizations/example.com/orderers/orderer.example.com/tls/:/var/hyperledger/orderer/tls
- orderer.example.com:/var/hyperledger/production/orderer
ports:
- 7050:7050
- 7053:7053
networks:
- test
orderer2.example.com:
container_name: orderer2.example.com
image: hyperledger/fabric-orderer:latest
labels:
service: hyperledger-fabric
environment:
# - FABRIC_LOGGING_SPEC=INFO
- FABRIC_LOGGING_SPEC=DEBUG
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=8050
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_GENERAL_BOOTSTRAPMETHOD=none
- ORDERER_CHANNELPARTICIPATION_ENABLED=true
- ORDERER_ADMIN_TLS_ENABLED=true
- ORDERER_ADMIN_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_ADMIN_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_ADMIN_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_TLS_CLIENTROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_LISTENADDRESS=0.0.0.0:7053
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ../system-genesis-block/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ../organizations/ordererOrganizations/example.com/orderers/orderer2.example.com/msp:/var/hyperledger/orderer/msp
- ../organizations/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/:/var/hyperledger/orderer/tls
- orderer2.example.com:/var/hyperledger/production/orderer
ports:
- 8050:8050
- 8053:7053
networks:
- test
orderer3.example.com:
container_name: orderer3.example.com
image: hyperledger/fabric-orderer:latest
labels:
service: hyperledger-fabric
environment:
# - FABRIC_LOGGING_SPEC=INFO
- FABRIC_LOGGING_SPEC=DEBUG
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_LISTENPORT=9050
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_GENERAL_BOOTSTRAPMETHOD=none
- ORDERER_CHANNELPARTICIPATION_ENABLED=true
- ORDERER_ADMIN_TLS_ENABLED=true
- ORDERER_ADMIN_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_ADMIN_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_ADMIN_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_TLS_CLIENTROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_ADMIN_LISTENADDRESS=0.0.0.0:7053
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
volumes:
- ../system-genesis-block/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ../organizations/ordererOrganizations/example.com/orderers/orderer3.example.com/msp:/var/hyperledger/orderer/msp
- ../organizations/ordererOrganizations/example.com/orderers/orderer3.example.com/tls/:/var/hyperledger/orderer/tls
- orderer3.example.com:/var/hyperledger/production/orderer
ports:
- 9050:9050
- 9053:7053
networks:
- test
docker container log of orderer.example.com:
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] Step -> INFO 7be 1 is starting a new election at term 1 channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 7bf 1 became pre-candidate at term 1 channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] poll -> INFO 7c0 1 received MsgPreVoteResp from 1 at term 1 channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7c1 1 [logterm: 1, index: 3] sent MsgPreVote request to 2 at term 1 channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7c2 1 [logterm: 1, index: 3] sent MsgPreVote request to 3 at term 1 channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.common.cluster] NewStream -> DEBU 7c3 Created new stream to orderer2.example.com:8050 with ID of 26 and buffer size of 10
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7c4 Sending msg of 28 bytes to 2 on channel mychannel took 98.44µs
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] send -> INFO 7c5 Successfully sent StepRequest to 2 after failed attempt(s) channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.common.cluster] NewStream -> DEBU 7c6 Created new stream to orderer3.example.com:9050 with ID of 26 and buffer size of 10
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7c7 Sending msg of 28 bytes to 3 on channel mychannel took 71.392µs
2021-07-26 21:36:40.751 UTC [orderer.consensus.etcdraft] send -> INFO 7c8 Successfully sent StepRequest to 3 after failed attempt(s) channel=mychannel node=1
2021-07-26 21:36:40.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7c9 Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer3(orderer3.example.com:9050) took 44.832µs
2021-07-26 21:36:40.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7ca Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer2(orderer2.example.com:8050) took 20.721µs
2021-07-26 21:36:48.403 UTC [core.comm] ServerHandshake -> DEBU 7cb Server TLS handshake completed in 2.630606ms server=Orderer remoteaddress=
2021-07-26 21:36:48.403 UTC [orderer.common.server] Deliver -> DEBU 7cc Starting new Deliver handler
2021-07-26 21:36:48.403 UTC [common.deliver] Handle -> DEBU 7cd Starting new deliver loop for
2021-07-26 21:36:48.404 UTC [common.deliver] Handle -> DEBU 7ce Attempting to read seek info message from
2021-07-26 21:36:48.404 UTC [common.deliver] deliverBlocks -> WARN 7cf [channel: mychannel] Rejecting deliver request for because of consenter error
2021-07-26 21:36:48.404 UTC [orderer.common.server] func1 -> DEBU 7d0 Closing Deliver stream
2021-07-26 21:36:48.404 UTC [comm.grpc.server] 1 -> INFO 7d1 streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address= grpc.code=OK grpc.call_duration=675.251µs
2021-07-26 21:36:48.405 UTC [grpc] InfoDepth -> DEBU 7d2 [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2021-07-26 21:36:48.750 UTC [orderer.consensus.etcdraft] Step -> INFO 7d3 1 is starting a new election at term 1 channel=mychannel node=1
2021-07-26 21:36:48.750 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 7d4 1 became pre-candidate at term 1 channel=mychannel node=1
2021-07-26 21:36:48.750 UTC [orderer.consensus.etcdraft] poll -> INFO 7d5 1 received MsgPreVoteResp from 1 at term 1 channel=mychannel node=1
2021-07-26 21:36:48.750 UTC [orderer.consensus.etcdraft] campaign -> INFO 7d6 1 [logterm: 1, index: 3] sent MsgPreVote request to 2 at term 1 channel=mychannel node=1
2021-07-26 21:36:48.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7d7 1 [logterm: 1, index: 3] sent MsgPreVote request to 3 at term 1 channel=mychannel node=1
2021-07-26 21:36:48.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7d8 Sending msg of 28 bytes to 2 on channel mychannel took 4.077µs
2021-07-26 21:36:48.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7d9 Sending msg of 28 bytes to 3 on channel mychannel took 3.599µs
2021-07-26 21:36:48.751 UTC [orderer.common.cluster] 1 -> DEBU 7da Stream 26 to orderer3(orderer3.example.com:9050) is aborted
2021-07-26 21:36:48.751 UTC [orderer.common.cluster] 1 -> DEBU 7db Stream 26 to orderer2(orderer2.example.com:8050) is aborted
2021-07-26 21:36:48.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7dc Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer3(orderer3.example.com:9050) took 61.407µs but failed due to EOF
2021-07-26 21:36:48.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7dd Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer2(orderer2.example.com:8050) took 66.918µs but failed due to EOF
2021-07-26 21:36:48.751 UTC [orderer.common.cluster.step] serviceStream -> DEBU 7df Stream 26 to (orderer2.example.com:8050) terminated with total lifetime of 7.999445309s
2021-07-26 21:36:48.751 UTC [orderer.common.cluster.step] serviceStream -> DEBU 7de Stream 26 to (orderer3.example.com:9050) terminated with total lifetime of 7.999522414s
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] Step -> INFO 7e0 1 is starting a new election at term 1 channel=mychannel node=1
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 7e1 1 became pre-candidate at term 1 channel=mychannel node=1
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] poll -> INFO 7e2 1 received MsgPreVoteResp from 1 at term 1 channel=mychannel node=1
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7e3 1 [logterm: 1, index: 3] sent MsgPreVote request to 3 at term 1 channel=mychannel node=1
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7e4 1 [logterm: 1, index: 3] sent MsgPreVote request to 2 at term 1 channel=mychannel node=1
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7e5 Sending msg of 28 bytes to 3 on channel mychannel took 25.709µs
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] logSendFailure -> ERRO 7e6 Failed to send StepRequest to 3, because: EOF channel=mychannel node=1
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7e7 Sending msg of 28 bytes to 2 on channel mychannel took 5.017µs
2021-07-26 21:36:56.751 UTC [orderer.consensus.etcdraft] logSendFailure -> ERRO 7e8 Failed to send StepRequest to 2, because: EOF channel=mychannel node=1
2021-07-26 21:37:04.750 UTC [orderer.consensus.etcdraft] Step -> INFO 7e9 1 is starting a new election at term 1 channel=mychannel node=1
2021-07-26 21:37:04.750 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 7ea 1 became pre-candidate at term 1 channel=mychannel node=1
2021-07-26 21:37:04.751 UTC [orderer.consensus.etcdraft] poll -> INFO 7eb 1 received MsgPreVoteResp from 1 at term 1 channel=mychannel node=1
2021-07-26 21:37:04.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7ec 1 [logterm: 1, index: 3] sent MsgPreVote request to 2 at term 1 channel=mychannel node=1
2021-07-26 21:37:04.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7ed 1 [logterm: 1, index: 3] sent MsgPreVote request to 3 at term 1 channel=mychannel node=1
2021-07-26 21:37:04.751 UTC [orderer.common.cluster] NewStream -> DEBU 7ee Created new stream to orderer2.example.com:8050 with ID of 27 and buffer size of 10
2021-07-26 21:37:04.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7ef Sending msg of 28 bytes to 2 on channel mychannel took 271.523µs
2021-07-26 21:37:04.751 UTC [orderer.consensus.etcdraft] send -> INFO 7f0 Successfully sent StepRequest to 2 after failed attempt(s) channel=mychannel node=1
2021-07-26 21:37:04.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7f1 Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer2(orderer2.example.com:8050) took 130.526µs
2021-07-26 21:37:04.752 UTC [orderer.common.cluster] NewStream -> DEBU 7f2 Created new stream to orderer3.example.com:9050 with ID of 27 and buffer size of 10
2021-07-26 21:37:04.752 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7f3 Sending msg of 28 bytes to 3 on channel mychannel took 627.237µs
2021-07-26 21:37:04.752 UTC [orderer.consensus.etcdraft] send -> INFO 7f4 Successfully sent StepRequest to 3 after failed attempt(s) channel=mychannel node=1
2021-07-26 21:37:04.752 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7f5 Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer3(orderer3.example.com:9050) took 98.032µs
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] Step -> INFO 7f6 1 is starting a new election at term 1 channel=mychannel node=1
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 7f7 1 became pre-candidate at term 1 channel=mychannel node=1
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] poll -> INFO 7f8 1 received MsgPreVoteResp from 1 at term 1 channel=mychannel node=1
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7f9 1 [logterm: 1, index: 3] sent MsgPreVote request to 2 at term 1 channel=mychannel node=1
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] campaign -> INFO 7fa 1 [logterm: 1, index: 3] sent MsgPreVote request to 3 at term 1 channel=mychannel node=1
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7fb Sending msg of 28 bytes to 2 on channel mychannel took 4.668µs
2021-07-26 21:37:12.751 UTC [orderer.consensus.etcdraft] consensusSent -> DEBU 7fc Sending msg of 28 bytes to 3 on channel mychannel took 2.001µs
2021-07-26 21:37:12.751 UTC [orderer.common.cluster] 1 -> DEBU 7fd Stream 27 to orderer3(orderer3.example.com:9050) is aborted
2021-07-26 21:37:12.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 7fe Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer3(orderer3.example.com:9050) took 102.584µs but failed due to EOF
2021-07-26 21:37:12.751 UTC [orderer.common.cluster.step] serviceStream -> DEBU 7ff Stream 27 to (orderer3.example.com:9050) terminated with total lifetime of 7.999196657s
2021-07-26 21:37:12.751 UTC [orderer.common.cluster] 1 -> DEBU 800 Stream 27 to orderer2(orderer2.example.com:8050) is aborted
2021-07-26 21:37:12.751 UTC [orderer.common.cluster.step] sendMessage -> DEBU 801 Send of ConsensusRequest for channel mychannel with payload of size 28 to orderer2(orderer2.example.com:8050) took 24.421µs but failed due to EOF
2021-07-26 21:37:12.751 UTC [orderer.common.cluster.step] serviceStream -> DEBU 802 Stream 27 to (orderer2.example.com:8050) terminated with total lifetime of 7.999721572s
Actual error on the terminal:
Setting anchor peer for org1...
Using organization 1
Fetching channel config for channel mychannel
Using organization 1
Fetching the most recent configuration block for the channel
+ peer channel fetch config config_block.pb -o orderer.example.com:7050 --ordererTLSHostnameOverride orderer.example.com --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -c mychannel
2021-07-26 21:26:40.010 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2021-07-26 21:26:40.012 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{SERVICE_UNAVAILABLE}
Error: can't read the block: &{SERVICE_UNAVAILABLE}
Decoding config block to JSON and isolating config to Org1MSPconfig.json
+ configtxlator proto_decode --input config_block.pb --type common.Block
+ jq '.data.data[0].payload.data.config'
configtxlator: error: open config_block.pb: no such file or directory, try --help
+ jq '.channel_group.groups.Application.groups.Org1MSP.values += {"AnchorPeers":{"mod_policy": "Admins","value":{"anchor_peers": [{"host": "peer0.org1.example.com","port": 7051}]},"version": "0"}}' Org1MSPconfig.json
Generating anchor peer update transaction for Org1 on channel mychannel
+ configtxlator proto_encode --input Org1MSPconfig.json --type common.Config
configtxlator: error: Error decoding: error decoding input: error unmarshaling intermediate JSON: EOF
+ configtxlator proto_encode --input Org1MSPmodified_config.json --type common.Config
configtxlator: error: Error decoding: error decoding input: error unmarshaling intermediate JSON: EOF
+ configtxlator compute_update --channel_id mychannel --original original_config.pb --updated modified_config.pb
configtxlator: error: Error computing update: error computing config update: no channel group included for original config
+ configtxlator proto_decode --input config_update.pb --type common.ConfigUpdate
+ jq .
++ cat config_update.json
+ echo '{"payload":{"header":{"channel_header":{"channel_id":"mychannel", "type":2}},"data":{"config_update":{' '"channel_id":' '"",' '"isolated_data":' '{},' '"read_set":' null, '"write_set":' null '}}}}'
+ configtxlator proto_encode --input config_update_in_envelope.json --type common.Envelope
2021-07-26 21:26:40.664 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'mychannel': error authorizing update: ConfigUpdate for channel '' but envelope for channel 'mychannel'
Anchor peer update failed
One of the speculation I have for the problem is that orderers are unable to talk to each other because as soon as I remove the additional 2 orderers from consenters(configtx.yaml) everything seems to work
If anyone can provide me any solution it would be a great help.
If you think the added orderer is not working,
you can check orderer's raft status here
http://[YOUR ORDERER IP]:8443/metrics
the port 8443 defined in orderer.yaml
Operations:
ListenAddress: [] < -- here
and Ctrl + F, search "consensus_etcdraft_is_leader"
The problem is the second and third orderer not join the channel yet.
The solution is join the second and third orderer to the channel with osnadmin command. Similar command you can found in createChannel.sh in the sample test-network. Here is the command for joining the orderer to the cahnnel
# join orderer 2
osnadmin channel join --channelID $CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o localhost:8053 --ca-file "$ORDERER_CA" --client-cert "$ORDERER2_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER2_ADMIN_TLS_PRIVATE_KEY" >&log.txt
# join orderer 3
osnadmin channel join --channelID $CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o localhost:9053 --ca-file "$ORDERER_CA" --client-cert "$ORDERER3_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER3_ADMIN_TLS_PRIVATE_KEY" >&log.txt
Full function in createChannel.sh will be :
createChannel() {
setGlobals 1
# Poll in case the raft leader is not set yet
local rc=1
local COUNTER=1
while [ $rc -ne 0 -a $COUNTER -lt $MAX_RETRY ] ; do
sleep $DELAY
set -x
infoln "Orderer join channel"
osnadmin channel join --channelID $CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o localhost:7053 --ca-file "$ORDERER_CA" --client-cert "$ORDERER_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER_ADMIN_TLS_PRIVATE_KEY" >&log.txt
infoln "Orderer2 join channel"
osnadmin channel join --channelID $CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o localhost:8053 --ca-file "$ORDERER_CA" --client-cert "$ORDERER2_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER2_ADMIN_TLS_PRIVATE_KEY" >&log.txt
infoln "Orderer3 join channel"
osnadmin channel join --channelID $CHANNEL_NAME --config-block ./channel-artifacts/${CHANNEL_NAME}.block -o localhost:9053 --ca-file "$ORDERER_CA" --client-cert "$ORDERER3_ADMIN_TLS_SIGN_CERT" --client-key "$ORDERER3_ADMIN_TLS_PRIVATE_KEY" >&log.txt
res=$?
{ set +x; } 2>/dev/null
let rc=$res
COUNTER=$(expr $COUNTER + 1)
done
cat log.txt
verifyResult $res "Channel creation failed"
}
Note : make sure you have added the Orderer2 and Orderer3 TLS certificate Variable in envVar.sh
I am quite sure that this answer doesn't directly apply to OPs question but it somewhat applies to the same warning, so i am putting this here. It might help someone.
Create application channel command (execute from peerOrg admin cli)
peer channel create \
-c mychannel \
-o orderer-org0:6050 \
++ -t 50s
-f "/pathtoartifacts/artifacts/mychannel.tx" \
--outputBlock "/pathtoartifacts/artifacts/mychannel.block" \
--tls \
--cafile "/pathtoorderer/msp/tlscacerts/ca.pem"
The command timed out, returning multiple times the warning bellow.
2021-07-26 12:42:59.856 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{SERVICE_UNAVAILABLE}
Solution
After add the -t timeout flag to a higher value, the channel block was successfully created and stored to the output artifacts path.
I am not sure of the reason, but i assume due to the kubernetes based hyperledger fabric deployment, the inter-containers (orderers, endorses, admin-cli) communication is a bit more slow, than when running everything in localhost.
Note: the warning still appear, but i now receive a block and the expected output artifact in the artifacts folder.
This question already has answers here:
Why are Futures within Futures running sequentially when started on Akka Dispatcher
(3 answers)
Closed 3 years ago.
I'm trying to test the ExecutionContext behaviour in a play app, and found that I'm not able to achieve any degree of parallelism when I'm using the default dispatcher either by calling as.dispatcher, as.dispatchers.lookup("akka.actor.default-dispatcher") or passing the default execution context as a parameter to my Controller class:
class HomeController #Inject()(cc: ControllerComponents)(implicit ec: ExecutionContext)
I'm building on the play examples available in here. And adding/altering the following configuration:
routes
GET /futures controllers.HomeController.testFutures(dispatcherId: String)
common.conf
akka {
my-dispatcher {
executor = "fork-join-executor"
fork-join-executor {
# vm-cores = 4
parallelism-min = 4
parallelism-factor = 2.0
# 2x vm-cores
parallelism-max = 8
}
}
actor.default-dispatcher {
executor = "fork-join-executor"
fork-join-executor {
# vm-cores = 4
parallelism-min = 4
parallelism-factor = 2.0
# 2x vm-cores
parallelism-max = 8
}
}
}
HomeController
#Singleton
class HomeController #Inject()(cc: ControllerComponents, as: ActorSystem) extends AbstractController(cc) {
import HomeController._
def testFutures(dispatcherId: String) = Action.async { implicit request =>
implicit val dispatcher = as.dispatchers.lookup(dispatcherId)
Future.sequence((0 to 10).map(i => Future {
val time = 1000 + Random.nextInt(200)
log.info(s"Sleeping #$i for $time ms")
Thread.sleep(time)
log.info(s"Awakening #$i")
})).map(_ => Ok("ok"))
}
}
For some reason, calls to http://localhost:9000/futures?dispatcherId=akka.actor.default-dispatcher (default dispatcher) don't parallelize and produce the following output:
[info] c.HomeController - Sleeping #0 for 1044 ms
[info] c.HomeController - Awakening #0
[info] c.HomeController - Sleeping #1 for 1034 ms
[info] c.HomeController - Awakening #1
[info] c.HomeController - Sleeping #2 for 1031 ms
[info] c.HomeController - Awakening #2
[info] c.HomeController - Sleeping #3 for 1065 ms
[info] c.HomeController - Awakening #3
[info] c.HomeController - Sleeping #4 for 1082 ms
[info] c.HomeController - Awakening #4
[info] c.HomeController - Sleeping #5 for 1057 ms
[info] c.HomeController - Awakening #5
[info] c.HomeController - Sleeping #6 for 1090 ms
[info] c.HomeController - Awakening #6
[info] c.HomeController - Sleeping #7 for 1165 ms
[info] c.HomeController - Awakening #7
[info] c.HomeController - Sleeping #8 for 1173 ms
[info] c.HomeController - Awakening #8
[info] c.HomeController - Sleeping #9 for 1034 ms
[info] c.HomeController - Awakening #9
[info] c.HomeController - Sleeping #10 for 1056 ms
[info] c.HomeController - Awakening #10
But calls to this http://localhost:9000/futures?dispatcherId=akka.my-dispatcher (using another dispatcher) parallelize correclty and produce the following output.
[info] c.HomeController - Sleeping #1 for 1191 ms
[info] c.HomeController - Sleeping #0 for 1055 ms
[info] c.HomeController - Sleeping #7 for 1196 ms
[info] c.HomeController - Sleeping #4 for 1121 ms
[info] c.HomeController - Sleeping #6 for 1040 ms
[info] c.HomeController - Sleeping #2 for 1016 ms
[info] c.HomeController - Sleeping #5 for 1107 ms
[info] c.HomeController - Sleeping #3 for 1165 ms
[info] c.HomeController - Awakening #2
[info] c.HomeController - Sleeping #8 for 1002 ms
[info] c.HomeController - Awakening #6
[info] c.HomeController - Sleeping #9 for 1127 ms
[info] c.HomeController - Awakening #0
[info] c.HomeController - Sleeping #10 for 1016 ms
[info] c.HomeController - Awakening #5
[info] c.HomeController - Awakening #4
[info] c.HomeController - Awakening #3
[info] c.HomeController - Awakening #1
[info] c.HomeController - Awakening #7
[info] c.HomeController - Awakening #8
[info] c.HomeController - Awakening #10
[info] c.HomeController - Awakening #9
Any ideas why this could be happening?
I think the behavior is given by akka.actor.default-dispatcher that is of the type BatchingExecutor and this will try optimize in the cases of operations such as map/flatmap by executing them in the same thread to avoid unnecessary schedules . In the case where we are going to block we can indicate it with a hint as scala.concurrent.blocking (Thread.sleep (time)) and in this way a mark is stored in a ThreadLocal[BlockContext] which indicates the intention to block and does not apply the optimizations but throws the operation in another thread.
if you change this line Thread.sleep(time) for this scala.concurrent.blocking(Thread.sleep(time)) you will get the desired behavior
#Singleton
class HomeController #Inject()(cc: ControllerComponents, as: ActorSystem) extends AbstractController(cc) {
import HomeController._
def testFutures(dispatcherId: String) = Action.async { implicit request =>
implicit val dispatcher = as.dispatchers.lookup(dispatcherId)
Future.sequence((0 to 10).map(i => Future {
val time = 1000 + Random.nextInt(200)
log.info(s"Sleeping #$i for $time ms")
scala.concurrent.blocking(Thread.sleep(time))
log.info(s"Awakening #$i")
})).map(_ => Ok("ok"))
}
}
[info] play.api.Play - Application started (Dev) (no global state)
Sleeping #0 for 1062 ms
Sleeping #1 for 1128 ms
Sleeping #2 for 1189 ms
Sleeping #3 for 1105 ms
Sleeping #4 for 1169 ms
Sleeping #5 for 1178 ms
Sleeping #6 for 1057 ms
Sleeping #7 for 1003 ms
Sleeping #8 for 1164 ms
Sleeping #9 for 1029 ms
Sleeping #10 for 1005 ms
Awakening #7
Awakening #10
Awakening #9
Awakening #6
Awakening #0
Awakening #3
Awakening #1
Awakening #8
Awakening #4
Awakening #5
Awakening #2
I configured manually a Hyperledger Fabric environment using the binary files (without using Docker or fabric samples scripts). I deployed successfully 1 orderer node and 2 peers node (one peer per org), but fails creating the channel.
I used crytogen for crypto material and cryptotxgen to create the genesis block and the channel transaction.
Orderer Logs:
2019-01-06 19:01:05.601 UTC [cauthdsl] func1 -> DEBU 176 0xc0001b8ea0 gate 1546801265601901160 evaluation starts
2019-01-06 19:01:05.601 UTC [cauthdsl] func2 -> DEBU 177 0xc0001b8ea0 signed by 0 principal evaluation starts (used [false])
2019-01-06 19:01:05.601 UTC [cauthdsl] func2 -> DEBU 178 0xc0001b8ea0 processing identity 0 with bytes of ebff60
2019-01-06 19:01:05.602 UTC [cauthdsl] func2 -> DEBU 179 0xc0001b8ea0 identity 0 does not satisfy principal: the identity is a member of a different MSP (expected OrdererMSP, got TheChainMSP)
2019-01-06 19:01:05.602 UTC [cauthdsl] func2 -> DEBU 17a 0xc0001b8ea0 principal evaluation fails
2019-01-06 19:01:05.602 UTC [cauthdsl] func2 -> DEBU 17b 0xc0001b8ea0 signed by 1 principal evaluation starts (used [false])
2019-01-06 19:01:05.602 UTC [cauthdsl] func2 -> DEBU 17c 0xc0001b8ea0 processing identity 0 with bytes of ebff60
2019-01-06 19:01:05.602 UTC [cauthdsl] func2 -> DEBU 17d 0xc0001b8ea0 identity 0 does not satisfy principal: the identity is a member of a different MSP (expected OrdererMSP, got TheChainMSP)
2019-01-06 19:01:05.602 UTC [cauthdsl] func2 -> DEBU 17e 0xc0001b8ea0 principal evaluation fails
2019-01-06 19:01:05.602 UTC [cauthdsl] func1 -> DEBU 17f 0xc0001b8ea0 gate 1546801265601901160 evaluation fails
2019-01-06 19:01:05.602 UTC [policies] Evaluate -> DEBU 180 Signature set did not satisfy policy /Channel/Orderer/OrdererTheChain/Writers
2019-01-06 19:01:05.602 UTC [policies] Evaluate -> DEBU 181 == Done Evaluating *cauthdsl.policy Policy /Channel/Orderer/OrdererTheChain/Writers
2019-01-06 19:01:05.602 UTC [policies] func1 -> DEBU 182 Evaluation Failed: Only 0 policies were satisfied, but needed 1 of [ OrdererTheChain.Writers ]
2019-01-06 19:01:05.602 UTC [policies] Evaluate -> DEBU 183 Signature set did not satisfy policy /Channel/Orderer/Writers
2019-01-06 19:01:05.602 UTC [policies] Evaluate -> DEBU 184 == Done Evaluating *policies.implicitMetaPolicy Policy /Channel/Orderer/Writers
2019-01-06 19:01:05.602 UTC [policies] func1 -> DEBU 185 Evaluation Failed: Only 0 policies were satisfied, but needed 1 of [ Consortiums.Writers Orderer.Writers ]
2019-01-06 19:01:05.602 UTC [policies] Evaluate -> DEBU 186 Signature set did not satisfy policy /Channel/Writers
2019-01-06 19:01:05.602 UTC [policies] Evaluate -> DEBU 187 == Done Evaluating *policies.implicitMetaPolicy Policy /Channel/Writers
2019-01-06 19:01:05.602 UTC [orderer.common.broadcast] ProcessMessage -> WARN 188 [channel: privatechannel] Rejecting broadcast of config message from 127.0.0.1:53992 because of error: Failed to reach implicit threshold of 1 sub-policies, required 1 remaining: permission denied
2019-01-06 19:01:05.602 UTC [orderer.common.server] func1 -> DEBU 189 Closing Broadcast stream
2019-01-06 19:01:05.602 UTC [comm.grpc.server] 1 -> INFO 18a streaming call completed {"grpc.start_time": "2019-01-06T19:01:05.599Z", "grpc.service": "orderer.AtomicBroadcast", "grpc.method": "Broadcast", "grpc.peer_address": "127.0.0.1:53992", "grpc.code": "OK", "grpc.call_duration": "3.042122ms"}
2019-01-06 19:01:05.605 UTC [common.deliver] Handle -> WARN 18b Error reading from 127.0.0.1:53988: rpc error: code = Canceled desc = context canceled
2019-01-06 19:01:05.605 UTC [grpc] warningf -> DEBU 18c transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:7050->127.0.0.1:53992: read: connection reset by peer
2019-01-06 19:01:05.605 UTC [orderer.common.server] func1 -> DEBU 18d Closing Deliver stream
2019-01-06 19:01:05.605 UTC [grpc] infof -> DEBU 18f transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2019-01-06 19:01:05.605 UTC [comm.grpc.server] 1 -> INFO 18e streaming call completed {"grpc.start_time": "2019-01-06T19:01:05.599Z", "grpc.service": "orderer.AtomicBroadcast", "grpc.method": "Deliver", "grpc.peer_address": "127.0.0.1:53988", "error": "rpc error: code = Canceled desc = context canceled", "grpc.code": "Canceled", "grpc.call_duration": "6.48536ms"}
2019-01-06 19:01:05.605 UTC [grpc] infof -> DEBU 190 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
peer channel create Logs:
2019-01-06 19:21:13.795 UTC [msp] setupSigningIdentity -> DEBU 035 Signing identity expires at 2029-01-03 00:49:00 +0000 UTC
2019-01-06 19:21:13.795 UTC [msp] Validate -> DEBU 036 MSP TheChainMSP validating identity
2019-01-06 19:21:13.795 UTC [msp] GetDefaultSigningIdentity -> DEBU 037 Obtaining default signing identity
2019-01-06 19:21:13.795 UTC [grpc] DialContext -> DEBU 038 parsed scheme:""
2019-01-06 19:21:13.795 UTC [grpc] DialContext -> DEBU 039 scheme "" not registered, fallback to default scheme
2019-01-06 19:21:13.795 UTC [grpc] watcher -> DEBU 03a ccResolverWrapper: sending new addresses to cc: [{localhost:7050 0 <nil>}]
2019-01-06 19:21:13.795 UTC [grpc] switchBalancer -> DEBU 03b ClientConn switching balancer to "pick_first"
2019-01-06 19:21:13.796 UTC [grpc] HandleSubConnStateChange -> DEBU 03c pickfirstBalancer: HandleSubConnStateChange: 0xc00032a490, CONNECTING
2019-01-06 19:21:13.798 UTC [grpc] HandleSubConnStateChange -> DEBU 03d pickfirstBalancer: HandleSubConnStateChange: 0xc00032a490, READY
2019-01-06 19:21:13.798 UTC [channelCmd] InitCmdFactory -> INFO 03e Endorser and orderer connections initialized
2019-01-06 19:21:13.799 UTC [msp] GetDefaultSigningIdentity -> DEBU 03f Obtaining default signing identity
2019-01-06 19:21:13.800 UTC [msp] GetDefaultSigningIdentity -> DEBU 040 Obtaining default signing identity
2019-01-06 19:21:13.800 UTC [msp.identity] Sign -> DEBU 041 Sign: plaintext: 0A96060A0B546865436861696E4D5350...53616D706C65436F6E736F727469756D
2019-01-06 19:21:13.800 UTC [msp.identity] Sign -> DEBU 042 Sign: digest: EDB773D3B4483F960DA91D9CE5E21CA9F0512B808C9AE15B56B2CB1CE663B494
2019-01-06 19:21:13.800 UTC [msp] GetDefaultSigningIdentity -> DEBU 043 Obtaining default signing identity
2019-01-06 19:21:13.800 UTC [msp] GetDefaultSigningIdentity -> DEBU 044 Obtaining default signing identity
2019-01-06 19:21:13.800 UTC [msp.identity] Sign -> DEBU 045 Sign: plaintext: 0AD2060A1A08021A0608A9AAC9E10522...898F89F93F5DEF87555ED63A455E5CFF
2019-01-06 19:21:13.800 UTC [msp.identity] Sign -> DEBU 046 Sign: digest: BAA15E471F224FBF378D144154CF6B126823800A73EF3F9122CB30888C69645F
2019-01-06 19:21:13.800 UTC [grpc] DialContext -> DEBU 047 parsed scheme: ""
2019-01-06 19:21:13.800 UTC [grpc] DialContext -> DEBU 048 scheme "" not registered, fallback to default scheme
2019-01-06 19:21:13.800 UTC [grpc] watcher -> DEBU 049 ccResolverWrapper: sending new addresses to cc: [{localhost:7050 0 <nil>}]
2019-01-06 19:21:13.800 UTC [grpc] switchBalancer -> DEBU 04a ClientConn switching balancer to "pick_first"
2019-01-06 19:21:13.800 UTC [grpc] HandleSubConnStateChange -> DEBU 04b pickfirstBalancer: HandleSubConnStateChange: 0xc000242cc0, CONNECTING
2019-01-06 19:21:13.801 UTC [grpc] HandleSubConnStateChange -> DEBU 04c pickfirstBalancer: HandleSubConnStateChange: 0xc000242cc0, READY
Error: got unexpected status: FORBIDDEN -- Failed to reach implicit threshold of 1 sub-policies, required 1 remaining: permission denied
Im not using TLS
My server is ubuntu 16.04, im using Hyperledger Fabric v1.4 release 2 (20 dec). I tried creating new crypto material and modify the configtx.yaml file, but no ones worked, i get the same error.
crypto-config.yaml:
OrdererOrgs:
- Name: Orderer
Domain: thechain.tech
Specs:
- Hostname: orderer
PeerOrgs:
- Name: AirMed Foundation
Domain: airmedfoundation.tech
Template:
Count: 2
Users:
Count: 3
- Name: The Chain
Domain: thechain.tech
Template:
Count: 2
Users:
Count: 3
configtxgen.yaml:
---
Capabilities:
# Channel capabilities apply to both the orderers and the peers and must be
# supported by both.
# Set the value of the capability to true to require it.
Channel: &ChannelCapabilities
# V1.3 for Channel is a catchall flag for behavior which has been
# determined to be desired for all orderers and peers running at the v1.3.x
# level, but which would be incompatible with orderers and peers from
# prior releases.
# Prior to enabling V1.3 channel capabilities, ensure that all
# orderers and peers on a channel are at v1.3.0 or later.
V1_3: true
# Orderer capabilities apply only to the orderers, and may be safely
# used with prior release peers.
# Set the value of the capability to true to require it.
Orderer: &OrdererCapabilities
# V1.1 for Orderer is a catchall flag for behavior which has been
# determined to be desired for all orderers running at the v1.1.x
# level, but which would be incompatible with orderers from prior releases.
# Prior to enabling V1.1 orderer capabilities, ensure that all
# orderers on a channel are at v1.1.0 or later.
V1_1: true
# Application capabilities apply only to the peer network, and may be safely
# used with prior release orderers.
# Set the value of the capability to true to require it.
Application: &ApplicationCapabilities
# V1.3 for Application enables the new non-backwards compatible
# features and fixes of fabric v1.3.
V1_3: true
# V1.2 for Application enables the new non-backwards compatible
# features and fixes of fabric v1.2 (note, this need not be set if
# later version capabilities are set)
V1_2: false
# V1.1 for Application enables the new non-backwards compatible
# features and fixes of fabric v1.1 (note, this need not be set if
# later version capabilities are set).
V1_1: false
Organizations:
- &OrdererOrg
Name: OrdererTheChain
ID: OrdererMSP
MSPDir: /home/medical/fabric/crypto-material/crypto-config/ordererOrganizations/thechain.tech/orderers/orderer.thechain.tech/msp
AdminPrincipal: Role.ADMIN
Policies:
Readers:
Type: Signature
Rule: "OR('OrdererMSP.admin', 'OrdererMSP.member')"
Writers:
Type: Signature
Rule: "OR('OrdererMSP.admin', 'OrdererMSP.member')"
Admins:
Type: Signature
Rule: "OR('OrdererMSP.admin')"
- &TheChainOrg
Name: TheChain
ID: TheChainMSP
AdminPrincipal: Role.ADMIN
AnchorPeers:
- Host: 127.0.0.1
Port: 7051
MSPDir: /home/medical/fabric/crypto-material/crypto-config/peerOrganizations/thechain.tech/users/Admin#thechain.tech/msp
Policies:
Readers:
Type: Signature
Rule: "OR('TheChainMSP.admin', 'TheChainMSP.peer', 'TheChainMSP.client')"
Writers:
Type: Signature
Rule: "OR('TheChainMSP.admin', 'TheChainMSP.client')"
Admins:
Type: Signature
Rule: "OR('TheChainMSP.admin')"
- &AirMedFoundationOrg
Name: AirMedFoundation
ID: AirMedFoundationMSP
AdminPrincipal: Role.ADMIN
AnchorPeers:
- Host: 127.0.0.1
Port: 17051
MSPDir: /home/medical/fabric/crypto-material/crypto-config/peerOrganizations/airmedfoundation.tech/users/Admin#airmedfoundation.tech/msp
Policies:
Readers:
Type: Signature
Rule: "OR('AirMedFoundationMSP.admin', 'AirMedFoundationMSP.peer', 'AirMedFoundationMSP.client')"
Writers:
Type: Signature
Rule: "OR('AirMedFoundationMSP.admin', 'AirMedFoundationMSP.client')"
Admins:
Type: Signature
Rule: "OR('AirMedFoundationMSP.admin')"
Orderer: &OrdererDefaults
# Orderer Type: The orderer implementation to start.
# Available types are "solo" and "kafka".
OrdererType: solo
Addresses:
- localhost:7050
# Batch Timeout: The amount of time to wait before creating a batch.
BatchTimeout: 2s
# Batch Size: Controls the number of messages batched into a block.
BatchSize:
# Max Message Count: The maximum number of messages to permit in a
# batch.
MaxMessageCount: 10
# Absolute Max Bytes: The absolute maximum number of bytes allowed for
# the serialized messages in a batch. If the "kafka" OrdererType is
# selected, set 'message.max.bytes' and 'replica.fetch.max.bytes' on the
# Kafka brokers to a value that is larger than this one.
AbsoluteMaxBytes: 98 MB
# Preferred Max Bytes: The preferred maximum number of bytes allowed for
# the serialized messages in a batch. A message larger than the
# preferred max bytes will result in a batch larger than preferred max
# bytes.
PreferredMaxBytes: 512 KB
# Max Channels is the maximum number of channels to allow on the ordering
# network. When set to 0, this implies no maximum number of channels.
MaxChannels: 0
Kafka:
# Brokers: A list of Kafka brokers to which the orderer connects. Edit
# this list to identify the brokers of the ordering service.
# NOTE: Use IP:port notation.
Brokers:
- kafka0:9092
- kafka1:9092
- kafka2:9092
- kafka3:9092
# Organizations is the list of orgs which are defined as participants on
# the orderer side of the network.
Organizations:
# Policies defines the set of policies at this level of the config tree
# For Orderer policies, their canonical path is
# /Channel/Orderer/<PolicyName>
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
# BlockValidation specifies what signatures must be included in the block
# from the orderer for the peer to validate it.
BlockValidation:
Type: ImplicitMeta
Rule: "ANY Writers"
# Capabilities describes the orderer level capabilities, see the
# dedicated Capabilities section elsewhere in this file for a full
# description
Capabilities:
<<: *OrdererCapabilities
Channel: &ChannelDefaults
# Policies defines the set of policies at this level of the config tree
# For Channel policies, their canonical path is
# /Channel/<PolicyName>
Policies:
# Who may invoke the 'Deliver' API
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
# Who may invoke the 'Broadcast' API
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
# By default, who may modify elements at this config level
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
# Capabilities describes the channel level capabilities, see the
# dedicated Capabilities section elsewhere in this file for a full
# description
Capabilities:
<<: *ChannelCapabilities
Application: &ApplicationDefaults
# Organizations is the list of orgs which are defined as participants on
# the application side of the network
Organizations:
# Policies defines the set of policies at this level of the config tree
# For Application policies, their canonical path is
# /Channel/Application/<PolicyName>
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "MAJORITY Admins"
Capabilities:
<<: *ApplicationCapabilities
Profiles:
TwoOrgsOrdererGenesis:
<<: *ChannelDefaults
Orderer:
<<: *OrdererDefaults
Organizations:
- *OrdererOrg
Capabilities:
<<: *OrdererCapabilities
Consortiums:
SampleConsortium:
Organizations:
- *TheChainOrg
- *AirMedFoundationOrg
TwoOrgsChannel:
Consortium: SampleConsortium
Application:
<<: *ApplicationDefaults
Organizations:
- *TheChainOrg
- *AirMedFoundationOrg
Capabilities:
<<: *ApplicationCapabilities
I found this question that have a similar problem
peer channel creation fails in Hyperledger Fabric, i tried his solution and nothing change. I think the problem is for policy problems and my peers are not sign the transaction with the correct credentials, but i dont know how to fix it.
Orderer logs:
2019-01-06 19:01:05.602 UTC [policies] func1 -> DEBU 185 Evaluation Failed: Only 0 policies were satisfied, but needed 1 of [ Consortiums.Writers Orderer.Writers ]
Peerlogs:
Error: got unexpected status: FORBIDDEN -- Failed to reach implicit threshold of 1 sub-policies, required 1 remaining: permission denied
Its clearly stating that you are not authorize to create a channel
Why:
Hyperledger fabric is designed in a secure way
For every operation, you need a valid Authorization and Authentication
How:
Please add admin credentials of majority orgs while creating the channel
Tips:
If you use CLI then add admin private key and certificate while creating the channel.
Help: If you need more details feel free to comment here iam more than happy to help
When my computer-on-module (Adlink comExpressBT) boots Linux 4.4.3 I get the following error which corresponds to a PnP failure which prevents the SMBus driver from loading: (FYI - The SMBus is used on the I2C interface)
Oct 27 01:22:28 buildroot kern.info kernel: [ 5.701720] i2c /dev entries driver
Oct 27 01:22:28 buildroot kern.warn kernel: [ 5.709082] ACPI Warning: SystemIO range 0x000000000000E000-0x000000000000E01F conflicts with OpRegion 0x000000000000E000-0x000000000000E00F (\_SB_.PCI0.SBUS.SMBI) (20150930/utaddress-254)
Oct 27 01:22:28 buildroot kern.info kernel: [ 5.742765] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
I removed ACPI (and acpid) from Buildroot and then got the SMBus driver to load. However, PnP then failed for a custom PCIe FPGA we are using which maps RAM. The MSI capability was not read properly leading to the following:
Oct 31 22:06:25 buildroot kern.crit kernel: [ 478.997584] /dev/dcb: initialising
Oct 31 22:06:25 buildroot kern.warn kernel: [ 479.007245] pci_drv_DCB 0000:01:00.0: using bridge 0000:00:1c.0 INT A to get IRQ 16
Oct 31 22:06:25 buildroot kern.info kernel: [ 479.025554] pci_drv_DCB 0000:01:00.0: PCI->APIC IRQ transform: INT A -> IRQ 16
Oct 31 22:06:25 buildroot kern.info kernel: [ 479.043131] pci_drv_DCB 0000:01:00.0: enabling device (0000 -> 0002)
Oct 31 22:06:25 buildroot kern.crit kernel: [ 479.059059] pci_DCB: memstart=0x0, memlen=0x0
Oct 31 22:06:25 buildroot kern.err kernel: [ 479.070813] Memory address conflict for device "0000:01:00.0", request_mem_region( memstart=0x0, memlen=0x0 ) failed.
Oct 31 22:06:25 buildroot kern.warn kernel: [ 479.095176] Unable to request memory for device.
Oct 31 22:06:25 buildroot kern.warn kernel: [ 479.107604] pci_drv_DCB: probe of 0000:01:00.0 failed with error -12**
Though the region address looks good, MSI capability is not read properly: (From "lspci -vvv")
01:00.0 RAM memory: Xilinx Corporation Default PCIe endpoint ID
Subsystem: Xilinx Corporation Default PCIe endpoint ID
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 0
Region 0: Memory at b0700000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at b0600000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME
Capabilities: [48] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000**
Capabilities: [60] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 1, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
It seems that I have to have ACPI enabled to some degree in order to have PnP-PCI discovery to work properly. Any ideas? Thanks in advance. -Ross R.