Peer fails to join a channel due to some MSP mismatch - hyperledger-fabric

Question

The problem
I'm trying to create a network with two organizations and one channel. The peer from the first organization joins the channel without an issue, but when I try to add a peer from the second one to the channel I get this error in its logs:
2020-08-12 18:53:08.351 UTC [endorser] Validate -> WARN 110 access denied: channel expected MSP ID RegulatorMSP, received BrokerMSP channel= txID=5096f00a
2020-08-12 18:53:08.351 UTC [comm.grpc.server] 1 -> INFO 111 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.23.0.1:36922 error="error validating proposal: access denied: channel [] creator org [BrokerMSP]" grpc.code=Unknown grpc.call_duration=5.7274ms
I've been fiddling with the configuration for the last 8 hours looking for mistakes, but came up short, and it's driving me insane.
The configuration
Here is a script that I use to spin up the network:
DOCK_FOLDER=$PWD
echo '[=======================================================]'
echo '[============== A: Set up the environment ==============]'
echo '[=======================================================]'
echo '[==================== A.1: Cleanup =====================]'
./clean.sh all # this just kills docker containers and cleans up hlf artifacts
cd config
echo ''
echo '[================= A.2: Crypto Config =================]'
cryptogen generate --config=crypto-config.yaml
export FABRIC_CFG_PATH=$PWD
echo ''
echo '[=============== A.3: Block Generation ================]'
configtxgen -outputBlock ./orderer/genesis.block -channelID ordererchannel -profile CompOrdererGenesis
echo ''
echo '[============= A.4: Transaction Generation ============]'
configtxgen -outputCreateChannelTx compchannel.tx -channelID compchannel -profile CompChannel
cd $DOCK_FOLDER
echo ''
echo '[================= A.5: Docker Compose ================]'
docker-compose -f ./config/docker-compose-base.yaml up -d
echo ''
echo '[=============== A.6: Initialize CouchDB ==============]'
sleep 5
curl -X PUT <username>:<password>#couchdb.regulator.com:5984/_users
curl -X PUT <username>:<password>#couchdb.regulator.com:5984/_replicator
curl -X PUT <username>:<password>#couchdb.regulator.com:5984/_global_changes
curl -X PUT <username>:<password>#couchdb.broker.com:6984/_users
curl -X PUT <username>:<password>#couchdb.broker.com:6984/_replicator
curl -X PUT <username>:<password>#couchdb.broker.com:6984/_global_changes
sleep 10
echo ''
echo '[=======================================================]'
echo '[=========== B. Manage the network: Regulator ==========]'
echo '[=======================================================]'
echo '[========== B.1: Set context to Regulator org ==========]'
export ORG_CONTEXT="regulator"
export ORG_NAME="Regulator"
export CORE_PEER_LOCALMSPID="RegulatorMSP"
export FABRIC_LOGGING_SPEC=INFO
export FABRIC_CFG_PATH=$PWD/config/regulator
export CORE_PEER_ADDRESS=peer.regulator.com:7051
export CORE_PEER_MSPCONFIGPATH=$PWD/config/crypto-config/peerOrganizations/regulator.com/users/Admin#regulator.com/msp
export ORDERER_ADDRESS=orderer.regulator.com:7050
echo ''
echo '[========== B.2: Create compchannel channel ==========]'
peer channel create -c compchannel -f ./config/compchannel.tx --outputBlock ./config/compchannel.block -o $ORDERER_ADDRESS
sleep 10s
echo ''
echo '[====== B.3: Join regulator peer to compchannel ======]'
peer channel join -b ./config/compchannel.block -o $ORDERER_ADDRESS
sleep 10s
echo ''
echo '[================= B.4: Update anchors =================]'
PEER_FABRIC_CFG_PATH=$FABRIC_CFG_PATH
FABRIC_CFG_PATH=./config
configtxgen -outputAnchorPeersUpdate ./config/peer-update.tx -asOrg $ORG_NAME -channelID compchannel -profile CompChannel
FABRIC_CFG_PATH=$PEER_FABRIC_CFG_PATH
peer channel update -f ./config/peer-update.tx -c compchannel -o $ORDERER_ADDRESS
sleep 10s
echo ''
echo '[=======================================================]'
echo '[============ C. Manage the network: Broker ============]'
echo '[=======================================================]'
echo '[=========== C.1: Set context to Broker org ============]'
export ORG_CONTEXT="broker"
export ORG_NAME="Broker"
export CORE_PEER_LOCALMSPID="BrokerMSP"
export FABRIC_LOGGING_SPEC=INFO
export FABRIC_CFG_PATH=$PWD/config/broker
export CORE_PEER_ADDRESS=peer.broker.com:8051
export CORE_PEER_MSPCONFIGPATH=$PWD/config/crypto-config/peerOrganizations/broker.com/users/Admin#broker.com/msp
export ORDERER_ADDRESS=orderer.regulator.com:7050
echo ''
echo '[======== C.3: Join broker peer to compchannel =======]'
peer channel join -b ./config/compchannel.block -o $ORDERER_ADDRESS ### !!! THIS STEP FAILS !!!
sleep 10s
echo ''
echo '[================= C.4: Update anchors =================]'
PEER_FABRIC_CFG_PATH=$FABRIC_CFG_PATH
FABRIC_CFG_PATH=./config
configtxgen -outputAnchorPeersUpdate ./config/peer-update.tx -asOrg $ORG_NAME -channelID compchannel -profile CompChannel
FABRIC_CFG_PATH=$PEER_FABRIC_CFG_PATH
peer channel update -f ./config/peer-update.tx -c compchannel -o $ORDERER_ADDRESS
orderer/orderer.yaml
General:
BootstrapMethod: file
BootstrapFile: /var/hyperledger/config/genesis.block
BCCSP:
Default: SW
SW:
HASH: SHA2
Security: 256
FileKeyStore:
Keystore:
LocalMSPDir: /var/hyperledger/msp
LocalMSPID: OrdererMSP
ListenAddress: 0.0.0.0
ListenPort: 7050
Cluster:
SendBufferSize: 10
ClientCertificate:
ClientPrivateKey:
ListenPort:
ListenAddress:
ServerCertificate:
ServerPrivateKey:
Keepalive:
ServerMinInterval: 60s
ServerInterval: 7200s
ServerTimeout: 20s
TLS:
Enabled: false
PrivateKey: ./server.key
Certificate: ./server.crt
RootCAs:
- ./ca.crt
ClientAuthRequired: false
ClientRootCAs:
FileLedger:
Location: /var/ledger
Prefix: hyperledger-fabric-ordererledger
Debug:
BroadcastTraceDir:
DeliverTraceDir:
Operations:
ListenAddress: 127.0.0.1:8443
TLS:
Enabled: false
Certificate:
PrivateKey:
ClientAuthRequired: false
RootCAs: []
Metrics:
Provider: disabled
Statsd:
Network: udp
Address: 127.0.0.1:8125
WriteInterval: 30s
Prefix:
Consensus:
WALDir: /var/hyperledger/production/orderer/etcdraft/wal
SnapDir: /var/hyperledger/production/orderer/etcdraft/snapshot
regulator/core.yaml (broker/core.yaml is nearly identical, just exchange regulator for broker)
peer:
id: peer.regulator.com
networkId: dev
listenAddress: 0.0.0.0:7051
address: 0.0.0.0:7051
addressAutoDetect: false
gomaxprocs: -1
keepalive:
minInterval: 60s
client:
interval: 60s
timeout: 20s
deliveryClient:
interval: 60s
timeout: 20s
gossip:
bootstrap:
useLeaderElection: false
orgLeader: true
membershipTrackerInterval: 5s
endpoint:
maxBlockCountToStore: 100
maxPropagationBurstLatency: 10ms
maxPropagationBurstSize: 10
propagateIterations: 1
propagatePeerNum: 3
pullInterval: 4s
pullPeerNum: 3
requestStateInfoInterval: 4s
publishStateInfoInterval: 4s
stateInfoRetentionInterval:
publishCertPeriod: 10s
skipBlockVerification: false
dialTimeout: 3s
connTimeout: 2s
recvBuffSize: 20
sendBuffSize: 200
digestWaitTime: 1s
requestWaitTime: 1500ms
responseWaitTime: 2s
aliveTimeInterval: 5s
aliveExpirationTimeout: 25s
reconnectInterval: 25s
externalEndpoint: peer.regulator.com:7051
election:
startupGracePeriod: 15s
membershipSampleInterval: 1s
leaderAliveThreshold: 10s
leaderElectionDuration: 5s
pvtData:
pullRetryThreshold: 60s
transientstoreMaxBlockRetention: 1000
pushAckTimeout: 3s
btlPullMargin: 10
reconcileBatchSize: 10
reconcileSleepInterval: 1m
reconciliationEnabled: true
tls:
enabled: false
clientAuthRequired: false
cert:
file: tls/server.crt
key:
file: tls/server.key
rootcert:
file: tls/ca.crt
clientRootCAs:
files:
- tls/ca.crt
clientKey:
file:
clientCert:
file:
authentication:
timewindow: 15m
fileSystemPath: /var/hyperledger/production
BCCSP:
Default: SW
SW:
Hash: SHA2
Security: 256
FileKeyStore:
KeyStore:
PKCS11:
Library:
Label:
Pin:
Hash:
Security:
FileKeyStore:
KeyStore:
mspConfigPath: /var/hyperledger/msp
localMspId: RegulatorMSP
client:
connTimeout: 3s
deliveryclient:
reconnectTotalTimeThreshold: 3600s
connTimeout: 3s
reConnectBackoffThreshold: 3600s
localMspType: bccsp
profile:
enabled: false
listenAddress: 0.0.0.0:6060
adminService:
handlers:
authFilters:
- name: DefaultAuth
- name: ExpirationCheck
decorators:
- name: DefaultDecorator
endorsers:
escc:
name: DefaultEndorsement
library:
validators:
vscc:
name: DefaultValidation
library:
validatorPoolSize:
discovery:
enabled: true
authCacheEnabled: true
authCacheMaxSize: 1000
authCachePurgeRetentionRatio: 0.75
orgMembersAllowedAccess: false
vm:
endpoint: unix:///var/run/docker.sock
docker:
tls:
enabled: false
ca:
file: docker/ca.crt
cert:
file: docker/tls.crt
key:
file: docker/tls.key
attachStdout: false
hostConfig:
NetworkMode: host
Dns:
# - 192.168.0.1
LogConfig:
Type: json-file
Config:
max-size: "50m"
max-file: "5"
Memory: 2147483648
chaincode:
id:
path:
name:
builder: $(DOCKER_NS)/fabric-ccenv:$(TWO_DIGIT_VERSION)
pull: false
golang:
runtime: $(DOCKER_NS)/fabric-baseos:$(TWO_DIGIT_VERSION)
dynamicLink: false
java:
runtime: $(DOCKER_NS)/fabric-javaenv:$(TWO_DIGIT_VERSION)
node:
runtime: $(DOCKER_NS)/fabric-nodeenv:$(TWO_DIGIT_VERSION)
externalBuilders: []
installTimeout: 300s
startuptimeout: 300s
executetimeout: 30s
mode: net
keepalive: 0
system:
_lifecycle: enable
cscc: enable
lscc: enable
escc: enable
vscc: enable
qscc: enable
logging:
level: info
shim: warning
format: "%{color}%{time:2006-01-02 15:04:05.000 MST} [%{module}] %{shortfunc} -> %{level:.4s} %{id:03x}%{color:reset} %{message}"
ledger:
blockchain:
state:
stateDatabase: CouchDB
totalQueryLimit: 100000
couchDBConfig:
couchDBAddress: couchdb.regulator.com:5984
username: <username>
password: <password>
maxRetries: 3
maxRetriesOnStartup: 20
requestTimeout: 35s
internalQueryLimit: 1000
maxBatchUpdateSize: 1000
warmIndexesAfterNBlocks: 1
createGlobalChangesDB: false
history:
enableHistoryDatabase: true
pvtdataStore:
collElgProcMaxDbBatchSize: 5000
collElgProcDbBatchesInterval: 1000
operations:
listenAddress: 127.0.0.1:9443
tls:
enabled: false
cert:
file:
key:
file:
clientAuthRequired: false
clientRootCAs:
files: []
metrics:
provider: disabled
statsd:
network: udp
address: 127.0.0.1:8125
writeInterval: 10s
prefix:
configtx.yaml
Capabilities:
Application: &ApplicationCapabilities
V2_0: true
Orderer: &OrdererCapabilities
V2_0: true
Channel: &ChannelCapabilities
V2_0: true
Organizations:
- &Orderer
Name: Orderer
ID: OrdererMSP
MSPDir: ./crypto-config/ordererOrganizations/regulator.com/msp
Policies: &OrdererPolicies
Readers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Writers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Admins:
Type: Signature
Rule: "OR('OrdererMSP.admin')"
Endorsement:
Type: Signature
Rule: "OR('OrdererMSP.member')"
- &Regulator
Name: Regulator
ID: RegulatorMSP
MSPDir: ./crypto-config/peerOrganizations/regulator.com/msp
Policies: &RegulatorPolicies
Readers:
Type: Signature
Rule: "OR('RegulatorMSP.member')"
Writers:
Type: Signature
Rule: "OR('RegulatorMSP.member')"
Admins:
Type: Signature
Rule: "OR('RegulatorMSP.admin')"
Endorsement:
Type: Signature
Rule: "OR('RegulatorMSP.member')"
AnchorPeers:
- Host: peer.regulator.com
Port: 7051
- &Broker
Name: Broker
ID: BrokerMSP
MSPDir: ./crypto-config/peerOrganizations/broker.com/msp
Policies: &BrokerPolicies
Readers:
Type: Signature
Rule: "OR('BrokerMSP.member')"
Writers:
Type: Signature
Rule: "OR('BrokerMSP.member')"
Admins:
Type: Signature
Rule: "OR('BrokerMSP.member')"
Endorsement:
Type: Signature
Rule: "OR('BrokerMSP.member')"
AnchorPeers:
- Host: peer.broker.com
Port: 7051
Orderer: &OrdererDefaults
OrdererType: solo
Addresses:
- orderer.regulator.com:7050
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "ANY Admins"
BlockValidation:
Type: ImplicitMeta
Rule: "ANY Writers"
BatchTimeout: 2s
BatchSize:
MaxMessageCount: 10
AbsoluteMaxBytes: 98 MB
PreferredMaxBytes: 512 KB
Capabilities:
<<: *OrdererCapabilities
Application: &ApplicationDefaults
ACLs: &ACLsDefault
lscc/ChaincodeExists: /Channel/Application/Readers
lscc/GetDeploymentSpec: /Channel/Application/Readers
lscc/GetChaincodeData: /Channel/Application/Readers
lscc/GetInstantiatedChaincodes: /Channel/Application/Readers
qscc/GetChainInfo: /Channel/Application/Readers
qscc/GetBlockByNumber: /Channel/Application/Readers
qscc/GetBlockByHash: /Channel/Application/Readers
qscc/GetTransactionByID: /Channel/Application/Readers
qscc/GetBlockByTxID: /Channel/Application/Readers
cscc/GetConfigBlock: /Channel/Application/Readers
cscc/GetConfigTree: /Channel/Application/Readers
cscc/SimulateConfigTreeUpdate: /Channel/Application/Readers
peer/Propose: /Channel/Application/Writers
peer/ChaincodeToChaincode: /Channel/Application/Readers
event/Block: /Channel/Application/Readers
event/FilteredBlock: /Channel/Application/Readers
_lifecycle/CheckCommitReadiness: /Channel/Application/Writers
_lifecycle/CommitChaincodeDefinition: /Channel/Application/Writers
_lifecycle/QueryChaincodeDefinition: /Channel/Application/Readers
Policies: &ApplicationDefaultPolicies
Endorsement:
Type: ImplicitMeta
Rule: "ANY Endorsement"
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "ANY Admins"
LifecycleEndorsement:
Type: ImplicitMeta
Rule: "ANY Endorsement"
Organizations:
Capabilities:
<<: *ApplicationCapabilities
Channel: &ChannelDefaults
Policies:
Readers:
Type: ImplicitMeta
Rule: "ANY Readers"
Writers:
Type: ImplicitMeta
Rule: "ANY Writers"
Admins:
Type: ImplicitMeta
Rule: "ANY Admins"
Capabilities:
<<: *ChannelCapabilities
Profiles:
CompOrdererGenesis:
<<: *ChannelDefaults
Orderer:
<<: *OrdererDefaults
Organizations:
- <<: *Orderer
Consortiums:
CompConsortium:
Organizations:
- <<: *Regulator
- <<: *Broker
Application:
<<: *ApplicationDefaults
Organizations:
- <<: *Regulator
- <<: *Broker
CompChannel:
<<: *ChannelDefaults
Consortium: CompConsortium
Application:
<<: *ApplicationDefaults
Organizations:
- <<: *Broker
- <<: *Regulator
crypto-config.yaml
OrdererOrgs:
- Name: Orderer
Domain: regulator.com
EnableNodeOUs: true
Specs:
- Hostname: orderer
PeerOrgs:
- Name: Regulator
Domain: regulator.com
EnableNodeOUs: true
Specs:
- Hostname: peer.regulator.com
CommonName: peer.regulator.com
Users:
Count: 1
- Name: Broker
Domain: broker.com
EnableNodeOUs: true
Specs:
- Hostname: peer.broker.com
CommonName: peer.broker.com
Users:
Count: 1
docker-compose-base.yaml
version: "2"
networks:
comp:
volumes:
data-orderer.regulator.com:
data-peer.regulator.com:
data-peer.broker.com:
couchdb-data-regulator:
driver: local
couchdb-data-broker:
driver: local
services:
couchdb.regulator.com:
container_name: couchdb.regulator.com
image: couchdb:latest
environment:
- COUCHDB_USER=<username>
- COUCHDB_PASSWORD=<password>
ports:
- 5984:5984
volumes:
- couchdb-data-regulator:/opt/couchdb/data
networks:
- comp
couchdb.broker.com:
container_name: couchdb.broker.com
image: couchdb:latest
environment:
- COUCHDB_USER=<username>
- COUCHDB_PASSWORD=<password>
ports:
- 6984:5984
volumes:
- couchdb-data-broker:/opt/couchdb/data
networks:
- comp
orderer.regulator.com:
container_name: orderer.regulator.com
image: hyperledger/fabric-orderer:latest
command: orderer
environment:
- FABRIC_CFG_PATH=/var/hyperledger/config
- FABRIC_LOGGING_SPEC=DEBUG
volumes:
- ${PWD}/config/orderer:/var/hyperledger/config
- ${PWD}/config/crypto-config/ordererOrganizations/regulator.com/orderers/orderer.regulator.com/msp:/var/hyperledger/msp
- ${PWD}/config/crypto-config/ordererOrganizations/regulator.com/orderers/orderer.regulator.com/tls:/var/hyperledger/tls
- data-orderer.regulator.com:/var/ledger
ports:
- 7050:7050
networks:
- comp
peer.regulator.com:
container_name: peer.regulator.com
image: hyperledger/fabric-peer:latest
environment:
- FABRIC_CFG_PATH=/var/hyperledger/config
- FABRIC_LOGGING_SPEC=DEBUG
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_net
- CORE_LEDGER_STATE_STATEDATABASE=CouchDB
- CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb.regulator.com:5984
- CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=<username>
- CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=<password>
command: [sh, -c, "sleep 10 && peer node start"]
volumes:
- ${PWD}/config/regulator:/var/hyperledger/config
- ${PWD}/config/crypto-config/peerOrganizations/regulator.com/peers/peer.regulator.com/msp:/var/hyperledger/msp
- ${PWD}/config/crypto-config/peerOrganizations/regulator.com/peers/peer.regulator.com/tls:/var/hyperledger/tls
- /var/run/:/var/run/
- data-peer.regulator.com:/var/hyperledger/production
depends_on:
- orderer.regulator.com
- couchdb.regulator.com
ports:
- 7051:7051
- 7052:7052
networks:
- comp
links:
- couchdb.regulator.com
peer.broker.com:
container_name: peer.broker.com
image: hyperledger/fabric-peer:latest
environment:
- FABRIC_CFG_PATH=/var/hyperledger/config
- FABRIC_LOGGING_SPEC=DEBUG
- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_net
- CORE_LEDGER_STATE_STATEDATABASE=CouchDB
- CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb.broker.com:6984
- CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=<username>
- CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=<password>
command: [sh, -c, "sleep 10 && peer node start"]
volumes:
- ${PWD}/config/regulator:/var/hyperledger/config
- ${PWD}/config/crypto-config/peerOrganizations/broker.com/peers/peer.broker.com/msp:/var/hyperledger/msp
- ${PWD}/config/crypto-config/peerOrganizations/broker.com/peers/peer.broker.com/tls:/var/hyperledger/tls
- /var/run/:/var/run/
- data-peer.broker.com:/var/hyperledger/production
depends_on:
- orderer.regulator.com
- couchdb.broker.com
ports:
- 8051:7051
- 8052:7052
networks:
- comp
links:
- couchdb.broker.com
extra_hosts:
- "couchdb.broker.com:<machines_ip>" # this one was due to some DNS resolution shenanigans
I hope somebody can help me solve this issue, cause I think I'm loosing sanity by the moment.

Answer 1

Turns out that I just made a goof. When you look at the docker-compose-base.yaml file under the services > peer.broker.com > volumes section, I messed up the config folder path, so both peers were running on the same configuration, but with different certificates, and that caused the discrepancy in peer authentication.