Hyperledger Fabric: Add Orderer in running raft network - hyperledger-fabric

Question

I want to add a new orderer node in a running hyperledger-fabric network. I've used etcdraft as consensus between the orderers.
I've followed step by step process of this accepted answer. In step #15, it is mentioned that, I've to fetch the latest config block from the system channel and use this block as genesis block of the new ordering service.
So, I've copied the config block to the channel-artifacts folder [file name: config_block.pb] and then spin up a new container using a docker-compose.yaml file. As a reference I am posting the docker-compose.yaml file here:
version: '2'
volumes:
orderer6.example.com:
networks:
byfn:
services:
orderer6.example.com:
image: hyperledger/fabric-orderer:latest
environment:
- FABRIC_LOGGING_SPEC=INFO
- ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
- ORDERER_GENERAL_GENESISMETHOD=file
- ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block
- ORDERER_GENERAL_LOCALMSPID=OrdererMSP
- ORDERER_GENERAL_LOCALMSPDIR=/var/hyperledger/orderer/msp
# enabled TLS
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
- ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1
- ORDERER_KAFKA_VERBOSE=true
- ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/var/hyperledger/orderer/tls/server.crt
- ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/var/hyperledger/orderer/tls/server.key
- ORDERER_GENERAL_CLUSTER_ROOTCAS=[/var/hyperledger/orderer/tls/ca.crt]
working_dir: /opt/gopath/src/github.com/hyperledger/fabric
command: orderer
container_name: orderer6.example.com
networks:
- byfn
volumes:
- ./channel-artifacts/config_block.pb:/var/hyperledger/orderer/orderer.genesis.block
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer6.example.com/msp:/var/hyperledger/orderer/msp
- ./crypto-config/ordererOrganizations/example.com/orderers/orderer6.example.com/tls/:/var/hyperledger/orderer/tls
- orderer6.example.com:/var/hyperledger/production/orderer
ports:
- 13050:7050
Finally, I'm running the orderer service using:
docker-compose -f orderer6.yaml up
The container starts and after a few seconds it crashes. Log file of the container shows:
orderer6.example.com | 2020-01-07 09:03:12.363 UTC [orderer.common.cluster.replication] obtainStream -> INFO 03f Sending request for block [2] to orderer.example.com:7050
orderer6.example.com | 2020-01-07 09:03:12.365 UTC [orderer.common.cluster.replication] pullBlocks -> INFO 040 Got block [2] of size 47 KB from orderer.example.com:7050
orderer6.example.com | 2020-01-07 09:03:12.377 UTC [orderer.common.cluster] appendBlock -> PANI 041 Failed to write block [2]: unexpected Previous block hash. Expected PreviousHash = [1c99accab83de7a415061120831519badf4807c97a0195ab5f42bab979a7dc09], PreviousHash referred in the latest block= [37f11444fb0b1ca51419b5f1a95fab616d5179ed44a87716677f1b706c0b1854]
orderer6.example.com | panic: Failed to write block [2]: unexpected Previous block hash. Expected PreviousHash = [1c99accab83de7a415061120831519badf4807c97a0195ab5f42bab979a7dc09], PreviousHash referred in the latest block= [37f11444fb0b1ca51419b5f1a95fab616d5179ed44a87716677f1b706c0b1854]
orderer6.example.com |
orderer6.example.com | goroutine 1 [running]:
orderer6.example.com | github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00014db80, 0x0, 0x0, 0x0)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x546
orderer6.example.com | github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc000010a58, 0xc00004ae04, 0x1548ed8, 0x1e, 0xc00030b458, 0x2, 0x2, 0x0, 0x0, 0x0)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0x101
orderer6.example.com | github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(...)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
orderer6.example.com | github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/cluster.(*Replicator).appendBlock(0xc000089bc0, 0xc0002e4a80, 0x16b8760, 0xc00068d0e0, 0xc000571f60, 0x10)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/replication.go:253 +0x283
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/cluster.(*Replicator).pullChannelBlocks(0xc000089bc0, 0xc000571f60, 0x10, 0xc00053e2a0, 0x3, 0x16b8760, 0xc00068d0e0, 0x0, 0x0)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/replication.go:221 +0xd8
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/cluster.(*Replicator).PullChannel(0xc000089bc0, 0xc000571f60, 0x10, 0x0, 0x0)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/replication.go:207 +0x411
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/cluster.(*Replicator).ReplicateChains(0xc000089bc0, 0xc00030ba88, 0x1, 0x1)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/cluster/replication.go:163 +0x3ed
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/server.(*replicationInitiator).replicateNeededChannels(0xc000214180, 0xc00011ce00)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/onboarding.go:102 +0x19a
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/server.(*replicationInitiator).replicateIfNeeded(0xc000214180, 0xc00011ce00)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/onboarding.go:48 +0x9a
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/server.Start(0x15309fe, 0x5, 0xc000394900)
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:124 +0x11c8
orderer6.example.com | github.com/hyperledger/fabric/orderer/common/server.Main()
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:91 +0x208
orderer6.example.com | main.main()
orderer6.example.com | /opt/gopath/src/github.com/hyperledger/fabric/orderer/main.go:15 +0x20
orderer6.example.com exited with code 2
Details log of the orderer is in here.
What's the problem in my approach? I've used byfn network as my test bed by running:
./byfn up -o etcdraft

Answer 1

There is an issue with respect to the volumes being mount to the new orderer container. It is using an existing volume for the new orderer, because probably you kept pulling down the "byfn.sh" script and just used it again to up the network.
Go to the docker-compose file which is used to bring up the new orderer node. Check the volumes section at the top of the file as well as within the container definition where it is actually mounted:
volumes:
orderer6.example.com:
and
- orderer6.example.com:/var/hyperledger/production/orderer.
Change the name of the volume being created and mounted to completely new name. Every step mentioned in the official documentation just works fine thereafter.