How to add a new Orderer Organization to existing Hyperledger Fabric network - hyperledger-fabric

Question

I am trying to add a new Orderer Organization to RAFT based existing ordering service. I am using the first-network from fabric-samples as the base network. While generating crypto-material, I have modified to generate crypto-material for 1 more orderer organization. The crypto-config.yaml looks like:
OrdererOrgs:
# ---------------------------------------------------------------------------
# Orderer
# ---------------------------------------------------------------------------
- Name: Orderer
Domain: example.com
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs" - See PeerOrgs below for complete description
# ---------------------------------------------------------------------------
Specs:
- Hostname: orderer
- Hostname: orderer2
- Hostname: orderer3
- Hostname: orderer4
- Hostname: orderer5
- Name: Orderer1
Domain: example1.com
EnableNodeOUs: true
Specs:
- Hostname: orderer
- Hostname: orderer2
- Hostname: orderer3
# ---------------------------------------------------------------------------
# "PeerOrgs" - Definition of organizations managing peer nodes
# ---------------------------------------------------------------------------
PeerOrgs:
# ---------------------------------------------------------------------------
# Org1
# ---------------------------------------------------------------------------
- Name: Org1
Domain: org1.example.com
EnableNodeOUs: true
Template:
Count: 2
# Start: 5
# Hostname: {{.Prefix}}{{.Index}} # default
# ---------------------------------------------------------------------------
# "Users"
# ---------------------------------------------------------------------------
# Count: The number of user accounts _in addition_ to Admin
# ---------------------------------------------------------------------------
Users:
Count: 1
# ---------------------------------------------------------------------------
# Org2: See "Org1" for full specification
# ---------------------------------------------------------------------------
- Name: Org2
Domain: org2.example.com
EnableNodeOUs: true
Template:
Count: 2
Users:
Count: 1
Also, the configtx.yaml for creating new Orderer organization's MSP in JSON format is given as:
Organizations:
- &Orderer1Org
# DefaultOrg defines the organization which is used in the sampleconfig
# of the fabric.git development environment
Name: Orderer1Org
# ID to load the MSP definition as
ID: Orderer1MSP
MSPDir: ../crypto-config/ordererOrganizations/example1.com/msp
# Policies defines the set of policies at this level of the config tree
# For organization policies, their canonical path is usually
# /Channel/<Application|Orderer>/<OrgName>/<PolicyName>
Policies:
Readers:
Type: Signature
Rule: "OR('Orderer1MSP.member')"
Writers:
Type: Signature
Rule: "OR('Orderer1MSP.member')"
Admins:
Type: Signature
Rule: "OR('Orderer1MSP.admin')"
Then I boot up the the network using byfn.sh. Now I use the cli container to modify the system channel configuration in following steps:
First I add the organization's JSON to the Orderer group as follows and submit channel update:
jq -s '.[0] * {"channel_group":{"groups":{"Orderer":{"groups": {"Orderer1Org":.[1]}}}}}' config.json orderer1org.json > modified_config.json
Then I add organization's JSON to the Consortium group as follows and submit channel update:
jq -s '.[0] * {"channel_group":{"groups":{"Consortiums":{"groups":{"SampleConsortium":{"groups": {"Orderer1MSP":.[1]}}}}}}}' config1.json orderer1org.json > modified_config1.json
Then I add organization's orderer1 TLS certificates to Consenters section and submit channel update:
cert=`base64 ../crypto/ordererOrganizations/example1.com/orderers/orderer.example1.com/tls/server.crt | sed ':a;N;$!ba;s/\n//g'`
cat modified_config1.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters += [{"client_tls_cert": "'$cert'", "host": "orderer.example1.com", "port": 7050, "server_tls_cert": "'$cert'"}] ' > modified_config2.json
Then I update the system channel configuration's Application group with new Orderer Org's JSON:
jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"Orderer1Org":.[1]}}}}}' config.json orderer1org.json > modified_config.json
Then I start one of the orderers in the new organization (orderer.example1.com) but the container is failing with following error:
2020-04-09 13:09:05.600 UTC [orderer.common.cluster.replication] fetchLastBlockSeq -> WARN 0e8 Received status:FORBIDDEN from orderer.example.com:7050: forbidden pulling the cha
nnel
2020-04-09 13:09:05.600 UTC [orderer.common.cluster.replication] func1 -> WARN 0e9 Received error of type 'forbidden pulling the channel' from {orderer.example.com:7050 [certs]}
The orderer.example.com logs throw this error:
2020-04-09 13:28:59.338 UTC [cauthdsl] deduplicate -> ERRO a3c Principal deserialization failure (the supplied identity is not valid: x509: certificate signed by unknown authorit
y) for identity 0
2020-04-09 13:28:59.338 UTC [cauthdsl] deduplicate -> ERRO a3d Principal deserialization failure (the supplied identity is not valid: x509: certificate signed by unknown authorit
y) for identity 0
2020-04-09 13:28:59.339 UTC [cauthdsl] deduplicate -> ERRO a3e Principal deserialization failure (the supplied identity is not valid: x509: certificate signed by unknown authorit
y) for identity 0
2020-04-09 13:28:59.340 UTC [cauthdsl] deduplicate -> ERRO a3f Principal deserialization failure (the supplied identity is not valid: x509: certificate signed by unknown authorit
y) for identity 0
2020-04-09 13:28:59.340 UTC [common.deliver] deliverBlocks -> WARN a40 [channel: byfn-sys-channel] Client authorization revoked for deliver request from 172.25.0.15:36196: implic
it policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Readers' sub-policies to be satisfied: permission denied
2020-04-09 13:28:59.341 UTC [comm.grpc.server] 1 -> INFO a41 streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=172.25.0.15:36196
grpc.peer_subject="CN=orderer.example1.com,L=San Francisco,ST=California,C=US" grpc.code=OK grpc.call_duration=4.992078ms

Answer 1

I was able to extend the first-network by adding a new Orderer Organization as follows:
Start the first-network through the byfn.sh script in the fabric-samples repo in the etcdraft mode.
I generated crypto-material like described in the crypto-config.yaml in the question above.
Use the configtxgen tool to print the new orderer organization's MSP into JSON format.
Mount or docker cp this JSON file to the running cli container.
Set the environment inside the cli container corresponding to existing ordering node. Import the latest system-channel configuration. Decode it to JSON format.
Edit the system channel configuration block's Orderer section to add the new orderer organization's MSP as follows:
jq -s '.[0] * {"channel_group":{"groups":{"Orderer":{"groups": {"Orderer1Org":.[1]}}}}}' config.json orderer1org.json > config1.json
Edit the system channel configuration block's Consortiums section to add the new orderer organization's MSP as follows:
jq -s '.[0] * {"channel_group":{"groups":{"Consortiums":{"groups":{"SampleConsortium":{"groups": {"Orderer1MSP":.[1]}}}}}}}' config1.json orderer1org.json > config2.json
Edit the system channel configuration block's Consenters section to add the TLS credentials for the new orderer organization's orderer.example1.com node as follows:
cert=`base64 ../crypto/ordererOrganizations/example1.com/orderers/orderer.example1.com/tls/server.crt | sed ':a;N;$!ba;s/\n//g'`
cat config2.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters += [{"client_tls_cert": "'$cert'", "host": "orderer.example1.com", "port": 7050, "server_tls_cert": "'$cert'"}] ' > modified_config.json
Encode the block, find delta, create channel update transaction, encode it as protobuf envelope and submit the channel update transaction.
Fetch the latest system channel configuration block.
Start one of the orderers (the one who was added to consenters list previously) using this latest fetched system channel configuration block as it's genesis.block file.
Perform docker exec into the cli container. Using the environment of an existing orderer node, fetch the latest system channel configuration.
Edit the system channel configuration block to add the new orderer's endpoint in the OrdererAddresses section as follows:
cat config.json | jq '.channel_group.values.OrdererAddresses.value.addresses += ["orderer.example1.com:7050"] ' > modified_config.json
Encode the block, find delta, create channel update transaction, encode it as protobuf envelope and get the block signed by Orderer1Org admin to satify the mod_policy for /Channel/OrdererAddresses resource which is set to Admins policy. This implicit meta policy expects the signature of MAJORITY Admins at that level of update. So, as the number of orderer organizations are 2 now, we need both the organization's admins to sign this system channel update transaction. Set the environment corresponding to Orderer1Org admin and run the following command:
peer channel signconfigtx -f ordorg_update_in_envelope.pb
Set the environment back to OrdererOrg admin and submit the channel update transaction. The peer channel update will automatically sign the transaction on behalf of OrdererOrg admin.
peer channel update -f ordorg_update_in_envelope.pb -c $CHANNEL_NAME -o orderer.example.com:7050 --tls true --cafile $ORDERER_CA
For updating any application channel, just replace the step 7 by updating the application channel configuration block's Application section to add the new orderer organization's MSP there.
Hope this helps!

Answer 2

I followed above sequence of steps to add new Orderer Organization into existing network, but channel (i.e updating config change to orderer channel) update throw error like below,
2020-09-29 00:53:49.794 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'e2e-orderer-syschan': error authorizing update: error validating DeltaSet: policy for [Value] /Channel/OrdererAddresses not satisfied: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Admins' sub-policies to be satisfied
I modified-json blocks by using below setup calls (tried mix/match combination of this below json change steps)
jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"'${KL_NEW_ORDERER_NAME}'":.[1]}}}}}' config.json ${KL_NEW_ORDERER_NAME}.json > modified-config.json
jq -s '.[0] * {"channel_group":{"groups":{"Orderer":{"groups": {"'${KL_NEW_ORDERER_NAME}'":.[1]}}}}}' modified-config.json ${KL_NEW_ORDERER_NAME}.json > modified-config1.json
jq -s '.[0] * {"channel_group":{"groups":{"Consortiums":{"groups":{"'${KL_CONSORTIUM_NAME}'":{"groups": {"Orderermk01MSP":.[1]}}}}}}}' modified-config1.json ${KL_NEW_ORDERER_NAME}.json > modified-config2.json
LENGTH=$(jq '.channel_group.values.OrdererAddresses.value.addresses | length' modified-config2.json)
jq '.channel_group.values.OrdererAddresses.value.addresses['${LENGTH}'] |= "'${KL_NEW_ORDERER_URL}'"' modified-config2.json > modified-config3.json
cert=`base64 /hl-material/mk01-orderer/crypto-config/ordererOrganizations/${KL_DOMAIN}/orderers/orderer.mk01.${KL_DOMAIN}/tls/server.crt | sed ':a;N;$!ba;s/\n//g'`
cat modified-config3.json | jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters += [{"client_tls_cert": "'$cert'", "host": "raft0.mk01.'${KL_DOMAIN}'", "port": 32050, "server_tls_cert": "'$cert'"}] ' > modified-config4.json
My network setup based on HLF 2.2 LTS with 5 raft nodes under K8s cluster.
I have successful setup with multi channel, multiple peer organization on 2.2 LTS (know how to get this in dynamic way without bringdown any network). But now looking for scaling orderer organization into multiple cluster/orgs dynamically. Is any tip or update needed on above steps ? Thanks again
Mariya