TLS and MSP certificates of Orderers and Peers in my network were expired. So, I renewed them and change TLS, MSP folder to new one. After restart all my Orderers and Peers, Orderers keep SERVICE_UNAVAILABLE: rejected by Consenter: channel identitych is not serviced by me error from all orderers. Querying the chaincode to each peer works well, so it seems like an orderer's problem.
When my orderer started, the orderer logged a warning like below:
[orderer.consensus.etcdraft] detectSelfID -> WARN 015 Could not find -----BEGIN CERTIFICATE-----
MIIC8DCCApagAwIBAgIUOhF5HhLQW...
-----END CERTIFICATE-----
among [-----BEGIN CERTIFICATE-----
MIIC5TCCAougAwIBAgIUApCtD3xfo9JLFyPd...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC5DCCAougAwIBAgIUQ/ztvrcb3Z6LB8...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC5DCCAougAwIBAgIUSCeIphcSIWrs...
-----END CERTIFICATE-----
]
The location of this warning and channel ... is not serviced by me error are same, so I guess these are same problem.
The problem looks like that my new TLS signed certificate is not recognized from blocks' meta information.
Thanks for some comments from the community, I found below link and it seems very close to my problem.
https://jira.hyperledger.org/browse/FAB-16953
However, even I set TLSHandshakeTimeShift to 20s and restart orderers one by one, still the problem occurs. My environment setting is like below:
docker container run -d --name $NODE \
--log-driver json-file --log-opt max-size=1g --log-opt max-file=1 \
-e FABRIC_LOGGING_SPEC=INFO \
-e ORDERER_GENERAL_LISTENADDRESS=0.0.0.0 \
-e ORDERER_GENERAL_GENESISMETHOD=file \
-e ORDERER_GENERAL_GENESISFILE=/artifacts/genesis.block \
-e ORDERER_GENERAL_LOCALMSPID=$MSP \
-e ORDERER_GENERAL_LOCALMSPDIR=/artifacts/msp \
-e ORDERER_GENERAL_TLS_ENABLED=true \
-e ORDERER_GENERAL_TLS_PRIVATEKEY=/artifacts/tls/keystore/key.pem \
-e ORDERER_GENERAL_TLS_CERTIFICATE=/artifacts/tls/signcerts/cert.pem \
-e "ORDERER_GENERAL_TLS_ROOTCAS=[/artifacts/tls/tlscacerts/ca-cert.pem]" \
-e ORDERER_KAFKA_TOPIC_REPLICATIONFACTOR=1 \
-e ORDERER_KAFKA_VERBOSE=true \
-e ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/artifacts/tls/keystore/key.pem \
-e ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/artifacts/tls/signcerts/cert.pem \
-e "ORDERER_GENERAL_CLUSTER_ROOTCAS=[/artifacts/tls/tlscacerts/ca-cert.pem]" \
-e ORDERER_OPERATIONS_LISTENADDRESS=$NODE:8443 \
-e ORDERER_GENERAL_CLUSTER_TLSHANDSHAKETIMESHIFT=48h \
-w="/opt/gopath/src/github.com/hyperledger/fabric" \
-v "$(pwd)"/artifacts:/artifacts \
-v "$(pwd)"/blocks:/var/hyperledger/production/orderer/ \
--network $NETWORK \
hyperledger/fabric-orderer:$VERSION orderer
Did I do some wrong to set TLSHANDSHAKETIMESHIFT value? Still I could not find any solution for my problem. Any kind of helps are welcome. Thanks!
Related
I have been working on rotating Orderer node certs from cryptogen to Fabric CA.
I have been following the official documentation here-
https://hyperledger-fabric.readthedocs.io/en/release-1.4/raft_configuration.html
Below are the steps that i have tried till now -
generate new certs with *fabric CA server for each of the nodes.
update the configtx.yaml msp path with the new msp path created for orderer.
create new Orderer.json based on the new the configtx.yaml
update the system channel with new certs using jq.
Currently i am getting an issue while i try to update the system channel 'testchainid'. The error seen is -
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'testchainid': error authorizing update: error validating DeltaSet: invalid mod_policy for element [Group] /Channel/Application: mod_policy not set
Sharing the steps performed on the peer cli:
peer channel fetch config config_block.pb -o orderer.org.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA
configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json
# Find the diff between current config and new config, then output a new json file
jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"OrdererMSP":.[1]}}}}}' config.json Orderer.json > modified_config.json
# add fabric ca tls certs
jq 'del(.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters[])' modified_config.json > modified_config_1.json
#new ca cert for orderer to update system channel
cert1=$(base64 /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/org.com/orderer/tls-msp/signcerts/cert.pem | sed ':a;N;$!ba;s/\n//g')
#new ca cert for orderer1 to update in system channel
cert2=$(base64 /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/org.com/orderer1/tls-msp/signcerts/cert.pem | sed ':a;N;$!ba;s/\n//g')
#new ca cert for orderer2 to update in system channel
cert3=$(base64 /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/org.com/orderer2/tls-msp/signcerts/cert.pem | sed ':a;N;$!ba;s/\n//g')
#update the new certs on to channel
jq '.channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters += [{"client_tls_cert": "'$cert1'", "host": "orderer.org.com", "port": 7050, "server_tls_cert": "'$cert1'"}] | .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters += [{"client_tls_cert": "'$cert2'", "host": "orderer1.org.com", "port": 7050, "server_tls_cert": "'$cert2'"}] | .channel_group.groups.Orderer.values.ConsensusType.value.metadata.consenters += [{"client_tls_cert": "'$cert3'", "host": "orderer2.org.com", "port": 7050, "server_tls_cert": "'$cert3'"}] ' modified_config_1.json > modified_config_2.json
# Converts config.json into config.pb
configtxlator proto_encode --input config.json --type common.Config --output config.pb
# Converts modified_config.pb into modified_config.json
configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb
# Converts modified_config.pb into modified_config.json
configtxlator proto_encode --input modified_config_1.json --type common.Config --output modified_config_1.pb
# Converts modified_config.pb into modified_config.json
configtxlator proto_encode --input modified_config_2.json --type common.Config --output modified_config_2.pb
# Calculates the delta between modified_config.json config.json then output
configtxlator compute_update --channel_id $CHANNEL_NAME --original config.pb --updated modified_config_2.pb --output Orderer_ca_update.pb
configtxlator proto_decode --input Orderer_ca_update.pb --type common.ConfigUpdate | jq . > Orderer_ca_update.json
echo '{"payload":{"header":{"channel_header":{"channel_id":"testchainid", "type":2}},"data":{"config_update":'"$(cat Orderer_ca_update.json)"'}}}' | jq . > Orderer_ca_update_in_envelope.json
configtxlator proto_encode --input Orderer_ca_update_in_envelope.json --type common.Envelope --output Orderer_ca_update_in_envelope.pb
peer channel signconfigtx -f Orderer_ca_update_in_envelope.pb
peer channel update -f Orderer_ca_update_in_envelope.pb -c $CHANNEL_NAME -o orderer.org.com:7050 --tls --cafile $ORDERER_CA
Any help on how to fix the issue would be much appreciated.
There are 3 orgs: Org1, Org2 and Org3.
Org1 and Org2 created a consortium named SampleConsortium
Now, I want to add Org3 to the SampleConsortium. There is no channel created yet.
In the documentation, there is a tutorial to add an org to an existing channel. I want org to join the consortium not the channel.
How can I do this? Please add resources that will be very helpful.
Thanks!
I have written the script to add/delete org into consortium and add/delete org into channel.
## Make sure network is up
## Make sure certificates are generated using cryptogen
## Make sure you are executing this script in cli
## docker exec -it cli bash
export ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
export CHANNEL_NAME=byfn-sys-channel
CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/users/Admin#example.com/msp
CORE_PEER_ADDRESS=orderer.example.com:7050
CORE_PEER_LOCALMSPID=OrdererMSP
CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt
peer channel fetch config config_block.pb -c $CHANNEL_NAME -o orderer.example.com:7050 --tls --cafile $ORDERER_CA
configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json
#### Add Org into Consortium ######
jq -s '.[0] * {"channel_group":{"groups":{"Consortiums":{"groups": {"SampleConsortium": {"groups": {"Org3MSP":.[1]}}}}}}}' config.json ./channel-artifacts/org3.json > modified_config.json
#### Delete Org from Consortium ######
cat config.json | jq "del(.channel_group.groups.Consortiums.groups.SampleConsortium.groups.Org3MSP)" > modified_config.json
#### Add Organization to channel #####
jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"Org3MSP":.[1]}}}}}' config.json ./channel-artifacts/org3.json > modified_config.json
#### Delete Oraganization from channel ####
jq 'del(.channel_group.groups.Application.groups.Org3MSP)' config.json > modified_config.json
configtxlator proto_encode --input config.json --type common.Config --output config.pb
configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb
configtxlator compute_update --channel_id $CHANNEL_NAME --original config.pb --updated modified_config.pb --output org3_update.pb
configtxlator proto_decode --input org3_update.pb --type common.ConfigUpdate | jq . > org3_update.json
echo '{"payload":{"header":{"channel_header":{"channel_id":"byfn-sys-channel", "type":2}},"data":{"config_update":'$(cat org3_update.json)'}}}' | jq . > org3_update_in_envelope.json
configtxlator proto_encode --input org3_update_in_envelope.json --type common.Envelope --output org3_update_in_envelope.pb
peer channel signconfigtx -f org3_update_in_envelope.pb
peer channel update -f org3_update_in_envelope.pb -c $CHANNEL_NAME -o orderer.example.com:7050 --tls --cafile $ORDERER_CA
Thanks to #alexander for this tutorial link.
The tutorial is written by Allison Irvin
Updating the Consortium Definition in Hyperledger Fabric
The answer to this question:
We have to update the system genesis block to add a new organization
in the consortium. Then only the new organization can create a channel.
Tutorial's Intro:
The creation of channels is controlled by members of Consortia, which consist of one or more organizations that are defined at the network level. As Fabric networks evolve and grow, it is expected that the list of organizations requiring the ability to create channels will change. Therefore, we need the ability to add or modify Consortia definitions without interrupting any of the network components.
We are getting an error while trying a peer node to join a channel. Both members have separate peer nodes and separate EC2 machines acting as clients for the network. Member1 has created a channel and joined its peer node to the channel. Member2 has pulled the genesis block from their client but when trying to join the channel, we get the error:
Error: genesis block file not found open ourchannel.block: no such file or directory
despite seeing ourchannel.block in the directory we run the command. The command we're using to join the channel is:
docker exec -e "CORE_PEER_TLS_ENABLED=true" \
-e "CORE_PEER_TLS_ROOTCERT_FILE=/opt/home/managedblockchain-tls-chain.pem" \
-e "CORE_PEER_ADDRESS=$PEER" \
-e "CORE_PEER_LOCALMSPID=$MSP" \
-e "CORE_PEER_MSPCONFIGPATH=$MSP_PATH" \
cli peer channel join -b ourchannel.block \
-o $ORDERER --cafile /opt/home/managedblockchain-tls-chain.pem --tls
You only have the genesis block (the file with .block extension) if you create a channel in a separate container, which is cli container in your case. Please check by using ls command inside the cli container.
A simple command to jump into cli container:
docker exec -it cli bash
Just in case you can't find genesis block, you can get it from the orderer (remember to modify the path to the cafile fit your project):
peer channel fetch 0 ourchannel.block -o orderer.example.com:7050 -c ourchannel --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
After this, you can join the channel with peer channel join command.
Simple issue, we had to specify the absolute path to the block file.
Instead of
cli peer channel join -b ourchannel.block
we used
cli peer channel join -b /opt/home/ourchannel.block \
and it worked.
I have my own network on and try to use the chaincode_example02 chaincode to test it out, following the first-network example. When it comes to chaincodeInvoke in my shellscipt, the console shows
Sending invoke transaction on peer0.bank peer0.caseManager...
--tlsRootCertFiles
--peerAddresses peer0.bank.snts.com:7051 --tlsRootCertFiles
--tlsRootCertFiles
--peerAddresses peer0.bank.snts.com:7051 --tlsRootCertFiles --peerAddresses peer0.caseManager.snts.com:7051 --tlsRootCertFiles
+ peer chaincode invoke -o orderer.snts.com:7050 --tls true --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/snts.com/orderers/orderer.snts.com/msp/tlscacerts/tlsca.snts.com-cert.pem -C sntschannel -n mycc --peerAddresses peer0.bank.snts.com:7051 --tlsRootCertFiles --peerAddresses peer0.caseManager.snts.com:7051 --tlsRootCertFiles -c '{"Args":["invoke","a","b","10"]}'
+ res=1
+ set +x
2019-01-01 16:38:40.670 UTC [chaincodeCmd] validatePeerConnectionParameters -> WARN 001 received more TLS root cert files (2) than peer addresses (1)
Error: error validating peer connection parameters: number of peer addresses (1) does not match the number of TLS root cert files (2)
!!!!!!!!!!!!!!! Invoke execution on peer0.bank peer0.caseManager failed !!!!!!!!!!!!!!!!
========= ERROR !!! FAILED to execute End-2-End Scenario ===========
bank has 2 peers and caseManager has 3. What can lead to this problem? In first-network example, both orgs has 2 peers, and it works perfectly. Can someone help? Thank you.
In your request, you have used --tlsRootCertFiles flag but did not specify any path for it? are you setting this externally? if not try specifying it like
--tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt
The problem is with the way TLSINFO is fetched.
Below command is used in utils.sh file where we are mentioning the TLSCert file. You need to modify it according to your requirement.
TLSINFO=$(eval echo "--tlsRootCertFiles \$PEER$1_Org$2_CA")
You need to modify the global variables so that they are generic.
It is necessary to change, in the setOrderererGlobals() method of the file utils.sh :
setGlobals() {
PEER=$1
ORG=$2
### ADD THESES LINES !!! ###
CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org$ORG.supplychainnet.ch/peers/peer$PEER.org$ORG.supplychainnet.ch/tls/ca.crt
CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org$ORG.supplychainnet.ch/users/Admin\#org$ORG.supplychainnet.ch/msp
### ADD THESES LINES !!! ###
if [ $ORG -eq 1 ]; then
CORE_PEER_LOCALMSPID="Org1MSP"
### COMMENT THESES LINES !!! ###
#CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG1_CA
#CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.supplychainnet.ch/users/Admin\#org1.supplychainnet.ch/msp
### COMMENT THESES LINES !!! ###
if [ $PEER -eq 0 ]; then
CORE_PEER_ADDRESS=peer0.org1.supplychainnet.ch:7051
else
CORE_PEER_ADDRESS=peer1.org1.supplychainnet.ch:8051
fi
else
echo "================== ERROR !!! ORG Unknown =================="
fi
As a result, the environment paths for each peer will automatically be updated with the numbers $PEER and $ORG instead of retrieving the global PEERx_ORGy_CA paths defined at the top of the file.
pass ca.crt file path for respective peer in --tlsRootCertFiles
Try This and it works
docker exec -it cli bash
peer chaincode invoke -o orderer.example.com:7050 --tls true --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C testchannel -n mycc --peerAddresses peer0.org1.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt --peerAddresses peer0.org2.example.com:7051 --tlsRootCertFiles /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt -c '{"Args":["invoke","a","b","20"]}'
I am trying to run the balance-transfer example in the link: https://github.com/hyperledger/fabric-samples/tree/master/balance-transfer#sample-rest-apis-request
I want to ask that why I can still invoke the chaincode and run successfully when the authorization: Bearer is org1 but the invoke is done on Org2(["peer1.org2.example.com")? Why this will not cause authorization error?
In addition, why can I query the chaincode use Org1 certificate on Org2. Are there any setup issues related to the CA?
echo "POST invoke chaincode on peers of Org1"
echo
TRX_ID=$(curl -s -X POST \
http://localhost:4000/channels/mychannel/chaincodes/mycc \
-H "authorization: Bearer $ORG1_TOKEN" \
-H "content-type: application/json" \
-d '{
"peers": ["peer1.org2.example.com"],
"fcn":"move",
"args":["a","b","10"]
}')
echo "Transacton ID is $TRX_ID"
echo
echo
Thank you very much.
This is because the peers in JSON string refers to the endorsing peer. It can be left empty. The authorization is based on the token and the transaction will be completed on org1 peers based on the log.