I have a hyperledger fabric network v2.2.1 deployed in a kubernetes cluster. Initially, the network components are deployed in such a way that they all can communicate within the cluster and were not exposed outside the cluster.
Based on a requirement, later, we have made the peer, orderer and CA services to be accessible from outside the cluster by using haproxy ingress. Created ingress for all the services and updated the environment variables and tls certs with the new ingress endpoints. After such migration, when I try to fetch the latest configuration block from the system channel, I am unable to fetch and getting "SERVICE UNAVAILABLE" error.
Below are the environment variables I used to set to orderer,
export FABRIC_LOGGING_SPEC=INFO
export CORE_PEER_ID=ordererCli
export CORE_PEER_TLS_ENABLED=true
export ORDERER_CA=/go/scripts/organizations/ordererOrganizations/ordererOrg/orderers /orderer1/tls/tlsintermediatecerts/tls-orderer-ica-svc-7054-orderer-ica.pem
export CORE_PEER_TLS_CERT_FILE=/go/scripts/organizations/ordererOrganizations/ordererOrg/orderers/orderer1/tls/server.crt
export CORE_PEER_TLS_KEY_FILE=/go/scripts/organizations/ordererOrganizations/ordererOrg/orderers/orderer1/tls/server.key
export CORE_PEER_LOCALMSPID=OrdererMSP
export CORE_PEER_TLS_ROOTCERT_FILE=/go/scripts/organizations/ordererOrganizations/ordererOrg/orderers/orderer1/tls/ica.crt
export CORE_PEER_MSPCONFIGPATH=/go/scripts/organizations/ordererOrganizations/ordererOrg/users/Admin#orderer/msp
export CORE_PEER_ADDRESS=orderer1-svc.hyperledger.svc.cluster.local
Command used to fetch the block,
peer channel fetch config config_block.pb -o orderer1.example.com:443 --ordererTLSHostnameOverride orderer1.example.com -c syschannel --tls --cafile $ORDERER_CA
Error I am getting,
[channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
[cli.common] readBlock -> INFO 002 Expect block, but got status: &{SERVICE_UNAVAILABLE}
Can anyone please guide me to overcome the issue? Thanks in Advance!
Related
I have set up a network with raft ordering service (5 orderers), 2 orgs and one peer each, the TLS and client authentication are both enabled. I have created the channel and joined it successfully, but I am getting TLS handshake failed error when updating the anchor peer from cli to orderer, and I am not able to create a second channel for the same error. Besides this TLS error why I was able to create the first channel I am very confused, surely incorrect configs causing that.
CLI Commands:
export ORDERER_CA=crypto/ordererOrganizations/example.com/tlsca/ca-chain.pem
export CLIENT_KEY_FILE=crypto/peerOrganizations/org1.example.com/users/Admin#org1.example.com/tls/client.key
export CLIENT_CERT_FILE=crypto/peerOrganizations/org1.example.com/users/Admin#org1.example.com/tls/client.pem
*Worked: peer channel create -o orderer0.example.com:7050 -c mychannel -f ./channel-artifacts/channel.tx --tls --cafile $ORDERER_CA --clientauth --keyfile $CLIENT_KEY_FILE --certfile $CLIENT_CERT_FILE
*Worked: peer channel join -b muchannel.block
*FAILED: peer channel update -o orderer0.example.com:7050 -c mychannel -f ./channel-artifacts/anchors_channel.tx --tls --cafile $ORDERER_CA --clientauth --keyfile $CLIENT_KEY_FILE --certfile $CLIENT_CERT_FILE
CLI yaml config:
- CORE_PEER_TLS_ENABLED=true
- CORE_PEER_TLS_CERT_FILE=crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/server.pem
- CORE_PEER_TLS_KEY_FILE=crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/server.key
- CORE_PEER_TLS_ROOTCERT_FILE=crypto/peerOrganizations/org1.example.com/tlsca/ca-chain.pem
- CORE_PEER_TLS_CLIENTAUTHREQUIRED=true
- CORE_PEER_TLS_CLIENTCERT_FILE=crypto/peerOrganizations/org1.example.com/users/Admin#org1.example.com/tls/client.pem
- CORE_PEER_TLS_CLIENTKEY_FILE=crypto/peerOrganizations/org1.example.com/users/Admin#org1.example.com/tls/client.key
Orderer yaml config:
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=crypto/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=crypto/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.pem
- ORDERER_GENERAL_TLS_ROOTCAS=[crypto/ordererOrganizations/example.com/tls/ca-chain.pem]
- ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED=true
- ORDERER_GENERAL_TLS_CLIENTROOTCAS=[crypto/ordererOrganizations/example.com/tlsca/ca-chain.pem,crypto/peerOrganizations/org1.example.com/tlsca/ca-chain.pem,crypto/peerOrganizations/org2.example.com/tlsca/ca-chain.pem]
Try running the command with following environment in your CLI container:
export CORE_PEER_ADDRESS=orderer.example.com:7050
export CORE_PEER_LOCALMSPID=OrdererMSP
export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/users/Admin#example.com/msp
export CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt
export ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt
export CHANNEL_NAME=mychannel
This is because issuing channel update transaction requires updating the channel configuration file, which according to your present policies, must be signed by an Orderer Organization admin.
I installed the chaincode on the network but when I'm trying to instantiate,
I'm getting :
"Error: error getting broadcast client: orderer client failed to connect to 127.0.0.1:7050: failed to create new connection: context deadline exceeded"
Also, I tried to add TLS but then I get the following error:
Error: error getting broadcast client: failed to load config for OrdererClient: unable to load orderer.tls.rootcert.file:
And, the orderer logs are:
My orderer TLS is enabled now:
I'll be glad if someone can help or guide me atleast, I'm new to HyperledgerFabric.Thank you.
Adapt to your case:
peer chaincode instantiate -n mychannel -v 1.0 -C petshop -c '{"Args":[""]}' -o 127.0.0.1:7050 --tls --cafile /path/to/orderer/tls/ca.pem
Check the "orderer" logs precisely at the time you are instantiating the chaincode (first without TLS and then with TLS). You can see that time from your command-line when you are running that command.
Update your question with that section of orderer logs.
Also, if I need to guess from your current orderer nodes, you might not have enabled TLS for the orderer container.
I have a problem with instantiating chaincode in Hyperledger fabric Network with comm:
peer chaincode instantiate -o orderer.ex.com:7050 -C roaming -n chaincode -v 1.1 -c '{"Args":[]}' -P "AND ('ORG1MSP.member')" --tls --cafile /etc/hyperledger/fabric/config/ca.crt
The peer is joined the channel and chaincode is installed on it.
The log of orderer is:
ERRO 029 TLS handshake failed with error tls: oversized record received with length 64774 server=Orderer
WARN 02a Error reading from 172.16.0.81:34562: rpc error: code = Canceled desc = context canceled
INFO 02b streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=172.16.0.81:34562 error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call
WARN 02c Error reading from 172.16.0.81:35318: rpc error: code = Canceled desc = context canceled
In the log of the peer - nothing interesting.
The error looks like the client is trying to communicate speak HTTP/GRPC to a server expecting HTTPS/GRPCS (or viceversa)
If your fabric network endpoints are not encrypted, remove the --tls and --cafile flags.
Otherwise, make sure to use an encrypted endpoint for the orderer (e.g. orderer.ex.com:443 if it's exposed via HTTPS with an ingress controller) and the peer as well (CORE_PEER_ADDRESS)
I have also found when enabling TLS that the following environment variables are needed in addition to the --tls and --cafile above:
CORE_PEER_TLS_ROOTCERT_FILE=/path/to/ca/pem (e.g. `letsencryptauthorityx3.pem`)
CORE_PEER_TLS_ENABLED=true
I am working on prototyping a game using hyperledger-composer. I need to decrease the block time out however the instructions provided in the fabric-dev-servers do not work.
I followed the howtobuild.txt found in ~/fabric-dev-servers/fabric-scripts/hlfv12/composer however installing the network using composer network start --networkName .... the network fails to start with the error:
Error: Error trying to start business network. Error: Failed to connect to any peer event hubs. It is required that at least 1 event hub has been connected to receive the commit event
the versions i am uising are composer#20 and fabric docker images with the tag 1.2.1 (as downloaded by) ./startFabric.sh
The steps I have taken are exactly as follows:
cd ~/fabric-dev-servers/fabric-scripts/hlfv12/composer
nano configtx.yaml
inside this file I
Move the profile block to the bottom of the ymal (to stop the weird error)
change BatchTimeout to 200ms
changed MaxMessageCount to 1
after saving the file I do
rm -r crypto-config
cryptogen generate --config=./crypto-config.yaml
get the new key and update docker-composer.yml and docker-compose-dev.yml
$(ls -1 crypto-config/peerOrganizations/org1.example.com/ca/*_sk`
get key from keystore
ls -1 crypto-config/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp/keystore/*_sk
update peerAdminCard sell script with the new key
nano ../createPeerAdminCard.sh
Generate
configtxgen -profile ComposerChannel -outputCreateChannelTx ./composer-channel.tx -channelID composerchannel`
configtxgen -profile ComposerOrdererGenesis -outputBlock ./composer-genesis.block
create peer admin card
cd ../ && ./createPeerAdminCard.sh
At this point it starts downloading the fabric docker images, which all run successfully
I then create my BNA file and try to start the network
cd /my/buisness/network
composer archive create -t dir -n .
composer network install --card PeerAdmin#hlfv1 --archiveFile my-game#0.0.1.bna
composer network start --networkName my-game --networkVersion 0.0.1 --networkAdmin admin --networkAdminEnrollSecret adminpw --card PeerAdmin#hlfv1 --file networkadmin.card
which gives the error
Error: Error trying to start business network. Error: Failed to connect to any peer event hubs. It is required that at least 1 event hub has been connected to receive the commit event
The Debugging steps i have taken are as follows
After viewing the peer logs i can see delivering blocks to the orderer fails because composerchannel does not exist, so i manually try to create and join the channel manually
docker exec peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c composerchannel -f /etc/hyperledger/configtx/composer-channel.tx
which is a success
2019-03-23 00:07:38.131 UTC [cli/common] readBlock -> INFO 05e Received block: 0
I then try to join the peer to the channel
docker exec peer0.org1.example.com peer channel join -o orderer.example.com:7050 -b composerchannel.block --tls --cafile /etc/hyperledger/msp/users/Admin#org1.example.com/tls/ca.crt --keyfile /etc/hyperledger/msp/users/Admin#org1.example.com/tls/client.key --certfile /etc/hyperledger/msp/users/Admin#org1.example.com/tls/client.crt
which gives the error
proposal failed (err: bad proposal response 500: access denied for [JoinChain][composerchannel]: [Failed verifying that proposal's creator satisfies local MSP principal during channelless check policy with policy [Admins]: [This identity is not an admin]])
There are a lot of different 'Admin' Pems inside the /etc/hyperledger/msp and /etc/hyperledger/tls so I'm pretty much trying all combinations
I'm at a loss, I've been on google for hours trying to find anyone with the same issue but have come up short. Any help will be appreciated.
UPDATE
Turns out explicitly defining the certificates where not needed. running this command allowed me to join the channel.
docker exec peer0.org1.example.com peer channel join -o orderer.example.com:7050 -b composerchannel.block --clientauth --tls
at this point all the docker containers were running and the channel was working. i manage to be able to start the network and everything ran fine (i could request the chaincode, ledger was updating fine etc). However it still seems to take 2 seconds to process a transaction. Again i did some debugging to make sure the config on the order was correct.
fetch config block from channel on orderer
docker exec peer0.org1.example.com peer channel fetch config config_block.pb -o http_s_://orderer.example.com:7050 -c composerchannel --tls --cafile /etc/hyperledger/peer/msp/tlscacerts/tlsca.org1.example.com-cert.pem
copy out of container to local machine
docker cp 5eeaf8c650f8:/root/config_block.pb config_block.pb
convert from proto-buf to json (using configtxlator binary found in fabric-samples)
configtxlator docker exec peer0.org1.example.com proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json
After getting the config of the channel i could see that the BatchTimeout was 200ms and BatchAmount was 1, yet i still have the 2 seconds timeout per transaction.
While working with Hyperledger composer now you have to keep in mind that it is obselete. I have followed these same steps many times in the past and built custom networks. I think your issue could be of version mismatch. Try downgrading composer to 0.19.
Additionally, did you try doing a docker ps to make sure that all the required containers are running?
Also, before doing the configtxgen command there's a simple command
export FABRIC_CFG_PATH=$PWD
Did you do this?
The official hyperledger fabric v1.0.0 gives a simple demo by using docker. Here is the link.
What i am doing is to avoid docker and directly run them on the machine. Thanks to the answer from my previous question, I have successfully launch 1 solo orderer and two peers, each from one org.
This is part of orderer's config orderer.yaml and i am sure the tls-related path is set correctly.
General:
# Ledger Type: The ledger type to provide to the orderer.
# Two non-production ledger types are provided for test purposes only:
# - ram: An in-memory ledger whose contents are lost on restart.
# - json: A simple file ledger that writes blocks to disk in JSON format.
# Only one production ledger type is provided:
# - file: A production file-based ledger.
LedgerType: file
# Listen address: The IP on which to bind to listen.
ListenAddress: 127.0.0.1
# Listen port: The port on which to bind to listen.
ListenPort: 7040
# TLS: TLS settings for the GRPC server.
TLS:
Enabled: true
PrivateKey: ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.key
Certificate: ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt
RootCAs:
- ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt
ClientAuthEnabled: false
ClientRootCAs:
# Log Level: The level at which to log. This accepts logging specifications
# per: fabric/docs/Setup/logging-control.md
LogLevel: debug
However, when I want to create the channel using command as follows:
export FABRIC_CFG_PATH=$PWD
export CHANNEL_NAME=my_channel
export CORE_PEER_LOCALMSPID="Org1MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
export CORE_PEER_MSPCONFIGPATH=crypto-config/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp
export CORE_PEER_ADDRESS=127.0.0.1:7001
peer channel create -o 127.0.0.1:7040 -c $CHANNEL_NAME -f channel-artifacts/channel.tx --tls true --cafile $ORDERER_CA >&log.txt
The log reports the error saying as follows:
Error: Error connecting due to rpc error: code = Internal desc = connection error: desc = "transport: authentication handshake failed: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs"
I think it is related to a tls configuration problem. I am not very familiar about it. Can any one help me to solve this simple problem and give me a simple explanation?
The error is caused due to hostname verification failing at the TLS layer.
The TLS certificates don't have IP Subject Alternative Names (SANs) - they only have DNS-based SANs.
The easiest thing to do is to just add host entries into your /etc/hosts file (assuming you are running on Linux or macOS). Assuming that you are running everything on the same machine, then adding the following line to your /etc/hosts should resolve your issue:
127.0.0.1 localhost orderer peer0.org1.example.com peer0.org2.example.com
And then make sure to use the hostnames rather than IP addresses:
export FABRIC_CFG_PATH=$PWD
export CHANNEL_NAME=my_channel
export CORE_PEER_LOCALMSPID="Org1MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
export CORE_PEER_MSPCONFIGPATH=crypto-config/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp
export CORE_PEER_ADDRESS=peer0.org1.example.com:7001
peer channel create -o orderer:7040 -c $CHANNEL_NAME -f channel-artifacts/channel.tx --tls true --cafile $ORDERER_CA >&log.txt