My orderers and peers aren't able to communicate . They are giving bad certificate errorr. I don't want to loose any data .
2020-12-18 12:47:46.820 UTC [core.comm] ServerHandshake -> ERRO 990 TLS handshake failed with error remote error: tls: bad certificate server=PeerServer remoteaddress=172.29.0.7:58636
For peers, it's very straightforward:
issue new TLS certificates from the same CA which issued their current certs
replace the existing certificates (or update core.yaml to point to the new certificates if using different paths/filenames)
restart the peer(s)
For orderers using Raft, it's a bit trickier. Assuming you are on the latest 1.4.x version, what you need to do for orderers is use their existing private keys to generate new CSRs and then have the CA issue a new public certificate. You will then replace / update orderer.yaml the public certificate and restart the orderers one at a time.
Related
I created a test network which has 2 orgs (each with one peer), 1 orderer in kubernetes using BAF. Orgs are joined in the channel called testchannel. Orderer msp, peer msp and tls certs expired within 1 day. Before it expired, I renewed all the certs using dcm tool and kept it in my local. First I updated the orderer tls cert in system channel and in application channel from orderer cli, by fetching the channel config, decoded, updated renewed orderer tls certs under consenters, encoded and updated the channel config using peer channel update command. I received a successfully submitted message.
Later replaced orderer msp, peer msp and tls certs in the vault and restarted all the services. When I checked the orderer logs, it didn't show any expiry error.
Now my network's previous certificate expired. Now it is using renewed certs. I am able to invoke and query transactions.
Now I want to add a new org called org3 to the existing (testchannel).
I created a new org (org3). When I tried to join that org to the channel, during peer channel update it failed. It shows the following error,
Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'testchannel': error authorizing update: error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 2 of the 'Admins' sub-policies to be satisfied
This error is because of wrong admin certs. Then only I found that during certificate renewal, I updated only orderer tls certs. But the channel config also contains each org's admin certs and cacerts. Now the channel config contains expired certs. But the orgs and vault contain renewed certs.
This is the cause for the above mentioned error.
Can anyone suggest a way to resolve this? How can we update org's admin certificate in the channel config which has expired certificate?
Thanks in advance......
To simulate the MSP and certificate expiry of nodes [orderer & peer], I updated the expiry time to 1h (the default is 8760h which is 1 year in hours) in the fabric-ca-server-config.yaml file. I am not sure, if this is the correct way to do it?
Please feel free to correct me
Hyperledger Fabric v2.3 and Fabric CA v.1.5.0
fabric-ca-server-config.yaml: Org1 - the Org1 Fabric CA server is using this config
signing:
default:
usage:
- digital signature
expiry: 1h
profiles:
ca:
usage:
- cert sign
- crl sign
expiry: 43800h
caconstraint:
isca: true
maxpathlen: 0
tls:
usage:
- signing
- key encipherment
- server auth
- client auth
- key agreement
expiry: 1h
Right before the expiry time, I successfully (did not get any error) reenrolled the MSP and TLS certs of all using the fabric-ca-client reenroll command along with --csr.keyrequest.reusekey. More info on the command usage can be found here
I checked the logs in all the nodes and the peer nodes were spitting logs as:
2021-07-27 13:51:11.747 UTC [core.comm] ServerHandshake -> ERRO 0e9 Server TLS handshake failed in 1.807206ms with error remote error: tls: bad certificate server=PeerServer remoteaddress=172.26.0.9:49338
Also, I was not able to transact on the network, which is obvious because of the above error. I believe this affects the MSP and TLS certs generated for the organization admin and user as well.
Could anyone be kind enough to shed some lights on the
fabric-ca-server-config.yaml and how to reenroll or re-issue the
certificates so the fabric-network can function normally?
After the certificates are expired already, the reenroll command is giving error - not a valid certificate. The certificate which is referred in the --tls.certfiles flag is the root CA tls certificate. The root tls certificate of the CA should not expire??
In the Hyperledger Fabric official website, it states to use the fabric-ca-client reenroll if a certificate is about to expire. I tried to reenoll the certificates but an x.509 error occurred:
[grpc] createTransport -> DEBU bab grpc: addrConn.createTransport failed to connect to {orderer:7050 0 }. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
I have used the same CA to reenroll the certificate, I don't know why "unknown authority" error shown.
It would be grateful if anyone can give me a full steps for replacing the admin certificate for the Hyperledger Fabric.
You can use akc-dcm-cli. It support renew user of Hyperledger Fabric by running command:
dcm certificate renew --parent-cert=../data/intermediate-ca/signcerts/ica-cert.pem
--parent-private-key=../data/intermediate-ca/keystore/ica-key.pem
--old-cert=../data/peer/output/peer-cert.pem
--old-private-key=../data/peer/keystore/peer-key.pem
--output=../data/peer/output/renew-peer-cert.pem
--days=365
parent-cert and parent-private-key are key-pair of CA
old-cert and old-private-key are key-pair of expired user
output is the path of new cert
After renewing the cert, you replace the expired cert of the user with the new cert. Keep current private key of user' cert
I noticed that HLf admincerts and tls certs, by default, have an expiration date of 1 year. In my case, they have expired and my orderer and peer nodes are giving me a bad certificate error.
Is there any way to do certificate rotation? I don't want to loose any data as the network is in a production enviroment. I am using hyperledger fabric 1.4 version with the raft ordering service.
You can change the default expiry time of an x509 certificate by changing its duration in the fabric-ca-server configuration file. You'll find this file inside CA's docker container.
Reference to the fabric-ca-server configuration file can be found here.
Change the value of expiry from 8760h to xxxx in the signing section of the file. signing section looks like this:
signing:
default:
usage:
- digital signature
expiry: 8760h
profiles:
ca:
usage:
- cert sign
- crl sign
expiry: 43800h
caconstraint:
isca: true
maxpathlen: 0
tls:
usage:
- signing
- key encipherment
- server auth
- client auth
- key agreement
expiry: 8760h
After changing the value of expiry, you'll have to restart your CA container for the changes to take effect.
Once your container is restarted, re-enroll your admin to create a new x509 certificate with the updated expiry duration. You can read about reenrolling an identity from here.
If you want to avoid such scenarios in the future for any user, you can reenroll it before doing a transaction. You can perform the re-enroll operation if the certificate has already expired or going to expire in the next x amount of time.
I am on fabric v2.3 and using Fabric CA.
When we re-enroll the peers, orderer or admin it will generate a new pair of certifcates.
Please correct me if i am wrong, these newly generated certificates needs to be updated in the channel configuration as well? At least this is what the doc says here
I tried to follow the doc and it does contains the root_certs and tls_certs which we need to replace. A usual sample config block after decoding it into json files looks something like this. The respective certs needs to replaced here which seems quite error prone since it's manual and a lot of certs needs to be replaced (depending on the number of peers, orderers, and organisations you've got). And, also not to forget these certs are referenced by the ccp.yaml and ccp.json as well which is used by the fabric sdk.
I am not sure if my understanding is correct about this, as I am struggling with this thing as well. I haven't been able to successfully pull this of as of now. It's strange that hyperledger fabric/Fabric CA doesn't really has a straightforward way of doing this.
Our case: k8s, fabric v2.2, mutual tls requied for all hyperledger connections, certificates expired 3 days ago.
First of all we need to pass through expired tls. Add to the orderer config:
ORDERER_GENERAL_TLS_TLSHANDSHAKETIMESHIFT: 120h
ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS: "true"
The latest parameter will allow to use expired MSP entities later.
To skip tls checks on the peer side, use the --tlsHandshakeTimeShift 120h parameter.
After the connection was restored, we need to update certificates in the blockchain. Unfortunately there is no analog of the _NOEXPIRATIONCHECKS option for the peer, so we need to use a time-shifted environment. I didn't manage to run the one in the kubernetes, so I had to use a notebook with ntp disabled and port-forwarded orderer endpoint. Be aware that if your kubernetes authentication is token-based, it can stop working because of having the incorrect time. In Azure, this is fixed by getting admin config.
That environment should be enough for step-by-step replaying of your standard certificate rotating scripts. If you don't have any, dive into this issue.
I have below queries with regards to client/user identities created by enrollment process for nodejs client. When I try to enroll identities then fabric-ca issue certificates for that particular user. Certificate issued to that client has below properties.
Certificate Information:
Common Name: admin
Organization Unit: client
Valid From: May 6, 2020
Valid To: May 6, 2021
Issuer: fabric-ca-server, Hyperledger
Now when I try to use the same identity to install and instantiate chaincode or interact with fabric runtime via TestAPI it throws below error :
2020-05-05 18:35:06.670 UTC [protoutils] ValidateProposalMessage -> WARN 0e4 channel [ichannel]: MSP error: the supplied identity is not valid: x509: certificate signed by unknown authority
I have created crypto certificates for all organizations using cryptogen tool. These certificates have been issued by the certificate authority defined in network-config.yaml file.
I suspect that the difference in the certificate issuer it's causing identity issue. Please let me know if my understanding is correct.
If yes then how I can ensure that these nodejs client certificates are being used by the same certificate authority defined in network-config.yaml file.
I was able to resolve this issue by updating Certificate Authorities environment variables inside the docker-compose file. Earlier they were not getting picked from /etc/hyperledger/fabric-ca-server-config/ location.I did update path for below environment variables of all certificate Authorities before regenerating crypto configurations and restarting the whole network:
FABRIC_CA_SERVER_CA_KEYFILE
FABRIC_CA_SERVER_CA_CERTFILE
Note: TLS configs were disabled inside the network.
This has worked for me.