Hyperledger Go SDK Client error connection is in TRANSIENT_FAILURE remote error: tls: bad certificate - hyperledger-fabric

Question

I know there are several errors like this one, but I see most of them are with the test-network, using a local environment and my case is different.
I have a blockchain running using k8s it used to be running in V1.4 and we migrated it to V2.3 so the certificates were generated without a SAN but to make them work with the new version we are using the env variable GODEBUG="x509ignoreCN=0" I can connect to it using a console client to do changes in the channel and configuration, and I can connect with a node js client and execute contracts, I want to connect using a Go client for a different app but I'm having trouble with the config.
I'm getting a TRANSIENT_FAILURE error in the Go Client and a tls: bad certificate in the peer but the certificates are the same that work for the other environments, the config.yaml I'm using is based on my configuration from the test-network that's also working.
Something weird though, is that the configPath doesn't seem to work and I need to add the users individualy to make it run.
So this is my config.yaml
---
name: network-company
version: 1.0.0
client:
organization: company
logging:
level: debug
connection:
timeout:
peer:
endorser: "300"
cryptoconfig:
path: /home/company
credentialStore:
path: /home/company/users
cryptoStore:
path: /home/company/users/admin#mycompany.com/msp
BCCSP:
security:
enabled: true
default:
provider: "SW"
hashAlgorithm: "SHA2"
softVerify: true
level: 256
tlsCerts:
systemCertPool: false
client:
cert:
path: /home/company/users/admin#mycompany.com/tls/cert.pem
key:
path: /home/company/users/admin#mycompany.com/tls/key.pem
organizations:
company:
mspid: company
cryptoPath: /home/company/users/admin#mycompany.com/msp
users:
admin:
cert:
path: /home/company/users/admin#mycompany.com/msp/signcerts/cert.pem
key:
path: /home/company/users/admin#mycompany.com/msp/keystore/key.pem
peers:
- peer1.mycompany.com
- peer2.mycompany.com
peers:
peer1.mycompany.com:
url: grpcs://peer1.mycompany.com:443
tlsCACerts:
path: /home/company/users/admin#mycompany.com/tls/ca.pem
grpcOptions:
ssl-target-name-override: peer1.mycompany.com
hostnameOverride: peer1.mycompany.com
keep-alive-time: 10s
keep-alive-timeout: 20s
keep-alive-permit: false
fail-fast: false
httpOptions:
verify: false
peer2.mycompany.com:
url: grpcs://peer2.mycompany.com:443
tlsCACerts:
path: /home/company/users/admin#mycompany.com/tls/ca.pem
grpcOptions:
ssl-target-name-override: peer2.mycompany.com
hostnameOverride: peer2.mycompany.com
keep-alive-time: 10s
keep-alive-timeout: 20s
keep-alive-permit: false
fail-fast: false
allow-insecure: true
httpOptions:
verify: false
channels:
mychannel:
peers:
peer1.mycompany.com:
endorsingPeer: true
chaincodeQuery: true
ledgerQuery: true
eventSource: true
peer2.mycompany.com:
endorsingPeer: true
chaincodeQuery: true
ledgerQuery: true
eventSource: true
policies:
queryChannelConfig:
minResponses: 1
maxTargets: 1
retryOpts:
attempts: 5
initialBackoff: 500ms
maxBackoff: 5s
backoffFactor: 2.0
entityMatchers:
peer:
- pattern: peer1.mycompany.com
urlSubstitutionExp: peer1.mycompany.com:443
sslTargetOverrideUrlSubstitutionExp: peer1.mycompany.com
mappedHost: peer1.mycompany.com
- pattern: peer2.mycompany.com
urlSubstitutionExp: peer2.mycompany.com:443
sslTargetOverrideUrlSubstitutionExp: peer2.mycompany.com
mappedHost: peer2.mycompany.com
These are my certificates paths:
/home/company/users/admin#mycompany.com
+ msp
+ admincerts/<empty>
+ signcerts/cert.pem
+ cacerts/ca.pem
+ intermediatecerts/ca.pem
+ tlscacerts/ca.pem
+ tlsintermediatecerts/ca.pem
+ keystore/key.pem
+ tls
+ ca.pem
+ cert.pem
+ key.pem
These are the logs from the client:
[fabsdk/core] 2022/01/04 14:34:09 UTC - sw.getOptsByConfig -> DEBU Initialized SW cryptosuite
[fabsdk/fab] 2022/01/04 14:34:09 UTC - n/a -> DEBU KeyStore opened at [/home/company/users/admin#mycompany.com/msp/keystore]...done
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU Client is: {Organization:company TLSCerts:{Client:{Key:{Path:/home/company/users/admin-tls#mycompany.com/tls/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/admin-tls#mycompany.com/tls/cert.pem Pem: bytes:[]}}}}
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU channels are: map[mychannel:{Orderers:[] Peers:map[peer1.mycompany.com:{EndorsingPeer:true ChaincodeQuery:true LedgerQuery:true EventSource:true} peer2.mycompany.com:{EndorsingPeer:true ChaincodeQuery:true LedgerQuery:true EventSource:true}] Policies:{QueryChannelConfig:{MinResponses:1 MaxTargets:1 RetryOpts:{Attempts:5 InitialBackoff:500ms MaxBackoff:5s BackoffFactor:2 RetryableCodes:map[]}} Discovery:{MinResponses:0 MaxTargets:0 RetryOpts:{Attempts:0 InitialBackoff:0s MaxBackoff:0s BackoffFactor:0 RetryableCodes:map[]}} Selection:{SortingStrategy: Balancer: BlockHeightLagThreshold:0} EventService:{ResolverStrategy: MinBlockHeightResolverMode: Balancer: BlockHeightLagThreshold:0 PeerMonitor: ReconnectBlockHeightLagThreshold:0 PeerMonitorPeriod:0s}}}]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU organizations are: map[company:{MSPID:company CryptoPath:/home/company/users/admin#mycompany.com/msp Users:map[admin:{Key:{Path:/home/company/users/admin#mycompany.com/msp/keystore/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/admin#mycompany.com/msp/signcerts/cert.pem Pem: bytes:[]}}] Peers:[peer1.mycompany.com peer2.mycompany.com] CertificateAuthorities:[]}]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU orderers are: map[]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU peers are: map[peer1.mycompany.com:{URL:grpcs://peer1.mycompany.com:443 GRPCOptions:map[allow-insecure:true fail-fast:false hostnameoverride:peer1.mycompany.com keep-alive-permit:false keep-alive-time:10s keep-alive-timeout:20s ssl-target-name-override:peer1.mycompany.com] TLSCACerts:{Path:/home/company/users/admin#mycompany.com/tls/ca.pem Pem: bytes:[]}} peer2.mycompany.com:{URL:grpcs://peer2.mycompany.com:443 GRPCOptions:map[allow-insecure:true fail-fast:false hostnameoverride:peer2.mycompany.com keep-alive-permit:false keep-alive-time:10s keep-alive-timeout:20s ssl-target-name-override:peer2.mycompany.com] TLSCACerts:{Path:/home/company/users/admin#mycompany.com/tls/ca.pem Pem: bytes:[]}}]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).compileMatchers -> DEBU Matchers are: {matchers:map[]}
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).compileMatchers -> DEBU Entity matchers are not configured
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadTLSClientCerts -> DEBU Reading pk from config, unable to retrieve from cert: Could not find matching key for SKI: Failed getting key for SKI [[91 7 147 74 144 157 11 46 234 18 159 199 54 134 208 140 16 147 185 225 158 7 112 91 182 170 49 46 139 190 194 99]]: key with SKI 5b07934a909d0b2eea129fc73686d08c1093b9e19e07705bb6aa312e8bbec263 not found in /home/company/users/admin#mycompany.com/msp/keystore
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadPrivateKeyFromConfig -> DEBU pk read from config successfully
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).loadIdentityConfigEntities -> DEBU Client is: {Organization:company Logging:{Level:debug} CryptoConfig:{Path:/home/company} TLSCerts:{Client:{Key:{Path:/home/company/users/admin-tls#mycompany.com/tls/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/admin-tls#mycompany.com/tls/cert.pem Pem: bytes:[]}} SystemCertPool:false} CredentialStore:{Path:/home/company/users CryptoStore:{Path:/home/company/users/admin#mycompany.com/msp}}}
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).loadIdentityConfigEntities -> DEBU organizations are: map[company:{MSPID:company CryptoPath:/home/company/users/admin#mycompany.com/msp Users:map[admin:{Key:{Path:/home/company/users/admin#mycompany.com/msp/keystore/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/admin#mycompany.com/msp/signcerts/cert.pem Pem: bytes:[]}}] Peers:[peer1.mycompany.com peer2.mycompany.com] CertificateAuthorities:[]}]
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).loadIdentityConfigEntities -> DEBU certificateAuthorities are: map[]
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).compileMatchers -> DEBU Matchers are: {matchers:map[]}
[fabsdk] 2022/01/04 14:34:09 UTC - fabsdk.initSDK -> DEBU SDK initialized successfully
SDK created
resourceManagerClient created
[fabsdk/fab] 2022/01/04 14:34:09 UTC - chconfig.(*params).SetChConfigRefreshInterval -> DEBU RefreshInterval: 1m30s
[fabsdk/util] 2022/01/04 14:34:09 UTC - lazyref.WithRefreshInterval.func1 -> DEBU Checking refreshIntervalSetter
[fabsdk/fab] 2022/01/04 14:34:09 UTC - peer.(*peerEndorser).ProcessTransactionProposal -> DEBU Processing proposal using endorser: peer2.mycompany.com:443
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).DialContext -> DEBU DialContext: peer2.mycompany.com:443
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).createConn -> DEBU creating connection [peer2.mycompany.com:443]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).createConn -> DEBU storing connection [peer2.mycompany.com:443]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).removeConn -> DEBU removing connection [peer2.mycompany.com:443]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - txn.SendProposal.func1 -> DEBU Received error response from txn proposal processing: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE
[fabsdk/common] 2022/01/04 14:34:09 UTC - retry.(*RetryableInvoker).Invoke -> DEBU Failed with err [queryChaincode failed: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE] on attempt #1. Checking if retry is warranted...
[fabsdk/common] 2022/01/04 14:34:09 UTC - retry.(*RetryableInvoker).Invoke -> DEBU ... retry for err [queryChaincode failed: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE] is NOT warranted after 1 attempt(s).
2022/01/04 14:34:09 error inicializando el SDK: error creando el cliente del canal: event service creation failed: could not get chConfig cache reference: QueryBlockConfig failed: QueryBlockConfig failed: queryChaincode failed: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE
These are the logs from the peer:
2022-01-04 15:23:56.740 UTC [core.comm] ServerHandshake -> ERRO 335 Server TLS handshake failed in 7.869808ms with error remote error: tls: bad certificate server=PeerServer remoteaddress=10.250.37.211:33666
2022-01-04 15:23:56.740 UTC [grpc] WarningDepth -> DEBU 336 [core]grpc: Server.Serve failed to complete security handshake from "10.250.37.211:33666": remote error: tls: bad certificate
A couple of weird things: I need to add the users to the organization, some how it's not able to extract the users from the configPath and I get an error user not found.
It's looking for the tls private key in the msp/keystore folder and of course that isn't there
I tried adding the GODEBUG="x509ignoreCN=0" but it doesn't change the result
I generated an extra TLS certificates just for the client adding the SAN to them as "email:admin-tls#mycompany.com" and mapped it to the configuration with the same result.
I have tried adding and removing the entityMatchers with the same result.
I also did a tcp dump on it server answers to the Client hello with Server hello and then fails with a 400 bad request error.
I have used the configurations referenced from here and here but everything looks good to me.
I'm running out of ideas, is my configuration right? why do I need to force the clients and configPath doesn't work? why is it looking for the tls key in the msp keystore? but mostly why do I get a tls bad certificate if the certificates are good and pass an openssl verify against the ca.pem?
Thanks
UPDATE
This is my certificate info using openssl x509 -in certificate.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
56:00:01:ba:18:d9:cf:ed:a7:00:b9:af:b3:00:01:00:01:ba:18
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = CO, L = Medellin, O = mycompany, CN = CA My Company, emailAddress = admin#mycompany.com
Validity
Not Before: Jan 5 12:36:08 2022 GMT
Not After : Jan 5 12:36:08 2023 GMT
Subject: C = CO, ST = ANTIOQUIA, L = MEDELLIN, O = mycompany, OU = admin, CN = admin-tls#mycompany.com
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:d0:c8:3b:20:2f:8f:3b:91:72:f8:71:27:30:cf:
b5:6e:38:e3:5f:e6:1b:42:77:fd:00:f2:8c:1b:ae:
6c:2e:7d:50:03:75:d5:03:db:72:d5:33:27:f3:65:
c1:2a:09:72:6a:d5:cc:c5:45:ca:e5:e5:8c:65:4b:
f2:51:dc:dd:ac
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage:
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, Microsoft Encrypted File System, E-mail Protection, TLS Web Client Authentication, Any Extended Key Usage
X509v3 Subject Alternative Name:
DNS:mycompany.com
X509v3 Subject Key Identifier:
9E:C5:4B:E8:67:B1:84:BF:67:E9:BE:A3:DE:B5:EB:E3:3B:4E:D3:D0
X509v3 Authority Key Identifier:
keyid:40:54:B4:93:16:92:20:E3:DB:AA:7A:A3:13:06:EC:65:4F:83:B3:58
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.epm.com.co/CRL/CAEPMClaseI.crl
URI:http://crl2.epm.com.co/CRL/CAEPMClaseI.crl
Authority Information Access:
OCSP - URI:http://ocsp.epm.com.co/OCSP
CA Issuers - URI:http://crl.epm.com.co/CRL/CAEPMClaseI.crt
1.3.6.1.4.1.311.21.7:
0-.%+.....7.....-...K..=...>...k8...,...A..d..
S/MIME Capabilities:
......0...+....0050...*.H..
..*.H..
Signature Algorithm: sha256WithRSAEncryption
6f:d6:85:b1:08:cd:07:78:3d:92:50:85:14:3a:f0:2e:1f:84:
10:7c:1d:15:10:0f:9f:66:e7:1a:41:63:12:83:0c:7c:04:c7:
c8:f4:8b:97:d9:6e:28:78:cb:9b:af:89:6b:1a:f3:06:40:1e:
6f:90:81:8f:f9:03:ff:92:5a:de:8c:7c:35:f9:21:00:fd:85:
c0:73:48:71:c0:82:7a:17:6d:09:ce:b6:03:5a:c5:1f:4f:47:
6d:95:94:a3:e2:cd:19:03:3d:85:a0:7e:5e:cf:a5:55:32:9f:
28:cf:d4:10:c8:42:57:a1:9b:cc:9a:4b:82:75:86:15:92:b9:
ef:d2:b2:1c:5d:a9:fe:ff:d4:f2:83:9d:31:46:50:26:ea:78:
88:9c:72:0a:1e:81:42:88:7b:04:4b:4b:2b:86:3d:ed:f9:4d:
8f:5e:47:1b:0b:8e:a9:8a:ed:ed:8a:09:92:58:6b:14:6e:aa:
50:c1:ae:2f:75:41:5b:e9:28:2b:85:9f:70:0a:d3:27:64:44:
f4:f6:61:99:a0:db:90:05:8c:af:09:fa:36:a5:67:3a:b8:f5:
b3:06:fe:ba:50:aa:1f:2e:43:af:a6:3a:81:01:d1:26:cc:61:
3b:c3:d9:b5:50:d7:27:61:f6:3e:a3:1d:99:0f:f1:37:20:d4:
ea:6d:59:a9:56:09:09:74:90:d4:52:64:69:b5:ad:09:b8:67:
74:ca:52:76:0a:37:0e:29:e0:5b:5a:bc:24:42:a1:14:77:1f:
72:f5:e9:f8:64:17:2e:fc:57:e1:d3:74:3a:1d:a5:fb:f4:c3:
a3:1e:97:f3:ba:d6:33:6a:24:d5:79:d7:9c:07:b4:9d:d0:52:
24:dc:0d:d5:a1:37:42:25:09:3b:96:a5:91:19:6c:7e:58:71:
99:ff:04:14:4d:ea:25:57:23:58:2b:44:84:f9:c3:c4:f6:a6:
ed:81:75:0a:07:e4:ab:bf:74:ec:e6:8c:9e:b8:75:82:47:20:
75:64:94:45:93:2f:ec:8b:8f:2c:19:c8:b0:29:a0:4c:f7:05:
6d:c8:3d:54:78:9e:ac:17:1c:19:5e:3c:1a:ba:93:db:1c:41:
ab:09:e2:d7:e3:11:87:eb:b7:38:e6:cf:85:1e:8a:75:9d:8c:
68:44:5d:d6:60:e7:91:52:ec:08:5b:26:7b:a4:42:d6:49:fc:
3c:74:6c:96:23:99:86:6b:6b:57:89:94:ad:08:98:35:1d:38:
07:09:ae:c3:8c:1f:3c:36:94:a9:da:74:73:ff:5e:c1:0d:f7:
45:a6:85:94:1d:34:46:56:fd:82:3f:8d:c0:d0:a1:20:1b:76:
d3:2e:f8:a7:6c:3f:25:f4

Answer 1

So the issue seemed to be with the TLS certificates from the peers, they where generated before go 1.16 and didn't had the SANs in them, and the error seemed to be getting reflected to the go SDK. #yacovm thank you for your help figuring it out.