AKS: 1.20.7
Elasticsearch : 7.12.0
client: openssl
Description of the problem
Cluster fails to come live when PEM files are downloaded from cloud and mounted # /usr/share/elasticsearch/config/certs without a custom Elasticsearch image
Works fine with a custom Elasticsearch image and PEM files copied to /usr/share/elasticsearch/config/certs during image building
PEM files are downloaded from KeyVault using CSI driver
PEM files downloaded from cloud provider (Error case)
[elasticsearch#test-rp-search-master-0 ~]$ openssl s_client -connect 127.0.0.1:9200
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = US, ST = US, O = Self Signed, CN = test-rp-search-data
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = US, O = Self Signed, CN = test-rp-search-data
verify error:num=21:unable to verify the first certificate
verify return:1
140680078673728:error:0407E086:rsa routines:RSA_verify_PKCS1_PSS_mgf1:last octet
invalid:crypto/rsa/rsa_pss.c:88:
140680078673728:error:1417B07B:SSL routines:tls_process_cert_verify:bad
signature:ssl/statem/statem_lib.c:505:
---
Certificate chain
0 s:C = US, ST = US, O = Self Signed, CN = test-rp-search-data
i:C = US, ST = US, O = Self Signed, CN = Research Platform Issuing CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEzDCCArSgAwIBAgIUIQThqQtH1QPR3YocBxLrrj+RQ9owDQYJKoZIhvcNAQEL
BQAwVzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlVTMRQwEgYDVQQKDAtTZWxmIFNp
PEM files in the Elasticsearch custom images (Works fine)
[elasticsearch#test-rp-search-master-0 ~]$ openssl s_client -connect 127.0.0.1:9200
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = US, ST = US, O = Self Signed, CN = test-rp-search-data
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = US, O = Self Signed, CN = test-rp-search-data
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:C = US, ST = US, O = Self Signed, CN = test-rp-search-data
i:C = US, ST = US, O = Self Signed, CN = Research Platform Issuing CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEzDCCArSgAwIBAgIUIQThqQtH1QPR3YocBxLrrj+RQ9YwDQYJKoZIhvcNAQEL
BQAwVzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlVTMRQwEgYDVQQKDAtTZWxmIFNp
Related
I'm trying to figure out why Python is throwing CERTIFICATE_VERIFY_FAILED exceptions for a certain endpoint I have configured, but other tools like OpenSSL / sslscan / sslyze seem to be fine with it.
The context of this is that we started receiving these errors around the same time we were rotating certificates. However, I have checked the certificate ordering (Actual cert + intermediates + root, in one file, in that order from top to bottom) and that is not the problem.
The simplest example I conjured up to test this is the following:
import certifi
import os
import socket
import ssl
SERVER = "myhost.example.com"
PORT = 443
context_instance = ssl.SSLContext()
context_instance.verify_mode = ssl.CERT_REQUIRED
context_instance.load_verify_locations(
cafile=os.path.relpath(certifi.where()), capath=None, cadata=None
)
s = socket.socket()
ssl_socket = context_instance.wrap_socket(s)
ssl_socket.connect((SERVER, PORT))
print("Version of the SSL Protocol:", ssl_socket.version())
print("Cipher used:", ssl_socket.cipher())
For example, using facebook.com as the SERVER yields the following:
Version of the SSL Protocol: TLSv1.3
Cipher used: ('TLS_CHACHA20_POLY1305_SHA256', 'TLSv1.3', 256)
But when I use this to test our internal endpoint, I get the following error (I also get this when I use google.com also, which is weird):
Traceback (most recent call last):
File "test.py", line 33, in <module>
ssl_socket.connect((SERVER, PORT))
File "/usr/local/lib/python3.8/ssl.py", line 1342, in connect
self._real_connect(addr, False)
File "/usr/local/lib/python3.8/ssl.py", line 1333, in _real_connect
self.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)
My questions are:
What's a foolproof way in Python 3 to verify if a TLS-enabled endpoint (let's just use HTTPS for now) is configured correctly?
Is there a way to achieve a more verbose error output from Python to see what the problem on my internal endpoint is? The exception does not give any more details as to what could be wrong.
For reference, this is what I'm using:
Python 3.8.13
OS: Debian 11
certifi==2022.6.15
And this is the output from OpenSSL:
$ openssl s_client -connect myhost.example.com:443 -4 <<< "Q"
CONNECTED(00000003)
depth=2 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
verify return:1
depth=1 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
verify return:1
depth=0 CN = *.example.com
verify return:1
---
Certificate chain
0 s:CN = *.example.com
i:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
1 s:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
i:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
2 s:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
i:C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
3 s:C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
i:C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
---
Server certificate
<masked>
subject=CN = *.example.com
issuer=C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5687 bytes and written 409 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
For the future, there are a couple of insightful answers here, and all of them contribute to the overall answer:
dave_thompson_085's answer was useful in determining that, if you want to check whether your TLS-enabled server endpoint is serving the right certificate, it's useful to run one of the following commands:
For cases where the server only serves one hostname:
$ openssl s_client -connect myhost.example.com:443 -4 <<< "Q"
For cases where the server has multiple TLS hosts that it serves, and you want to check what happens when you don't pass a server hostname:
$ openssl s_client -noservername -connect myhost.example.com:443 -4 <<< "Q"
This will show a default backend if it's an HTTP server.
For the Python bit, you need to pass in the server_hostname. Something like this was able to work for all valid HTTPS sites I tested:
import os
import socket
import ssl
import certifi
SERVER = "myhost.example.com"
PORT = 443
context_instance = ssl.SSLContext()
context_instance.verify_mode = ssl.CERT_REQUIRED
context_instance.check_hostname = True
context_instance.load_verify_locations(
cafile=os.path.relpath(certifi.where()), capath=None, cadata=None
)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_socket = context_instance.wrap_socket(s, server_hostname=SERVER)
ssl_socket.connect((SERVER, PORT))
print("Server hostname:", ssl_socket.server_hostname)
print("Version of the SSL Protocol:", ssl_socket.version())
print("Cipher used:", ssl_socket.cipher())
Finally, the real underlying problem was the version of aiohttp==3.7.0 that we were using in our application. In the CHANGELOG for version 3.7.1, the following line shows the problem we were having:
Fix a variable-shadowing bug causing ThreadedResolver.resolve to return the resolved IP as the hostname in each record, which prevented validation of HTTPS connections. #5110
The script I ran to test the broken versions of aiohttp was the following:
import asyncio
import aiohttp
SERVER = "myhost.example.com"
async def main():
async with aiohttp.ClientSession(raise_for_status=True) as session:
async with session.get(f"https://{SERVER}") as r:
body = await r.json()
print(body)
if __name__ == "__main__":
asyncio.run(main())
I'm trying to get root certificate for gitlab.com so that I can download files/build artifacts from repository using GitLab API (I need this due for HTTPS authentication from an IoT device).
For GitHub, I used the following command, provided by the MCU manufacture for OTA:
echo "" | \
openssl s_client -showcerts -connect api.github.com:443 | \
sed -n "1,/Root/d; /BEGIN/,/END/p" | openssl x509 -outform PEM
However, the same (replacing api.github with gitlab.com) doesn't get the root certificate. I'm a beginner, so can some please help.
From issue 15364, you should be able to see the root CA (and 2 intermediate CAs) with:
echo | openssl s_client -connect gitlab.com:443 -showcerts
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = gitlab.com
verify return:1
CONNECTED(00000005)
---
Certificate chain
0 s:CN = gitlab.com
i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
-----BEGIN CERTIFICATE-----
<snip> <<<<====== Root CA
-----END CERTIFICATE-----
1 s:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
I have 3 files *.cer certificate files. Contents of file have,
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
These 3 certificates include A.cer, B.cer and TCA.cer. TCA.cer is the certificate authority file. Here are the details.
1) A.cer
a) Subject CN = A Gateway
b) Issuer CN = B CA
2) B.cer
a) Subject CN = B CA
b) Issuer CN = TCA
3) TCA.cer
a) Subject CN = TCA
b) Issuer CN = TCA
Now I need to create certificate chain from above files and pick one of them as certificate file. Unfortunately, I don't know how can I combine 2 files to create chain certificate and pick one as certificate data file.
It's required to put the server certificate file first, and then the intermediate certificate files. So in your case just append them: A + B + TCA
I created 3 certificates using Python: rootca.crt, intermediateca.crt and server.crt.
I used the rootca.crt to sign intermediateca.crt, which works as expected:
openssl verify -CAfile rootca.crt intermediateca.crt
intermediateca.crt: OK
Then I signed the server.crt with the intermediate ca, but verification fails:
openssl verify -CAfile rootca.crt -untrusted intermediateca.crt server.crt
server.crt: C = DE, ST = mein Bundesland, L = meine Stadt, O = meine Firma, CN = server.example.com, emailAddress = info#meine-firma.de
error 20 at 0 depth lookup:unable to get local issuer certificate
When I parse the certificates, the server.crt authority key identifier matches the intermediateca subject key identifier. Can anyone give me a hint what could be wrong? If I generate the same certificates with the openssl command line tool it works. The parsed content is identical, apart from the fact that the authority key identifier also contains a serial and a cn for the openssl generated certificate.
The intermediate CA cannot be used to verify the server certificate because its subject name does not match the issuer name specified in the server certificate.
Let's have openssl dump the subject and issuer names. The -xx_hash shows the hash that openssl uses to build up the certificate chain:
$ openssl x509 -subject -subject_hash -noout -in rootca.crt
subject=C = DE, ST = mein Bundesland, L = meine Stadt, O = meine Firma, OU = meine Abteilung, CN = serviceserver.example.com, emailAddress = info#meine-firma.de
347e2056
$ openssl x509 -issuer -issuer_hash -noout -in intermediateca.crt
issuer=C = DE, ST = mein Bundesland, L = meine Stadt, O = meine Firma, OU = meine Abteilung, CN = serviceserver.example.com, emailAddress = info#meine-firma.de
347e2056
Great, the intermediate's Issuer name matches the root's Subject name. That part of the chain works.
$ openssl x509 -subject -subject_hash -noout -in intermediateca.crt
subject=C = DE, ST = mein Bundesland, L = meine Stadt, O = meine Firma, CN = serviceserver.example.com, emailAddress = info#meine-firma.de
c4dff14c
$ openssl x509 -issuer -issuer_hash -noout -in server.crt
issuer=C = DE, ST = mein Bundesland, L = meine Stadt, O = meine Firma, OU = meine Abteilung, CN = serviceserver.example.com, emailAddress = info#meine-firma.de
347e2056
Oops: the hash is different, so openssl cannot connect the intermediate CA to the server certificate. The difference is that the intermediate's subject name contains a OU field whereas the server's issuer name does not. openssl was correct when it told you that it could not find an issuer.
I'm not sure how you got it in this state, my guess would be some misconfiguration of the subject or issuer name.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I followed this instruction to create a self-signed certificate
http://apetec.com/support/GenerateSAN-CSR.htm. However, the certificate is always failed to be verified and my tls connection program can't setup connection using this certificate.
Any idea why and how to solve it?
The following is the commands to generate the certificate and result of verification.
$ openssl genrsa -out private.key 2048
$ openssl req -new -out public.csr -key private.key -config openssl.conf
$ openssl req -text -noout -in public.csr
$ openssl x509 -req -days 365 -in public.csr -signkey private.key -out public.crt -extensions v3_req -extfile openssl.conf
$ openssl verify -CAfile public.crt public.crt
public.crt: O = My Company, L = My Town, ST = State or Providence, C = US
error 20 at 0 depth lookup:unable to get local issuer certificate
The following is the openssl.conf. The ip address is partially crossed out.
#
# OpenSSL configuration file.
#
# Establish working directory.
dir = .
[ ca ]
default_ca = CA_default
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ req ]
default_bits = 1024 # Size of keys
default_keyfile = key.pem # name of generated keys
default_md = md5 # message digest algorithm
string_mask = nombstr # permitted characters
distinguished_name = req_distinguished_name
req_extensions = v3_req
[ req_distinguished_name ]
# Variable name Prompt string
#------------------------- ----------------------------------
0.organizationName = Organization Name (company)
organizationalUnitName = Organizational Unit Name (department, division)
emailAddress = Email Address
emailAddress_max = 40
localityName = Locality Name (city, district)
stateOrProvinceName = State or Province Name (full name)
countryName = Country Name (2 letter code)
countryName_min = 2
countryName_max = 2
commonName = Common Name (hostname, IP, or your name)
commonName_max = 64
# Default values for the above, for consistency and less typing.
# Variable name Value
#------------------------ ------------------------------
0.organizationName_default = My Company
localityName_default = My Town
stateOrProvinceName_default = State or Providence
countryName_default = US
[ v3_ca ]
basicConstraints = CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
[ v3_req ]
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = #alt_names
[alt_names]
IP.1 = 1xx.1x.1xx.xxx
What you are generating is a self-signed root certificate. OpenSSL attempts to verify certificates by chaining certificates up to a trusted root that is present in its certificate store. Since yours is (obviously) not in that store it will always fail.
Here are three ways to get rid of the warnings:
Disable certificate verification
This is generally a bad idea because without certificate verification you have completely disabled the identity component of a TLS handshake. Use it only in development (and never let it leak to production!)
Add your root certificate to the trust store
This will work provided you're willing to install the certificate on every machine that needs to talk to this endpoint. (For OpenSSL this is a ca_bundle file that is located in a distribution specific location)
Buy a cert from a CA
The easiest, but also the one that costs $$$. If you do this then the site you're installing this certificate on will be trusted globally.