I'm using the "requests" module for getting JSON from my web service, using the next code:
import requests
import SSL
# With or without this line of code below, the output is the same
ssl.match_hostname = lambda cert, hostname: True
response = requests.get("MY_URL", cert=("client.pem", "client-key.pem"), verify="CAcert.cer")
When the SSL step seems to fail with the following message:
HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: {WEBSERVICE_URL_PATTERN} (Caused by SSLError(CertificateError("hostname 'x.x.x.x' doesn't match 'x.x.x.x'")))
I'm using Python 3.10.5 with the latest version of the "requests" module.
Does anyone know what could cause this kind of error and how to fix it?
I assume you've redacted actual names which are in fact different, because if you really did have a host named x.x.x.x using a cert with the same name it would match (unless it wasn't really the same because the CA, or a potentially-bogus 'subject'/'subscriber', used lookalike characters).
From the documentation of match_hostname
Changed in version 3.7: The function is no longer used to TLS connections. Hostname matching is now performed by OpenSSL. ...
Deprecated since version 3.7.
At the python.ssl level, or http.client or urllib.requests, you can still turn off only hostname checking with check_hostname=False in the SSLContext. However AFAIK requests doesn't give you access to the SSL level except for setting the cert(s) as you do or the sledgehammer option of turning off all verification with verify=False.
If at all possible, you should try to use a hostname and a host cert that do match. Note changing either the name you request or the cert can accomplish this.
The problem was solved, using a Subject Alternative Name (SAN) for the server, with a value of its own IP address.
I've found out that we use Simple-CA, and the request of getting a signed certificate from it was with a Common Name (CN), when we don't have a domain name.
After changing the signing action to SAN instead of CN, the problem was solved.
Thanks for the helpers!
Related
Hello fellow developers ! I'm stuck in a corner case and I'm starting to be out of hairs to pull... Here is the plot :
load-balancer.example.com:443 (TCP passthrough)
/\
/ \
/ \
/ \
s1.example.com:443 s2.example.com:443
(SSL/SNI) (SSL/SNI)
The goal is to stress-test the upstreams s1 and s2 directly using aiohttp with certificate-validation enable. Since the load-balancer does not belong to me I don't want to do the stress-test over it.
the code is not supposed to run on other platforms than GNU Linux with at least Python-v3.7 (but I can use any recent version if needed)
all servers serve a valid certificate for load-balancer.example.com
openssl validates the certificate from the upstreams when using openssl s_connect s1.example.com:443 -servername load-balancer.example.com
cURL needs curl 'https://load-balancer.example.com/' --resolve s1.example.com:443:load-balancer.example.com and also validates successfully
I am able to launch a huge batch of async ClientSession.get requests on both upstreams in parallel but for each request I need to somehow tell asyncio or aiohttp to use load-balancer.example.com as server_hostname, otherwise the SSL handshake fails.
Is there an easy way to setup the ClientSession to use a specific server_hostname when setting up the SSL socket ?
Does someone have already done something like that ?
EDIT : here is the most simple snippet with just a single request :
import aiohttp
import asyncio
async def main_async(host, port, uri, params=[], headers={}, sni_hostname=None):
if sni_hostname is not None:
print('Setting SNI server_name field ')
#
# THIS IS WHERE I DON'T KNOW HOW TO TELL aiohttp
# TO SET THE server_name FIELD TO sni_hostname
# IN THE SSL SOCKET BEFORE PERFORMING THE SSL HANDSHAKE
#
try:
async with aiohttp.ClientSession(raise_for_status=True) as session:
async with session.get(f'https://{host}:{port}/{uri}', params=params, headers=headers) as r:
body = await r.read()
print(body)
except Exception as e:
print(f'Exception while requesting ({e}) ')
if __name__ == "__main__":
asyncio.run(main_async(host='s1.example.com', port=443,
uri='/api/some/endpoint',
params={'apikey': '0123456789'},
headers={'Host': 'load-balancer.example.com'},
sni_hostname='load-balancer.example.com'))
When running it with real hosts, it throws
Cannot connect to host s1.example.com:443 ssl:True
[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] '
certificate verify failed: certificate has expired (_ssl.c:1131)')])
Note that the error certificate has expired indicates that the certificate proposed to the client is the default certificate since the SNI hostname is s1.example.com which is unknow by the webserver running there.
When running it against the load-balancer it works just fine, the SSL handshake happens with the upstreams which serve the certificate and everything is valid.
Also note that
sni_callback does not help since it is called after the handshake has started and the certificate was received (and at this point server_hostname is a read-only property anyway)
it does not seem to be possible to set server_hostname when creating an SSLContext allthough SSLContext.wrap_socket does support server_hostname but I was not able to make that work
I hope someone knows how to fill the comment block in that snippet ;-]
I am using the request library to fetch a response from an API, everything was working fine but as of recent, I have started to get local issuer certificate error from the API although the same code is working on my colleague's system.
I have already tried updating the node version and all the version changes regarding the request package of node
The exact error is:
unable to get local issuer certificate node
Here is the error snippet in the console
Could u please console only response..After that u can easily see that there is statusCode is available or not.
Or if u get the statusCode then check its defined or not.
Ex: if(res.statusCode)
console.log("print status code", this.res.statusCode);
This is an SSL error and yo need to make sure that your server's certificate is valid.
With your claim that it used to work - but it stopped working recently with the SSL error message, the possibilities are that either your servers certificate is expired in between or the Sectigo AddTrust External CA Root Expiring May 30, 2020.
https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT
Updating the certificate is outside the scope of this answer. You may have to do it yourself or contact the admin.
I want to import some data to Excel2016 from a postgresSQL table. I have tried it by clicking "new query" and selecting From Database -> From PostgresSQL Database:
But then I receive the following error:
Details: "TlsClientStream.ClientAlertException: CertificateUnknown: Server certificate was not accepted. Chain status: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.
. The specified hostname was not present in the certificate.
at TlsClientStream.TlsClientStream.ParseCertificateMessage(Byte[] buf, Int32& pos)
at TlsClientStream.TlsClientStream.TraverseHandshakeMessages()
at TlsClientStream.TlsClientStream.GetInitialHandshakeMessages(Boolean allowApplicationData)
at TlsClientStream.TlsClientStream.PerformInitialHandshake(String hostName, X509CertificateCollection clientCertificates, RemoteCertificateValidationCallback remoteCertificateValidationCallback, Boolean checkCertificateRevocation)"
Any suggestions on how to solve this? Thank you so much in advance!
This error is indicative of a connection being made to the PostgreSQL db where the server's certificate cannot be validated by the client making a connection. This error only happens when the "Trust Server Certificate" is set to FALSE in the library Excel uses to connect to PostgreSQL (npgsql).
There are several ways that may work to address this, in the order I'd suggest trying them:
If there's an option hidden in Excel (perhaps under advanced options or similar) to set the 'Trust Server Certificate' parameter to True, then your connection will start working. If it allows you to specify an entire connection string, then this can be done in the connection string as well.
The database should have a public key in an SSL cert listed in the postgresql.conf file for the db. If you (or your db administrator) can get that public key and add it to your machine (instructions will vary depending on your operating system).
I have finally found a workaround for my problem.
What you can do is to:
Install the current postgresql driver from here
Follow the instruction from this video
With this, you can connect to your postgreSQL database by ODBC.
Anaconda - Python 3.6
OpenSSL 1.0.2
Operating System: Windows 7
Phase 1 (Completed): Using selenium: launched, navigated, and extracted various data elements including a table from site. Extracted Hyperlinks contained in table that are direct links to documents.
Phase 2: Taking extracted hyperlink from table I need to download the files to a specified folder on the shared drive.
Tried:
import urllib.request
url = 'tts website/test.doc'
urllib.request.urlretrieve(url,'C:\Users\User\Desktop\')
Error I get is sslv3 alert handshake failure
With the site opened, I have clicked on the Lock icon and clicked "Install Certificate". I have saved the certificate to my "Trusted Root Certification Authorities" in the Certificate store.
I can see the certificate name (when i installed certificate) from the above step in the 58 CA Certificates shown by running the following code:
import socket
import ssl
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_REQUIRED
context.load_default_certs()
ca_certs = context.get_ca_certs()
print('{} CA Certificates loaded: ' .format(len(ca_certs)))
for cert_dict in ca_certs:
print(cert_dict['subject'])
print()
I can't figure out how to secure a SSL connection to the website/server in order to download the file from each of the hyperlinks?? This website uses Single Sign On(SSO) and automatically logins me in when I first launch the website.
I have tried to use server server.net 443 to connect to server, but can't seem to get the scripting right to connect and retrieve the document.
I have connected directly to the server and abstracted the certificate details shown here:
HOST, PORT = server.net, 443
ctx = ssl.create_default_context()
s = ctx.wrap_socket(socket.socket(), server_hostname=HOST)
c.connect((HOST, PORT))
cert = s.getpeercert()
print(cert)
When i run urlretrieve i am still getting the same error: handshake. When reviewing my ca certificates i see there is a Personal certificate for my Windows login (username) listed there, that must be how it is automatically logging me in using SSO. How do i take all of this information, connect to the website using my SSO, and retrieve the documents?
Latest UPDATE:
I am finding pycurl to be promising, however I feel like I need a little assistance making a few tweaks to get it working.
import pycurl
fp = open('Test.doc','wb')
curl = pycurl.Curl()
curl.setopt(pycurl.URL, url) # see url link to go to word doc
curl.setopt(pycurl.FOLLOWLOCATION, 1)
curl.setopt(pycurl.MAXREDIRS, 5)
curl.setopt(pycurl.CONNECTTIMEOUT,30)
curl.setopt(pycurl.TIMEOUT, 300)
try:
curl.setopt(pycurl.WRITEDATA, fp)
curl.perform()
except:
import traceback
traceback.print_exc(file=sys.stderr)
sys.stderr.flush()
curl.close()
fp.close()
This code yields no error, however the created word doc contains an error displaying a print screen of the log on page of the website.
Main Problem: HTTPS connection using Single Signon connection behind a corporate network proxy server.
I have been trying to get this to work to validate cacert, but I have been getting this error message now:
curl.setopt(pycurl.SSL_VERIFYPEER, 1)
curl.setopt(pycurl.SSL_VERIFYPEER, 2)
curl.setopt(pycurl.CAINFO, certifi.where())
but now i am getting ERROR: 51, CERT_TRUST_IS_UNTRUSTED_ROOT
How do i add proxy if that is causing the error? and Secondly, how do i attach the ca certificate file directly?
I am trying to access a SOAP server using zeep. My server uses SSL with a custom certificate, and connection to that server works, with my cert, or ignoring it:
python -mzeep "https://<server-ip>/servicemanager/1?wsdl" --no-verify
I get a long list of Prefixes, Global elements, Global types, Bindings and Service. The latter one says:
Service: ServiceManager
Port: servicemanager_1 (Soap11Binding: {http://soap.client.<snipped>.at}servicemanager_1Binding)
Operations:
getServices() -> return: ns0:service[]
So, from what I can say by now, I can create a client object and call it's service named getServices().
from zeep import CachingClient as Client
from zeep.wsse.signature import Signature
from zeep.transports import Transport
from requests import Session, Request
session = Session()
session.verify = False
transport = Transport(session=session)
c = Client('https://<server-ip>/servicemanager/1?wsdl', transport=transport)
c.service.getServices()
But that leads to an error in urllib3 (~/.virtualenvs/soap/lib/python3.5/site-packages/urllib3/util/connection.py):
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
[...]
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost',
port=443): Max retries exceeded with url: /servicemanager/1 (Caused by
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7f4e2a6f7d30>: Failed to establish a new connection: [Errno 111]
Connection refused',))
It does not matter if I ignore the SSL verification, or provide a CA_BUNDLE. both are accepted, the client is created, but I can't call the getServices() method.
What did I forget here? I don't think this is a zeep problem, as the underlying urllib3 throws the exception. But I tried for hours and searched the internet for a solution, without success.
Apart of the XML I get from the endpoint is:
<service name="ServiceManager">
<port name="servicemanager_1" binding="tns:servicemanager_1Binding">
<soap:address location="http://localhost/servicemanager/1"/>
</port>
</service>
And I don't know why it returns a "localhost" there - is zeep using that for its call? Then I would understand why permanent errors occur.
Any hints?
To change the endpoint address I use it this way:
client.service._binding_options['address'] = 'https://mynewaddress.com/service.wsdl'
As always, after days of searching, in the moment I ask at Stackoverflow, the answer came up through other channels.
If anyone has the same problems, here is the solution. My server provides me with the WSDL file, like said above:
<service name="ServiceManager">
<port name="servicemanager_1" binding="tns:servicemanager_1Binding">
<soap:address location="http://localhost/servicemanager/1"/>
</port>
</service>
And there it stands: localhost. Zeep (IMHO correctly) uses that service endpoint to communicate then with the server.
What I did for testing: I SSH-tunnelled the ports 80/443 to localhost, so zeep thought it talked to localhost.
And Shazaam, it worked.
So my server was the culprit - too bad I can't change that, as I have no control over it.
But now a workaround is possible.