Download documents from https site Error: sslv3 alert handshake failure - python-3.x

Anaconda - Python 3.6
OpenSSL 1.0.2
Operating System: Windows 7
Phase 1 (Completed): Using selenium: launched, navigated, and extracted various data elements including a table from site. Extracted Hyperlinks contained in table that are direct links to documents.
Phase 2: Taking extracted hyperlink from table I need to download the files to a specified folder on the shared drive.
Tried:
import urllib.request
url = 'tts website/test.doc'
urllib.request.urlretrieve(url,'C:\Users\User\Desktop\')
Error I get is sslv3 alert handshake failure
With the site opened, I have clicked on the Lock icon and clicked "Install Certificate". I have saved the certificate to my "Trusted Root Certification Authorities" in the Certificate store.
I can see the certificate name (when i installed certificate) from the above step in the 58 CA Certificates shown by running the following code:
import socket
import ssl
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_REQUIRED
context.load_default_certs()
ca_certs = context.get_ca_certs()
print('{} CA Certificates loaded: ' .format(len(ca_certs)))
for cert_dict in ca_certs:
print(cert_dict['subject'])
print()
I can't figure out how to secure a SSL connection to the website/server in order to download the file from each of the hyperlinks?? This website uses Single Sign On(SSO) and automatically logins me in when I first launch the website.
I have tried to use server server.net 443 to connect to server, but can't seem to get the scripting right to connect and retrieve the document.
I have connected directly to the server and abstracted the certificate details shown here:
HOST, PORT = server.net, 443
ctx = ssl.create_default_context()
s = ctx.wrap_socket(socket.socket(), server_hostname=HOST)
c.connect((HOST, PORT))
cert = s.getpeercert()
print(cert)
When i run urlretrieve i am still getting the same error: handshake. When reviewing my ca certificates i see there is a Personal certificate for my Windows login (username) listed there, that must be how it is automatically logging me in using SSO. How do i take all of this information, connect to the website using my SSO, and retrieve the documents?
Latest UPDATE:
I am finding pycurl to be promising, however I feel like I need a little assistance making a few tweaks to get it working.
import pycurl
fp = open('Test.doc','wb')
curl = pycurl.Curl()
curl.setopt(pycurl.URL, url) # see url link to go to word doc
curl.setopt(pycurl.FOLLOWLOCATION, 1)
curl.setopt(pycurl.MAXREDIRS, 5)
curl.setopt(pycurl.CONNECTTIMEOUT,30)
curl.setopt(pycurl.TIMEOUT, 300)
try:
curl.setopt(pycurl.WRITEDATA, fp)
curl.perform()
except:
import traceback
traceback.print_exc(file=sys.stderr)
sys.stderr.flush()
curl.close()
fp.close()
This code yields no error, however the created word doc contains an error displaying a print screen of the log on page of the website.
Main Problem: HTTPS connection using Single Signon connection behind a corporate network proxy server.
I have been trying to get this to work to validate cacert, but I have been getting this error message now:
curl.setopt(pycurl.SSL_VERIFYPEER, 1)
curl.setopt(pycurl.SSL_VERIFYPEER, 2)
curl.setopt(pycurl.CAINFO, certifi.where())
but now i am getting ERROR: 51, CERT_TRUST_IS_UNTRUSTED_ROOT
How do i add proxy if that is causing the error? and Secondly, how do i attach the ca certificate file directly?

Related

Specify SNI server_hostname when performing request with asyncio/aiohttp

Hello fellow developers ! I'm stuck in a corner case and I'm starting to be out of hairs to pull... Here is the plot :
load-balancer.example.com:443 (TCP passthrough)
/\
/ \
/ \
/ \
s1.example.com:443 s2.example.com:443
(SSL/SNI) (SSL/SNI)
The goal is to stress-test the upstreams s1 and s2 directly using aiohttp with certificate-validation enable. Since the load-balancer does not belong to me I don't want to do the stress-test over it.
the code is not supposed to run on other platforms than GNU Linux with at least Python-v3.7 (but I can use any recent version if needed)
all servers serve a valid certificate for load-balancer.example.com
openssl validates the certificate from the upstreams when using openssl s_connect s1.example.com:443 -servername load-balancer.example.com
cURL needs curl 'https://load-balancer.example.com/' --resolve s1.example.com:443:load-balancer.example.com and also validates successfully
I am able to launch a huge batch of async ClientSession.get requests on both upstreams in parallel but for each request I need to somehow tell asyncio or aiohttp to use load-balancer.example.com as server_hostname, otherwise the SSL handshake fails.
Is there an easy way to setup the ClientSession to use a specific server_hostname when setting up the SSL socket ?
Does someone have already done something like that ?
EDIT : here is the most simple snippet with just a single request :
import aiohttp
import asyncio
async def main_async(host, port, uri, params=[], headers={}, sni_hostname=None):
if sni_hostname is not None:
print('Setting SNI server_name field ')
#
# THIS IS WHERE I DON'T KNOW HOW TO TELL aiohttp
# TO SET THE server_name FIELD TO sni_hostname
# IN THE SSL SOCKET BEFORE PERFORMING THE SSL HANDSHAKE
#
try:
async with aiohttp.ClientSession(raise_for_status=True) as session:
async with session.get(f'https://{host}:{port}/{uri}', params=params, headers=headers) as r:
body = await r.read()
print(body)
except Exception as e:
print(f'Exception while requesting ({e}) ')
if __name__ == "__main__":
asyncio.run(main_async(host='s1.example.com', port=443,
uri='/api/some/endpoint',
params={'apikey': '0123456789'},
headers={'Host': 'load-balancer.example.com'},
sni_hostname='load-balancer.example.com'))
When running it with real hosts, it throws
Cannot connect to host s1.example.com:443 ssl:True
[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] '
certificate verify failed: certificate has expired (_ssl.c:1131)')])
Note that the error certificate has expired indicates that the certificate proposed to the client is the default certificate since the SNI hostname is s1.example.com which is unknow by the webserver running there.
When running it against the load-balancer it works just fine, the SSL handshake happens with the upstreams which serve the certificate and everything is valid.
Also note that
sni_callback does not help since it is called after the handshake has started and the certificate was received (and at this point server_hostname is a read-only property anyway)
it does not seem to be possible to set server_hostname when creating an SSLContext allthough SSLContext.wrap_socket does support server_hostname but I was not able to make that work
I hope someone knows how to fill the comment block in that snippet ;-]

Python requests module SSLError(CertificateError("hostname 'x.x.x.x' doesn't match 'x.x.x.x")))

I'm using the "requests" module for getting JSON from my web service, using the next code:
import requests
import SSL
# With or without this line of code below, the output is the same
ssl.match_hostname = lambda cert, hostname: True
response = requests.get("MY_URL", cert=("client.pem", "client-key.pem"), verify="CAcert.cer")
When the SSL step seems to fail with the following message:
HTTPSConnectionPool(host='x.x.x.x', port=443): Max retries exceeded with url: {WEBSERVICE_URL_PATTERN} (Caused by SSLError(CertificateError("hostname 'x.x.x.x' doesn't match 'x.x.x.x'")))
I'm using Python 3.10.5 with the latest version of the "requests" module.
Does anyone know what could cause this kind of error and how to fix it?
I assume you've redacted actual names which are in fact different, because if you really did have a host named x.x.x.x using a cert with the same name it would match (unless it wasn't really the same because the CA, or a potentially-bogus 'subject'/'subscriber', used lookalike characters).
From the documentation of match_hostname
Changed in version 3.7: The function is no longer used to TLS connections. Hostname matching is now performed by OpenSSL. ...
Deprecated since version 3.7.
At the python.ssl level, or http.client or urllib.requests, you can still turn off only hostname checking with check_hostname=False in the SSLContext. However AFAIK requests doesn't give you access to the SSL level except for setting the cert(s) as you do or the sledgehammer option of turning off all verification with verify=False.
If at all possible, you should try to use a hostname and a host cert that do match. Note changing either the name you request or the cert can accomplish this.
The problem was solved, using a Subject Alternative Name (SAN) for the server, with a value of its own IP address.
I've found out that we use Simple-CA, and the request of getting a signed certificate from it was with a Common Name (CN), when we don't have a domain name.
After changing the signing action to SAN instead of CN, the problem was solved.
Thanks for the helpers!

Twilio rejecting LetsEncrypt Cert with "Certificate Invalid" error 11237 using Twisted Web Server

I am running python3 (version 3.6.9) Twisted (version 18.4.0) on an Ubuntu 18.04 server. This server is used for webhooks for Twilio. The webhooks work fine over http. I installed a LetsEncrypt cert, and the LetsEncrypt ssl cert works fine for serving https over a FireFox browser.
However, when I point twilio to the https version of the webhook, I get the following error in the twilio debugger console:
Error - 11237
Certificate Invalid - Could not find path to certificate
Twilio tried to validate your SSL certificate but was unable to find it in our certificate store. Possible Causes
You are using a self signed certificate.
The certificate authority you are using is not on our list of approved certificate authorities.
Your certificate chain is incomplete and requires an additional download.
Possible Solutions
Do not use a self signed certificate.
Concatenate your certificate chain so that no additional download is required.
Twilio uses CAs that are approved by Mozilla, you can find the full list here.
For testing purposes you can disable SSL Certificate Validation in Console.
If I disable SSL Certificate Validation in Console as suggested by Twilio, the webhooks work. I do not want to disable SSL Certificate Validation.
Here is a self contained sample of the code I am running on the server:
import sys
from klein import Klein
from twisted.web.server import Site
from twisted.internet import reactor
from twisted.internet.endpoints import serverFromString
from twisted.python.log import startLogging
from [redacted] import get_data_folder_location
startLogging(sys.stdout)
klein_app = Klein()
path_to_letsencrypt_keys = get_data_folder_location()
#lensencrypt keys have been copied locally from /etc/letsencrypt/live/domain and chowned from root to local group:user
endpoint_description = "ssl:443:privateKey={0}/privkey.pem:certKey={0}/fullchain.pem".format(path_to_letsencrypt_keys)
klein_resource = klein_app.resource()
serverFromString(reactor, endpoint_description).listen(Site(klein_resource))
reactor.run()
Here is the log output from the self contained sample:
Note: the 404 on the last line of the log is me hitting the site over ssl with FireFox, which demonstrates FireFox (and therefore Mozilla) is Ok with the letsencrypt ssl cert
2021-04-26 17:54:58+0000 [-] Log opened.
2021-04-26 17:54:58+0000 [-] Site (TLS) starting on 443
2021-04-26 17:54:58+0000 [-] Starting factory <twisted.web.server.Site object at 0x7fe3c57aa048>
2021-04-26 17:55:18+0000 [-] "redacted" - - [26/Apr/2021:17:55:18 +0000] "GET / HTTP/1.1" 404 233 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0"
And finally, here are 2 screen shots of the Qualys SSL Report
My question: How can I get Twilio to accept my LetsEncrypt cert?
It looks like Twisted has problems loading the fullchain.pem.
You'll need to manually load the chain as described here.
from OpenSSL import crypto
from twisted.internet import ssl
privkey=open('{0}/privkey.pem'.format(path_to_letsencrypt_keys), 'rt').read()
certif=open('{0}/cert.pem'.format(path_to_letsencrypt_keys), 'rt').read()
chain=open('{0}/chain.pem'.format(path_to_letsencrypt_keys), 'rt').read()
privkeypyssl=crypto.load_privatekey(crypto.FILETYPE_PEM, privkey)
certifpyssl=crypto.load_certificate(crypto.FILETYPE_PEM, certif)
chainpyssl=[crypto.load_certificate(crypto.FILETYPE_PEM, chain)]
contextFactory=ssl.CertificateOptions(privateKey=privkeypyssl, certificate=certifpyssl, extraCertChain=chainpyssl)

Response as undefined after an API call

I am using the request library to fetch a response from an API, everything was working fine but as of recent, I have started to get local issuer certificate error from the API although the same code is working on my colleague's system.
I have already tried updating the node version and all the version changes regarding the request package of node
The exact error is:
unable to get local issuer certificate node
Here is the error snippet in the console
Could u please console only response..After that u can easily see that there is statusCode is available or not.
Or if u get the statusCode then check its defined or not.
Ex: if(res.statusCode)
console.log("print status code", this.res.statusCode);
This is an SSL error and yo need to make sure that your server's certificate is valid.
With your claim that it used to work - but it stopped working recently with the SSL error message, the possibilities are that either your servers certificate is expired in between or the Sectigo AddTrust External CA Root Expiring May 30, 2020.
https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT
Updating the certificate is outside the scope of this answer. You may have to do it yourself or contact the admin.

OpenSSL SSL:CERTIFCATE_VERIFY_FAILED

I'm trying to issue a get request to a https url which has a restful api that will respond back with an image that I wish to save off. I'm using python 3.5 on windows and the requests lib to make the https request.
I'm on a network in which every user is issued a pcks#12(.p12) cert and must install it in their browser to visit pretty much any of the webpages on the network including the site I'm targeting. Through my browser I am able to access the target website and download the image as desired. However, doing so programmatically has been a challenge.
I keep getting a SSL: CERTIFICATE_VERIFY_FAILED error. I can't copy the whole trace (its on a different computer) but below is a gist of what gets spit out in python console.
ssl.py, line 628, in do_handshake ssl.SSLError [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
requests\adapters.py, line 447, in send*
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
Here is my code. I'm converting from .p12 to .pem because I believe that is what the requests library requires.
Password = bytes(r'blahblahblah','utf-8')
P12 = OpenSSL.crypto.load_pkcs12(open("C:/Temp/certs/mypki.p12",'rb').read(),password)
key = OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYP_PEM, p12.get_privatekey())
mycert = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYP_PEM, p12.get_certificate())
#len(get_ca_certificates) return 2 so I know I've got all of them
cert1 = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYP_PEM, p12.get_ca_certificates()[0])
cert2 = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYP_PEM, p12.get_ca_certificates()[1])
At one point I was getting errors saying that I had a key mismatch. This code led me to the discovery that the get_certificate() method returns a certificate not found in the tuple returned by the get_ca_certificate() method. Which was the matching public key to my private key. I'm just including this so people know I got a matching key pair.
k = OpenSSl.crypto.Load_privatekey(OpenSSL.crypto.FILETYPE_PEM,key)
c = OpenSSl.crypto.Load_certificate(OpenSSL.crypto.FILETYPE_PEM,mycert)
context = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_1_METHOD)
context.use_privatekey(k)
context.use_certificate(c)
try:
context.check_privatekey()
return True
except OpenSSL.SSL.Error:
return FALSE
I've decided to create a .pem file with my key and certs all in one file. I've also tried separating the key from certs in different files but get the same error when doing so. I'm not 100% clear on the differences between .pem,.crt, etc and when to use which but I think I'm doing it correctly...?
pem_file = open("C:/Temp/Certs/cert.pem","wb")
pem_file.write(key)
pem_file.write(mycert)
pem_file.write(cert1)
pem_file.write(cert2)
pem_file.close()
response = request.get("https://somewebsite.com",proxies={"http": None,
"https:": None}, cert=("C:/Temp/Certs/cert.pem"))
For the sake of completeness this is the code I used when I tried writing out the key and certs to separate files.
response = request.get("https://somewebsite.com",proxies={"http": None,
"https:": None}, cert=("C:/Temp/Certs/cert.crt" , "C:/Temp/Certs/key.key"))
So the issue ended up being I was using the wrong pcks#12 file. I was given a digital_signature.p12 and a encryption_key.p12 (or something like that) and the website was expecting the private key and certs from the digital_signature.p12 instead of the other one I was using.

Resources