Accessing https pages with python3 - python-3.x

Question

I do not understand how to use urllib3 or requests to connect to an https web site. This is driving me nuts. I have installed certifi and I see the default .pem file it provides. I have tried to set the requests.verify option to requests to every .pem and .crt file on the machine my script runs on [I am not an admin on this device]. I get nothing but errors.
I switched to using urllib3 and am now getting:
H:\Projects\MyScraper\venv\Scripts\python.exe H:/Projects/MyScraper/MyScraper.py
Traceback (most recent call last):
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connection.py", line 344, in connect
ssl_context=context)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\util\ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 814, in __init__
self.do_handshake()
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "H:/Projects/MyScraper/MyScraper.py", line 15, in <module>
raw_html = HTTP.request('GET', 'https://portal.xsede.org/course-calendar/')
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\request.py", line 68, in request
**urlopen_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\poolmanager.py", line 323, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
**response_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
**response_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
**response_kw)
[Previous line repeated 6 more times]
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\util\retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='portal.xsede.org', port=443): Max retries exceeded with url: /course-calendar/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
Process finished with exit code 1
My code looks like:
#!/home/me/virtualenv/python3.6/3.6/bin/python
import certifi
import urllib3
from bs4 import BeautifulSoup
HTTP = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED',
ca_certs=certifi.where(),
retries=10
)
raw_html = HTTP.request('GET', 'https://portal.xsede.org/course-calendar/')
html = BeautifulSoup(raw_html, 'html.parser')
It blows up on the raw_html = HTTP.request(... line. Ideas?
Edit
Huh, this has something to do with my target host. If I go to google.com then several of my pem/crt files work.

Answer 1

The problem is, you are using wrong certificate to make request.
you can run this command to verify which certificate is used when any request is made, and then use that certificate in your request,
openssl s_client -showcerts -connect google.com:443
Please also make sure that you are passing verify the path to CA_BUNDLE file or directory with certificates of trusted CAs.
This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.
If this doesn't work out for you can explicitly merge the environment settings into your session,
When you are using the prepared request flow, keep in mind that it
does not take into account the environment. This can cause problems if
you are using environment variables to change the behaviour of
requests. For example: Self-signed SSL certificates specified in
REQUESTS_CA_BUNDLE will not be taken into account. As a result an SSL:
CERTIFICATE_VERIFY_FAILED is thrown. You can get around this behaviour
by explicity merging the environment settings into your session:
from requests import Request, Session
s = Session()
req = Request('GET', url)
prepped = s.prepare_request(req)
# Merge environment settings into session
settings = s.merge_environment_settings(prepped.url, None, None, None, None)
resp = s.send(prepped, **settings)
print(resp.status_code)