I have already tried the following and seems it is not related:
Upgrading certifi using: pip3 install --upgrade certifi
Disabling pythons http check using: export PYTHONHTTPSVERIFY=0 and then running the script
Updating and exporting the correct env. variable:
sudo update-ca-certificates --fresh
export SSL_CERT_DIR=/etc/ssl/certs
I'm using Python3.6.9
This is the output of the ssl paths:
> python3 -c 'import ssl; print(ssl.get_default_verify_paths())'
> DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')
The script I'm trying to run is a simple post request using requests:
r = requests.post(url, data=data, headers=headers)
Using data and headers that worked perfectly well until the change of certificates.
The script is trying to connect to an API hosted on the same server
Here is the full error output:
Traceback (most recent call last):
File "/home/montredo/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/montredo/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/home/montredo/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/home/montredo/.local/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect
ssl_context=context,
File "/home/montredo/.local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/lib/python3.6/ssl.py", line 817, in __init__
self.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
Has anyone had this issue still happen after trying all above solutions?
Related
I admit that I'm not a sysadmin by training, so I don't "get" some things that come up. This is one of those times.
I was setting up a server for a project. Stupidly, because the last time I'd been shoved into a sysadmin role I was working with Centos7, I opted for that OS out of familiarity. When installing Python 3.10+ (I specifically need 3.10+ for some packages I intend to use), openssl was an issue; after a certain version, 1.0.2 (the last shipped version with Centos7) is not an option, and I had to spot-setup 1.1.1k for this Python3 installation to work.
So I have this Python3.10 installation working:
python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.1.1k 25 Mar 2021
Now, I need to run queries against a remote postgres database's API using their Python3 sdk. I get openssl problems when I do. Minimum to reproduce:
from algosdk.v2client import indexer
indexer_url="https://mainnet-idx.algonode.cloud"
indexer_client = indexer.IndexerClient(indexer_token='', indexer_address=indexer_url)
indexer_client.search_assets(asset_id=0)
This fails:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/usr/local/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/usr/local/lib/python3.10/http/client.py", line 1454, in connect
self.sock = self._context.wrap_socket(self.sock,
File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.10/site-packages/algosdk/v2client/indexer.py", line 801, in search_assets
return self.indexer_request("GET", req, query, **kwargs)
File "/usr/local/lib/python3.10/site-packages/algosdk/v2client/indexer.py", line 74, in indexer_request
resp = urlopen(req)
File "/usr/local/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.10/urllib/request.py", line 519, in open
response = self._open(req, data)
File "/usr/local/lib/python3.10/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.10/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/local/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Okay. My initial impression was that this could be a firewall issue, so I put my firewall down. No change. Put the firewall back up. (_ssl.c:997) seems to mean "Unable to get local issuer certificate", and trying to find a resolution brings me to sites like this one.
This is where I start to lose my patience, because I just don't understand the issue. I've only handled ssl certificates in the context of setting up a website; I put up a site, the site needs a signature specifying certain things about it (which earns it an s at the end of http), so I obtain a certificate it and tie it to the site. That's the extent of my understanding. I'm not talking about a website here. Yes, I'll be hosting a website on this server, but I'm literally just talking about a Python3 script. The error above seems to indicate that the issue is that I don't have an ssl certificate. Why does that matter? This code runs on my local machine, and I never had to obtain an ssl certificate.
tl;dr: help a noob out, please. I don't understand the problem, and that also means that I don't know what to look up to fix it.
I'm posting this message since I fixed the problem with the same error codes. It may help others who encounter a similar issue.
I used to run the py script for sending emails. On another side, the mail server Cert got renewed and all of the sudden my script failing to connect to the server.
bash # python3 email_send.py
Traceback (most recent call last):
File "emailsend.py", line 143, in <module>
server.starttls(context=context)
File "/usr/lib/python3.8/smtplib.py", line 774, in starttls
self.sock = context.wrap_socket(self.sock,
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)
I solved this issue by adding the last two lines.
context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
Whenever I try to docker-compose up any project I get the following error.
I've tried with and without sudo
I am only having this issue on this machine. I am able to run the same containers on my Mac and Amazon WorkSpace.
(myslabs) [austin#localhost myslabs]$ sudo docker-compose up
[sudo] password for austin:
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1252, in request
File "http/client.py", line 1298, in _send_request
File "http/client.py", line 1247, in endheaders
File "http/client.py", line 1026, in _send_output
File "http/client.py", line 966, in send
File "docker/transport/unixconn.py", line 43, in connect
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 449, in send
File "urllib3/connectionpool.py", line 727, in urlopen
File "urllib3/util/retry.py", line 403, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1252, in request
File "http/client.py", line 1298, in _send_request
File "http/client.py", line 1247, in endheaders
File "http/client.py", line 1026, in _send_output
File "http/client.py", line 966, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/api/client.py", line 205, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 228, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bin/docker-compose", line 3, in <module>
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 123, in perform_command
File "compose/cli/command.py", line 69, in project_from_options
File "compose/cli/command.py", line 132, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 188, in __init__
File "docker/api/client.py", line 213, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
[142201] Failed to execute script docker-compose
I've not been able to find any articles about any similar issues.
I am using Centos 8 running inside of a VirtualBox on Windows.
(myslabs) [austin#localhost myslabs]$ docker --version
Docker version 19.03.13, build 4484c46d9d
(myslabs) [austin#localhost myslabs]$ docker-compose --version
docker-compose version 1.27.3, build 4092ae5d
Any help would greatly be appreciated.
I also encountered the same error as you,My problem is that docker does not start.
check docker status systemctl status docker
systemctl start docker
docker-compose up try again
systemctl enable docker to start on boot
If you're on a Mac, it may mean that Docker itself isn't running. I had rebooted my Mac and Docker wasn't set to automatically launch at login. You can set this as a Docker preference.
If nothing works, run:
sudo chmod +x /usr/local/bin/docker-compose
sudo docker-compose up -d
If On WSL2 ensure that the docker service has been started, use sudo service docker status to check. If it is not running, use the command sudo service docker start to get it running before sudo docker-compose up
Sometimes docker service is stopped. You can check it and restart.Normally happens if error iscontainer. can pruine and start again.
Please restart docker containers and it will work fine.
This issue comes mainly in the case of windows. restart your docker container.
then run the docker command from your terminal
I do not understand how to use urllib3 or requests to connect to an https web site. This is driving me nuts. I have installed certifi and I see the default .pem file it provides. I have tried to set the requests.verify option to requests to every .pem and .crt file on the machine my script runs on [I am not an admin on this device]. I get nothing but errors.
I switched to using urllib3 and am now getting:
H:\Projects\MyScraper\venv\Scripts\python.exe H:/Projects/MyScraper/MyScraper.py
Traceback (most recent call last):
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connection.py", line 344, in connect
ssl_context=context)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\util\ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 814, in __init__
self.do_handshake()
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "C:\Program Files (x86)\Python36-32\lib\ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "H:/Projects/MyScraper/MyScraper.py", line 15, in <module>
raw_html = HTTP.request('GET', 'https://portal.xsede.org/course-calendar/')
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\request.py", line 68, in request
**urlopen_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\poolmanager.py", line 323, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
**response_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
**response_kw)
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
**response_kw)
[Previous line repeated 6 more times]
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "H:\Projects\MyScraper\venv\lib\site-packages\urllib3\util\retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='portal.xsede.org', port=443): Max retries exceeded with url: /course-calendar/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
Process finished with exit code 1
My code looks like:
#!/home/me/virtualenv/python3.6/3.6/bin/python
import certifi
import urllib3
from bs4 import BeautifulSoup
HTTP = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED',
ca_certs=certifi.where(),
retries=10
)
raw_html = HTTP.request('GET', 'https://portal.xsede.org/course-calendar/')
html = BeautifulSoup(raw_html, 'html.parser')
It blows up on the raw_html = HTTP.request(... line. Ideas?
Edit
Huh, this has something to do with my target host. If I go to google.com then several of my pem/crt files work.
The problem is, you are using wrong certificate to make request.
you can run this command to verify which certificate is used when any request is made, and then use that certificate in your request,
openssl s_client -showcerts -connect google.com:443
Please also make sure that you are passing verify the path to CA_BUNDLE file or directory with certificates of trusted CAs.
This list of trusted CAs can also be specified through the REQUESTS_CA_BUNDLE environment variable.
If this doesn't work out for you can explicitly merge the environment settings into your session,
When you are using the prepared request flow, keep in mind that it
does not take into account the environment. This can cause problems if
you are using environment variables to change the behaviour of
requests. For example: Self-signed SSL certificates specified in
REQUESTS_CA_BUNDLE will not be taken into account. As a result an SSL:
CERTIFICATE_VERIFY_FAILED is thrown. You can get around this behaviour
by explicity merging the environment settings into your session:
from requests import Request, Session
s = Session()
req = Request('GET', url)
prepped = s.prepare_request(req)
# Merge environment settings into session
settings = s.merge_environment_settings(prepped.url, None, None, None, None)
resp = s.send(prepped, **settings)
print(resp.status_code)
I'm getting an SSLError while submitting a request against https://myaccount.xcelenergy.com (and am not nearly enough of an SSL expert to understand why). Can you help me see what the issue is (and how to address it)?
Environment
macOS 10.13.2
LibreSSL 2.2.7 (installed via Homebrew)
Python 3.6.4 (installed via Homebrew)
Requests 2.18.4 (installed via pipenv)
Certifi 2017.11.5 (installed via pipenv)
Code
import requests
requests.get('https://myaccount.xcelenergy.com')
Stacktrace
Traceback (most recent call last):
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/urllib3/connectionpool.py", line 595, in urlopen
self._prepare_proxy(conn)
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
conn.connect()
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in __init__
self.do_handshake()
File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='myaccount.xcelenergy.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Users/abach/.local/share/virtualenvs/pyxcel-PG9DCiyE/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='myaccount.xcelenergy.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
What I've Tried (to no avail)
# Specifically calling out local cert chain:
requests.get('https://myaccount.xcelenergy.com', verify='/usr/local/etc/openssl/cert.pem')
# Using Certifi's old chain:
requests.get('https://myaccount.xcelenergy.com', verify=certifi.old_where())
There is currently a known issue with OSX's Python3.6 and OpenSSL that might be related to what you're seeing. I was able to fix it by installing certifi, then making a symlink in OpenSSL's directory. All of that can be done for you by running the following from bash:
Applications/Python 3.6/Install Certificates.command
See this related question: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:747) on OS X