When trying to upload a python custom package to our internal registry its failing with the following error.
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='custom-nexus.com’, port=443): Max retries exceeded with url: /repository/pypi-internal/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
But the same url works fine when accessed through curl. Using curl -v option found that the ca-bundle used was in the location /etc/pki/tls/certs/ca-bundle.crt
So tried providing this in the twine upload command using the --cert option. But then also its not working and failing with the same error.
twine upload --config-file .pypirc --cert /etc/pki/tls/certs/ca-bundle.crt -r pypi dist/*
Python version used is 3.6. Please find below the list of library versions.
certifi==2020.4.5
setuptools==46.1.3
wheel==0.34.2
twine==3.1.1
pyOpenSSL==19.1.0
For some reason it didnt work for me even after providing the certificate using --cert option. Might be some issue with the certificate. Then I came across the below hack to skip ssl verification in python requests library
Disable Python requests SSL validation for an imported module
(For anyone who doesn't know, TWINE under the hood also uses python requests library)
Following this, I tried the below command and it worked!
export CURL_CA_BUNDLE="" && twine upload ...
Related
I'm working on a DoFn that writes to Elastic Search App Search (elastic_enterprise_search.AppSearch). It works fine when I run my pipeline using the DirectRunner.
But when I deploy to DataFlow the elasticsearch client fails because, I suppose, it can't access a certificate store:
File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket
context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
FileNotFoundError: [Errno 2] No such file or directory
Any advice on how to overcome this sort of problem? I'm finding it difficult to get any traction on how to solve this on google.
Obviously urllib3 is set up properly on my local machine for DirectRunner. I have "elastic-enterprise-search" in the REQUIRED_PACKAGES key of setup.py for my package along with all my other dependencies:
REQUIRED_PACKAGES = ['PyMySQL', 'sqlalchemy',
'cloud-sql-python-connector', 'google-cloud-pubsub', 'elastic-enterprise-search']
Can I package certificates up with my pipeline? How? Should I look into creating a custom docker image? Any hints on what it should look like?
Yes, creating a custom container that has the necessary credentials in it would work well here.
I'm using Python 3.9.5 and PyMongo 3.11.4. The version of my MongoDB database is 4.4.6. I'm using Windows 8.1
I'm learning MongoDB and I have a cluster set up in Atlas that I connect to. Whenever I try to insert a document into a collection, a ServerSelectionTimeoutError is raised, and inside its parentheses there are several [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate.
Troubleshooting TLS Errors in the PyMongo docs weren't too much help as they only provided tips for Linux and macOS users.
It's worth mentioning that if I set tlsAllowInvalidCertificates=True when initializing my MongoClient, everything works fine. That sounds insecure, and while I am working on a small project, I would still like to develop good habits and not override any security measures in place, so I'm hoping there is an alternative to that.
From all the searching I've done, I'm guessing that I'm missing certain certificates, or that Python can't find them. I've looked into the certifi package, but this part of the docs makes it seem that should only be necessary if I'm using Python 2.x, which I'm not.
So yeah, I'm kind of stuck right now.
Well, I eventually decided to install certifi and it worked.
client = MongoClient(CONNECTION_STRING, tlsCAFile=certifi.where())
Wish the docs were a bit clearer on this, but maybe I just didn't look hard enough.
In Flask server I solved by using:
import certifi
app = Flask(__name__)
app.config['MONGO_URI'] =
'mongodb+srv://NAME:<PWD><DBNAME>.9xxxx.mongodb.net/<db>? retryWrites=true&w=majority'
mongo = PyMongo(app,tlsCAFile=certifi.where())
collection_name = mongo.db.collection_name
By default, pymongo relies on the operating system’s root certificates.
You need to install certifi:
pip install certifi
It could be that Atlas itself updated its certificates or it could be that something on your OS changed. “certificate verify failed” often occurs because OpenSSL does not have access to the system’s root certificates or the certificates are out of date. For how to troubleshoot see TLS/SSL and PyMongo — PyMongo 3.12.0 documentation 107.
So try:
client = pymongo.MongoClient(connection, tlsCAFile=certifi.where())
This happens in django as well just add the above code to your settings.py in Django:
DATABASE = {
'default': {
'ENGINE': 'djongo',
"CLIENT": {
"name": <your_database_name>,
"host": <your_connection_string>,
"username": <your_database_username>,
"password": <your_database_password>,
"authMechanism": "SCRAM-SHA-1",
},
}
}
But in host you may get this issue:
"pymongo.errors.ServerSelectionTimeoutError:"[SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get
local issuer certificate (_ssl.c:997)
So for this you can add:
"mongodb+srv://sampleUser:samplePassword#cluster0-gbdot.mongodb.net/sampleDB??ssl=true&ssl_cert_reqs=CERT_NONE&retryWrites=true&w=majority"
Add
ssl=true&ssl_cert_reqs=CERT_NONE
after db name of your url string works fine
"mongodb+srv://username:Password#cluster0-gbdot.mongodb.net/DbName?**ssl=true&ssl_cert_reqs=CERT_NONE**&retryWrites=true&w=majority"
I saw an answer that worked for me, it appears i had not yet installed the python certificates on my mac, so from the following path i went and installed it
/Applications/Python 3.10/Install Certificates.command
Only change the version of your python, after that everything, worked fine for me
PS: I had been trying to solve the problem for half a day, I even asked ChatGPT
Step 1:
pip install certifi
Step 2:
client = pymongo.MongoClient(connection, tlsCAFile=certifi.where())
I am a novice programmer so pardon my mistakes. I have written the below code to verify a list of Websites are still active and all my work is based off this problem statement.
The script is able to check most sites but stumbled with below error for https://precisionit.net/
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>
The above URL opens fine in Firefox and Chrome but fails to open in Python code. I have updated certifi and used it in my code as suggested by many folks but the error would not go away.
I am using Conda Python Env and I also executed the below command
conda install -c conda-forge certifi
There were multiple posts that suggested running "Install Certificates.command" which does not apply to Conda Python so I downloaded Python 3.9 installer and executed "Install Certificates.command" and executed the script with Python 3.9 yet no luck. I feel the issue is that even with latest version of certifi the sites certificate is not validated. Although certifi page says the list is based off Mozilla’s root certificates I guess it's not an exact replica which is why Firefox is able to open the site. Not sure if my understanding makes sense and will be glad to be corrected.
Pasting my script below. I am not sure what else needs to be done to fix the issue, kindly advise.
import urllib.request
import sys
import certifi
import ssl
def checkURL(url):
try:
hdr = { 'User-Agent' : 'Mozilla/79.0 (Windows NT 6.1; Win64; x64)' }
req=urllib.request.Request(url,headers=hdr)
r = urllib.request.urlopen(req,timeout=100,context=ssl.create_default_context(cafile=certifi.where()))
except Exception as e:
#print(r.read())
print('Failed Connecting to Website')
print(e)
return(1)
print(r.status)
finalurl = r.geturl()
if r.status==200:
print(finalurl)
return(0)
else:
print("Website Not Found")
return(2)
checkURL('https://precisionit.net/')
I had a similar problem, and this is how I solved it.
First, check who the issuer of the site certificate is. You can do this in many ways (check in the browser, connect using openssl ...).
Easiest is probably to just go to https://www.digicert.com/help/ and search for https://precisionit.net.
You are likely missing Sectigo RSA Domain Validation Secure Server CA. Just go to their site (https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA01N000000rfBO) and download it.
Then get the location of the cacert.pem file where your certificates are saved with certifi.where(), and simply add the contents of the certificate you downloaded to said file.
The certificate should be in form
-----BEGIN CERTIFICATE-----
... some base64 encoded stuff ...
-----END CERTIFICATE-----
first : save the site public key as base74
second: add code for verfify with your saved file.
enter image description here
with requests.Session() as s:
CC_host = 'https://precisionit.net'
first_page = s.get(CC_host,verify='./theSiteCert.cer')
html = first_page.text
print(html)
Reference: How does one specify the equivalent of `--proxy-headers` curl argument into requests?
I am a newbie vis-a-vis Python.
I have a requirement, where a request to a destination(webpage) must go through a proxy server.
I need to pass headers to the "Proxy server" (same as --proxy-header of curl)
Need to add an SSL certificate (a '.cer' file) to read the passed headers to the proxy server(a 'Man In the Middle' scenario) on CONNECT.
The curl equivalent of my requirement is as follows:
curl -k --verbose --cacert /proxy/cert/folder/proxy-certificate.cer --proxy-header "header1: value1" --proxy 'http://localhost:8080/' 'https://destination.com'
I did come across a similar example How does one specify the equivalent of `--proxy-headers` curl argument into requests?. But I am unsure how to incorporate this with an SSL certificate.
My Code:
proxyheaders = { 'http://localhost:9090/': { 'header1': 'value1' } }
class ProxyHeaderAwareHTTPAdapter(requests.adapters.HTTPAdapter):
def proxy_headers(self, proxy):
if proxy in proxyheaders:
return proxyheaders[proxy]
else:
return None
s = requests.Session()
s.mount('http://', ProxyHeaderAwareHTTPAdapter())
s.mount('https://', ProxyHeaderAwareHTTPAdapter())
URL = "https://stackoverflow.com/"
cert_file_path = "/Path/to/certificate/proxy-certificate.cer"
try:
s.get(URL, verify=cert_file_path)
except Exception as e:
print(e)
I get the following error:
HTTPSConnectionPool(host='stackoverflow.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
THIS IS NOT A SOLUTION:
When i usually encounter certificate/verification errors, i just force it to not verify the certificate using the code below:
conda config --set ssl_verify false
Note that this is not usually recommended and i usually do it temporarily until i finish either running spicific script or downloading a library or so. If you want to try this and if it works for you, remember to turn it back on once done using the code below:
conda config --set ssl_verify true
This question already has answers here:
Getting 405 error while trying to download nltk data
(2 answers)
Closed 5 years ago.
I am facing some problem for accessing nltk data. I have tried nltk.download(). The gui page has come with HTTP Error 403: Forbidden error. I have also try to install from command line which is provided here.
python -m nltk.downloader all
and get this error.
C:\Python36\lib\runpy.py:125: RuntimeWarning: 'nltk.downloader' found in sys.modules after import of package 'nltk', but prior to execution of 'nltk.downloader'; this may result in unpredictable behaviour warn(RuntimeWarning(msg)) [nltk_data] Error loading all: HTTP Error 403: Forbidden.
I also go through How do I download NLTK data? and Failed loading english.pickle with nltk.data.load.
The problem is coming from the nltk download server. If you look at the gui's config, it's pointing to this link
https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml
If you access this link in the browser, you get this as a message :
Error 403 Forbidden.
Forbidden.
Guru Mediation:
Details: cache-lcy1125-LCY 1501134862 2002107460
Varnish cache server
So, I was going to file an issue on github, but someone else already did that here : https://github.com/nltk/nltk/issues/1791
A workaround was suggested here: https://github.com/nltk/nltk/issues/1787.
Based on the discussion on github:
It seems like the Github is down/blocking access to the raw content on
the repo.
The suggested workaround is to manually download as follows:
PATH_TO_NLTK_DATA=/home/username/nltk_data/
wget https://github.com/nltk/nltk_data/archive/gh-pages.zip
unzip gh-pages.zip
mv nltk_data-gh-pages/ $PATH_TO_NLTK_DATA
People also suggested using an laternative index as follows:
python -m nltk.downloader -u https://pastebin.com/raw/D3TBY4Mj punkt
Go to /nltk/downloader.py
And change the default url:
DEFAULT_URL = 'http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml'
to
DEFAULT_URL = 'http://nltk.github.com/nltk_data/'
For me the best solution is:
PATH_TO_NLTK_DATA=/home/username/nltk_data/
wget https://github.com/nltk/nltk_data/archive/gh-pages.zip
unzip gh-pages.zip
mv nltk_data-gh-pages/ $PATH_TO_NLTK_DATA
link
Alternative solution is not working for me
python -m nltk.downloader -u https://pastebin.com/raw/D3TBY4Mj punkt