Python requests equivalent of '--proxy-header' in curl with SSL certification - python-3.x

Reference: How does one specify the equivalent of `--proxy-headers` curl argument into requests?
I am a newbie vis-a-vis Python.
I have a requirement, where a request to a destination(webpage) must go through a proxy server.
I need to pass headers to the "Proxy server" (same as --proxy-header of curl)
Need to add an SSL certificate (a '.cer' file) to read the passed headers to the proxy server(a 'Man In the Middle' scenario) on CONNECT.
The curl equivalent of my requirement is as follows:
curl -k --verbose --cacert /proxy/cert/folder/proxy-certificate.cer --proxy-header "header1: value1" --proxy 'http://localhost:8080/' 'https://destination.com'
I did come across a similar example How does one specify the equivalent of `--proxy-headers` curl argument into requests?. But I am unsure how to incorporate this with an SSL certificate.
My Code:
proxyheaders = { 'http://localhost:9090/': { 'header1': 'value1' } }
class ProxyHeaderAwareHTTPAdapter(requests.adapters.HTTPAdapter):
def proxy_headers(self, proxy):
if proxy in proxyheaders:
return proxyheaders[proxy]
else:
return None
s = requests.Session()
s.mount('http://', ProxyHeaderAwareHTTPAdapter())
s.mount('https://', ProxyHeaderAwareHTTPAdapter())
URL = "https://stackoverflow.com/"
cert_file_path = "/Path/to/certificate/proxy-certificate.cer"
try:
s.get(URL, verify=cert_file_path)
except Exception as e:
print(e)
I get the following error:
HTTPSConnectionPool(host='stackoverflow.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))

THIS IS NOT A SOLUTION:
When i usually encounter certificate/verification errors, i just force it to not verify the certificate using the code below:
conda config --set ssl_verify false
Note that this is not usually recommended and i usually do it temporarily until i finish either running spicific script or downloading a library or so. If you want to try this and if it works for you, remember to turn it back on once done using the code below:
conda config --set ssl_verify true

Related

Python ldap3 lib unable to use Tls unknown error

i have currently have working code in linux which does ldapsearch here is the code
export LDAPTLS_CACERT=ldap.pem
ldapsearch -D 'CN=admuser,OU=Service Accounts,DC=InfoDir,DC=Dev,DC=AAA'
-H ldaps://server:1234 -b OU=People,DC=InfoDir,DC=Dev,DC=AAA -W "
(cn=staffid)"
this asks me password for admuser bind user and gives result. I need to code same in python. Here is my python code
from ldap3 import Server, Connection, Tls
import ssl
cacertfile='ldap.pem'
server = Server(host='hostname',port=1234)
conn = Connection(server,'CN=admuser,OU=Service
Accounts,DC=InfoDir,DC=Dev,DC=AAA','password,auto_bind=True)
print(conn)
this gives fine and gives me default ssl in conn object becuase ldap url has ldaps. However, this does not validate server cert which is not safe. Hence i further update my code to force tls here is the code..
from ldap3 import Server, Connection, Tls
import ssl
cacertfile='ldap.pem'
tls_conf = Tls(ssl.CERT_REQUIRED,ca_certs_file=cacertfile)
server = Server(host='hostname',port=1234,tls_conf)
conn = Connection(server,'CN=admuser,OU=Service
Accounts,DC=InfoDir,DC=Dev,DC=AAA','password,auto_bind=True)
print(conn)
When i run this i get exception
raise LDAPSocketOpenError('unable to open socket', exception_history)
ldap3.core.exceptions.LDAPSocketError:('unable to open socket',....'socket ssl wrapping error: unknown error(_ssl.c:3517..)
I am following the link for my code https://ldap3.readthedocs.io/en/latest/tutorial_intro.html
Please advise.

Python certificate verify failed: unable to get local issuer certificate

I am a novice programmer so pardon my mistakes. I have written the below code to verify a list of Websites are still active and all my work is based off this problem statement.
The script is able to check most sites but stumbled with below error for https://precisionit.net/
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>
The above URL opens fine in Firefox and Chrome but fails to open in Python code. I have updated certifi and used it in my code as suggested by many folks but the error would not go away.
I am using Conda Python Env and I also executed the below command
conda install -c conda-forge certifi
There were multiple posts that suggested running "Install Certificates.command" which does not apply to Conda Python so I downloaded Python 3.9 installer and executed "Install Certificates.command" and executed the script with Python 3.9 yet no luck. I feel the issue is that even with latest version of certifi the sites certificate is not validated. Although certifi page says the list is based off Mozilla’s root certificates I guess it's not an exact replica which is why Firefox is able to open the site. Not sure if my understanding makes sense and will be glad to be corrected.
Pasting my script below. I am not sure what else needs to be done to fix the issue, kindly advise.
import urllib.request
import sys
import certifi
import ssl
def checkURL(url):
try:
hdr = { 'User-Agent' : 'Mozilla/79.0 (Windows NT 6.1; Win64; x64)' }
req=urllib.request.Request(url,headers=hdr)
r = urllib.request.urlopen(req,timeout=100,context=ssl.create_default_context(cafile=certifi.where()))
except Exception as e:
#print(r.read())
print('Failed Connecting to Website')
print(e)
return(1)
print(r.status)
finalurl = r.geturl()
if r.status==200:
print(finalurl)
return(0)
else:
print("Website Not Found")
return(2)
checkURL('https://precisionit.net/')
I had a similar problem, and this is how I solved it.
First, check who the issuer of the site certificate is. You can do this in many ways (check in the browser, connect using openssl ...).
Easiest is probably to just go to https://www.digicert.com/help/ and search for https://precisionit.net.
You are likely missing Sectigo RSA Domain Validation Secure Server CA. Just go to their site (https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA01N000000rfBO) and download it.
Then get the location of the cacert.pem file where your certificates are saved with certifi.where(), and simply add the contents of the certificate you downloaded to said file.
The certificate should be in form
-----BEGIN CERTIFICATE-----
... some base64 encoded stuff ...
-----END CERTIFICATE-----
first : save the site public key as base74
second: add code for verfify with your saved file.
enter image description here
with requests.Session() as s:
CC_host = 'https://precisionit.net'
first_page = s.get(CC_host,verify='./theSiteCert.cer')
html = first_page.text
print(html)

SSL error while uploading python package to custom registry using twine

When trying to upload a python custom package to our internal registry its failing with the following error.
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='custom-nexus.com’, port=443): Max retries exceeded with url: /repository/pypi-internal/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
But the same url works fine when accessed through curl. Using curl -v option found that the ca-bundle used was in the location /etc/pki/tls/certs/ca-bundle.crt
So tried providing this in the twine upload command using the --cert option. But then also its not working and failing with the same error.
twine upload --config-file .pypirc --cert /etc/pki/tls/certs/ca-bundle.crt -r pypi dist/*
Python version used is 3.6. Please find below the list of library versions.
certifi==2020.4.5
setuptools==46.1.3
wheel==0.34.2
twine==3.1.1
pyOpenSSL==19.1.0
For some reason it didnt work for me even after providing the certificate using --cert option. Might be some issue with the certificate. Then I came across the below hack to skip ssl verification in python requests library
Disable Python requests SSL validation for an imported module
(For anyone who doesn't know, TWINE under the hood also uses python requests library)
Following this, I tried the below command and it worked!
export CURL_CA_BUNDLE="" && twine upload ...

Python 3: Authenticate using response-kerberos package

I'm trying to authenticate to a server using the requests_kerberos package, following the instructions here:
https://github.com/requests/requests-kerberos
import requests
from requests_kerberos import HTTPKerberosAuth
kerberos_auth = HTTPKerberosAuth()
r = requests.get(<myserver>, auth=kerberos_auth)
r.text
And here is the response:
'Apache Tomcat/6.0.53 - Error report HTTP Status 401 - Authentication requiredtype Status reportmessage Authentication requireddescription This request requires HTTP authentication.Apache Tomcat/6.0.53'
klist shows that I have a valid TGT.
I have tried setting the principal directly, but that didn't help. I can authenticate using curl:
curl -i -L --negotiate -u : "<server>"
I'm not sure what else to try; everything is happening "behind the scenes" so I don't know what I'm doing wrong.

Getting a WCF service (both host/client) to work on https on Linux with Mono

I have a small test console application that serves as the WCF host and another console application that serves as the client.
The client can reach the host via http, everything works fine so far.
But when switching to https, I get the following error:
Error: System.Net.WebException: Error: SendFailure (Error writing headers) --->
System.Net.WebException: Error writing headers --->
System.IO.IOException: The authentication or decryption has failed. --->
Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
...
The steps so far I have attempted to solve the issue:
I have verified that the ca-certificates-mono package is installed
I have imported the CA certs to the machine store with (why do I need this if I work with a selfsigned cert?)
sudo mozroots --import --machine --sync
I created a selfsigned cert for testing with (as described in the Mono Security FAQ)
makecert -r -eku 1.3.6.1.5.5.7.3.1 -n "CN=Cert4SSL" -sv cert.pvk cert.cer
I added it to the mono cert store
sudo certmgr -add -c -m Trust cert.cer
I have also did tests with other stores (Root, My) and also using not the maching but the user's store - none did work, the same error on each attempt
I assigned port my service uses to the cert
httpcfg -add -port 6067 -cert cert.cer -pvk cert.pvk
I added ignoring the certificate validation
ServicePointManager.ServerCertificateValidationCallback += (o, certificate, chain, errors) => true;
This did not help either (but it got called, the cert object looked allright in the debugger).
The client uses this code to call the WebService:
IService svcClient2 = null;
string address2 = "https://localhost:6067/TestService";
BasicHttpBinding httpBinding2 = new BasicHttpBinding();
httpBinding2.TransferMode = TransferMode.Buffered;
httpBinding2.Security.Mode = BasicHttpSecurityMode.Transport;
httpBinding2.Security.Transport.ClientCredentialType = HttpClientCredentialType.None;
httpBinding2.MessageEncoding = WSMessageEncoding.Text;
httpBinding2.UseDefaultWebProxy = true;
ChannelFactory<IService> channelFac2 = new ChannelFactory<IService>( httpBinding2, new EndpointAddress( address2 ) );
svcClient2 = channelFac2.CreateChannel();
string res2 = svcClient2.TestHello( "Bob" ); // <----- this is where I get the exception
Any help is appreciated, I feel like running in circles.
A few infos about the environment:
I am using Ubuntu 14.04 LTS and Mono 4.0.2, IDE is MonoDevelop
edit: I have now built the very same projects with visual studio and C#, there it works as expected. The client can connect to the host on both http and https.
If i copy over the mono version to my Windows machine, I run into the same issue and error message as on Ubuntu.
Could this be a mono-related issue?

Resources