Trouble passing HTTPS communication through mitmproxy - python-3.x

Question

I'm trying to debug a HTTPS connection using mitmproxy and Python 3. The idea is the following: the proxy is running locally, and I tell Python to use it for a HTTPS connection; Python must accept the self-signed certificate created by mitmproxy for this to work, and if all goes fine, the console of mitmproxy will show me the decoded request/response pair.
I am able to do what I want with Python 3 and requests, but I have to do it using the standard urllib, alas, and I'm failing. Here is the code:
#!/usr/bin/env python3
import urllib.request
import ssl
proxy = urllib.request.ProxyHandler({'https': 'localhost:8080'})
opener = urllib.request.build_opener(proxy)
urllib.request.install_opener(opener)
ssl_ctx = ssl.create_default_context()
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.CERT_NONE
#ssl_ctx = ssl._create_unverified_context()
req = urllib.request.Request('https://github.com')
with urllib.request.urlopen(req) as res:
#with urllib.request.urlopen(req, context=ssl_ctx) as res:
print(res.read().decode('utf8'))
When the above code is executed, I get a
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
error, which means the proxy is used, but its certificate doesn't validate. This is exactly what I expect to happen at this point.
When I use the commented out line to use the ssl_context (create in any of the ways), I get the contents of the page I request, but the proxy console never shows the decoded request information, it stays blank. It is as though using the ssl_context circumvents the proxy altogether.
Could someone please help me out and tell me what I'm doing wrong?
EDIT: just to make sure, I changed the proxy port to 8081, and now the code not using the ssl_ctx variable fails with 'connection refused' (as expected), and the code using the ssl_ctx works fine - this asserts my assumption that the proxy is not used at all.

Answer 1

Thanks for asking this and posting the code you tried. I'm not sure why the documentation claims that fetching HTTPS through a proxy is not supported, because it does work.
Instead of explicitly passing the SSL context, I created and installed an HTTPSHandler in addition to the ProxyHandler. This worked for me (Python 3.5 + mitmproxy):
import urllib.request
import ssl
ssl_ctx = ssl.create_default_context()
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.CERT_NONE
ssl_handler = urllib.request.HTTPSHandler(context=ssl_ctx)
proxy_handler = urllib.request.ProxyHandler({'https': 'localhost:8080'})
opener = urllib.request.build_opener(ssl_handler, proxy_handler)
urllib.request.install_opener(opener)
if __name__ == '__main__':
req = urllib.request.Request('https://...')
with urllib.request.urlopen(req) as res:
print(res.read().decode('utf8'))
Once this opener is installed, it's used as the default for all requests using urllib, even in third-party libraries -- which is what I needed.
I'm not sure, though, why your commented out line didn't work. Perhaps passing the context in urlopen installs a new opener that overrides the custom one.