Python requests fails when tryign to connect to .onion site - python-3.x

Question

I'm trying to get a webpage which is hosted in the tor network. I'm using the following code:
import requests
def get_tor_session():
session = requests.session()
session.proxies = {'http': 'socks5://127.0.0.1:9150',
'https': 'socks5://127.0.0.1:9150'}
return session
session = get_tor_session()
When I try to get a normal website it works fine, for example: print(session.get("http://httpbin.org/ip").text) prints {"origin": "80.67.172.162"}
But when I try it on a .onion site, it fails with this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/socks.py", line 813, in connect
negotiate(self, dest_addr, dest_port)
File "/usr/local/lib/python3.6/site-packages/socks.py", line 477, in _negotiate_SOCKS5
CONNECT, dest_addr)
File "/usr/local/lib/python3.6/site-packages/socks.py", line 540, in _SOCKS5_request
resolved = self._write_SOCKS5_address(dst, writer)
File "/usr/local/lib/python3.6/site-packages/socks.py", line 592, in _write_SOCKS5_address
addresses = socket.getaddrinfo(host, port, socket.AF_UNSPEC, socket.SOCK_STREAM, socket.IPPROTO_TCP, socket.AI_ADDRCONFIG)
File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
...
Traceback (most recent call last):
File "spider.py", line 13, in <module>
print(session.get("http://zqktlwi4fecvo6ri.onion/").text)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: SOCKSHTTPConnectionPool(host='zqktlwi4fecvo6ri.onion', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.contri
b.socks.SOCKSConnection object at 0x106fd62e8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

Answer 1

When using the socks5 scheme, domains are resolved locally by the client's DNS server. But 'normal' DNS servers can't resolve .onion domains and so your request fails.
From docs.python-requests.org:
Using the scheme socks5 causes the DNS resolution to happen on the client, rather than on the proxy server. This is in line with curl, which uses the scheme to decide whether to do the DNS resolution on the client or proxy. If you want to resolve the domains on the proxy server, use socks5h as the scheme.
So, in order to connect to .onion sites you should let TOR resolve the domain. This is possible if you use the socks5h sheme in the proxies dictionary.
import requests
session = requests.session()
session.proxies = {'http': 'socks5h://127.0.0.1:9150', 'https': 'socks5h://127.0.0.1:9150'}
response = session.get("https://3g2upl4pq6kufc4m.onion/")
print(response)
#<Response [200]>
Note that you may have to install extra dependencies.
pip install requests[socks]