http.client request method in Python 3 - python-3.x

When I run this code:
import http.client
hR = ["/index.html"]
conn = http.client.HTTPConnection("www.python.org", 80)
conn.connect()
conn.request("GET", hR)
response = conn.getresponse()
data = response.read()
print (data)
conn.close()
I receive the following error:
Traceback (most recent call last):
File "C:\Python32\files\fcon.py", line 5, in <module>
conn.request("GET", hR)
File "C:\Python32\lib\http\client.py", line 964, in request
self._send_request(method, url, body, headers)
File "C:\Python32\lib\http\client.py", line 992, in _send_request
self.putrequest(method, url, **skips)
File "C:\Python32\lib\http\client.py", line 877, in putrequest
if url.startswith('http'):
AttributeError: 'list' object has no attribute 'startswith'
Also, when I change the URL in line 3 to "http://python.org" I receive a different error:
Traceback (most recent call last):
File "C:\Python32\files\fcon.py", line 4, in <module>
conn.connect()
File "C:\Python32\lib\http\client.py", line 721, in connect
self.timeout, self.source_address)
File "C:\Python32\lib\socket.py", line 380, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed

The first error message tells you that hR should not be a list, but a string, this would work:
import http.client
hR = "/index.html"
conn = http.client.HTTPConnection("www.python.org", 80)
conn.connect()
conn.request("GET", hR)
response = conn.getresponse()
data = response.read()
print (data)
conn.close()
However you won't see any data, because python.org replies only with a http 301 respons redirecting to it's https page, which http.client does not automatically follow.
The second error you get because http://www.python.org is not a valid host name, www.python.org was correct here.
http.client is a rather low-level API, you should consider using urllib.request instead, or even betther the requests library.

Related

urllib and 'HTTPError: Bad Request'

I need to access a Twitter user's timeline as a JSON string and return the first 250 chars.
Twitter1.py:
import urllib.request, urllib.parse, urllib.error
import twurl
import ssl
TWITTER_URL = 'https://api.twitter.com/1.1/statuses/user_timeline.json'
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
while True:
print('')
acct = input('Enter Twitter Account:')
if (len(acct) < 1): break
url = twurl.augment(TWITTER_URL,
{'screen_name': acct, 'count': '2'})
print('Retrieving', url)
connection = urllib.request.urlopen(url, context=ctx)
data = connection.read().decode()
print(data[:250])
headers = dict(connection.getheaders())
print('Remaining', headers['x-rate-limit-remaining'])
An error related to urllib occurs in output:
Enter Twitter Account:jack
...
Traceback (most recent call last):
File "C:\Users\User\...\twitter1.py", line 18, in <module>
connection = urllib.request.urlopen(url, context=ctx)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
HTTPError: Bad Request
I cannot figure out the source of the issue. The syntax appears correct and the correct API information was entered into a separate python file 'hidden.py'. twurl and oauth were imported from twurl.py and oauth.py to access the data (included below). hidden.py simply returns my API info in JSON within a function oauth() and oauth is well known so it is also excluded here. Any guidance would be greatly appreciated.
twurl.py:
import urllib.request, urllib.parse, urllib.error
import oauth
import hidden
def augment(url, parameters):
secrets = hidden.oauth()
consumer = oauth.OAuthConsumer(secrets['consumer_key'],
secrets['consumer_secret'])
token = oauth.OAuthToken(secrets['token_key'], secrets['token_secret'])
oauth_request = oauth.OAuthRequest.from_consumer_and_token(consumer,
token=token, http_method='GET', http_url=url,
parameters=parameters)
oauth_request.sign_request(oauth.OAuthSignatureMethod_HMAC_SHA1(),
consumer, token)
return oauth_request.to_url()
Follow up: was resolved soon after I posted, the issue was regarding a domain being blocked by an antivirus filter.

What is the proper way of catching http error connection with Python module aiohttp?

I want to write a simple script that checks to see if website is up. If it is not, I want to catch the http return error code using the aiohttp module for Python. In the example below, I pass in a fake website 'http://www.googlesr2332.com' rather than returning the http error, I am getting the following:
Traceback (most recent call last):
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/connector.py", l
ine 967, in _create_direct_connection traces=traces), loop=self._loop)
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/connector.py", l
ine 830, in _resolve_host
self._resolver.resolve(host, port, family=self._family) File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/resolver.py", li
ne 30, in resolve
host, port, type=socket.SOCK_STREAM, family=family)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 784, in getaddrinfo
None, getaddr_func, host, port, family, type, proto, flags)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "main.py", line 19, in <module>
loop.run_until_complete(main())
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "main.py", line 8, in main
async with session.get(site) as response:
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/client.py", line
1012, in __aenter__
self._resp = await self._coro
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/client.py", line 483, in _request
timeout=real_timeout
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/connector.py", l
ine 523, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/connector.py", l
ine 859, in _create_connection req, traces, timeout)
File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/aiohttp/connector.py", l
ine 971, in _create_direct_connection
raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.googlesr2332.com:80 ssl:default [Name or service not known]
Here is the sample code I am running:
import aiohttp
import asyncio
sites = ['http://www.google.com', 'http://python.org', 'http://www.facebook.com', 'http://www.googlesr2332.com']
async def main():
async with aiohttp.ClientSession() as session:
for site in sites:
async with session.get(site) as response:
if response.status == 200:
print("Status:", response.status)
print("Content-type:", response.headers['content-type'])
html = await response.text()
print("Body:", html[15], "...")
else:
print(response.status)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
You have the code when there is a response. But there is no code to handle if the connection itself has got in trouble.
import aiohttp
import asyncio
sites = ['http://www.google.com', 'http://python.org', 'http://www.facebook.com', 'http://www.googlesr2332.com']
async def main():
async with aiohttp.ClientSession() as session:
for site in sites:
try:
async with session.get(site) as response:
if response.status == 200:
print("Status:", response.status)
print("Content-type:", response.headers['content-type'])
html = await response.text()
print("Body:", html[:15], "...")
else:
print(response.status)
except aiohttp.ClientConnectorError as e:
print('Connection Error', str(e))
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
When making a request to a website, you expect to get a response from it. But if your request can't reach the desired server, you can't get any response. You don't have any errors handling, so you get an error when trying to reach website that doesn't exist. The error message is pretty much self-explanatory: Cannot connect to host www.googlesr2332.com:80 ssl:default [Name or service not known]. Consider wrapping your request sending function with try except.

Error 405 when using "requests" module in Python

Update: Issue seems to be with Windows Powershell. Program works in Python IDLE.
So I have installed requests, urllib3 module properly. But whenever I try to use requests, I get HTTP 405 error. Please check the attached screenshot for my code and the error I get.
NOTE: I tried attaching images of my code and error but StackOverflow app gave me an error.
NOTE 2: I have tried GET method too but it doesn't work either, it throws the same HTTP 405 error.
My code:
from bs4 import BeautifulSoup
import requests
file = requests.post("https://w3schools.com/python/demopage.htm")
soup = BeautifulSoup(file,"lxml");
print(soup.prettify())
Error I get:
Traceback (most recent call last): File "requestspractice.py", line
1, in
import requests File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\site-packages\requests__init__.py",
line 43, in
import urllib3 File "C:\Users\Prasanna\Python1\urllib3.py", line 15, in
resp = urllib.request.urlopen(req) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 223, in urlopen
return opener.open(url, data, timeout) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 532, in open
response = meth(req, response) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 642, in http_response
'http', request, response, code, msg, hdrs) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 570, in error
return self._call_chain(*args) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 504, in _call_chain
result = func(*args) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 405: Method Not Allowed
I believe what you want to do is GET the page rather than POST anything to it.
file = requests.get("https://w3schools.com/python/demopage.htm")
Your URL is wrong should be at the end "html", but you're using: "https://w3schools.com/python/demopage.htm"

Using urllib.request.urlopen to get JSON data

I am trying to use an api to get some data about the prices of cryptocurrencies on an exchange site. When using urllib.request.urlopen, I keep getting errors.
import urllib
import urllib.parse
import urllib.request
import urllib.error
import json
def coin_price(coin):
url = 'https://yobit.net/api/3/ticker/'
pair = coin + '_btc'
final_url = url + pair
obj = urllib.request.urlopen(final_url)
jsonobj = obj.read().decode('utf-8')
data = json.loads(jsonobj)
item = data['ticker']
final = item['last']
print(final)
coin_price("ltc")
These are the errors I am getting
Traceback (most recent call last):
File "C:/Users/x/Downloads/PycharmProjects/test.py", line 20, in <module>
coin_price("ltc")
File "C:/Users/x/Downloads/PycharmProjects/test.py", line 12 incoin_price
obj = urllib.request.urlopen(final_url)
File "C:\Users\x\AppData\Local\Programs\Python\Python36 32\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\x\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Users\x\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\x\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Authorization error from Python twitter tools

Trying to use Python Twitter Tools to search for the tweeets containing a hashtag. (On a raspberry Pi with python3).
from twitter import *
token = "token"
token_key = "token_key"
con_secret = "con_secret"
con_secret_key = "con_secret_key"
t = Twitter(
auth=OAuth(token, token_key, con_secret, con_secret_key))
print(t.search.tweets(q="#test"))
But I always get a Authorization error.
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 319, in _handle_response
handle = urllib_request.urlopen(req, **kwargs)
File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 499, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Authorization Required
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "twitter-test.py", line 10, in <module>
print(t.search.tweets(q="#test"))
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 312, in __call__
return self._handle_response(req, uri, arg_data, _timeout)
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 345, in _handle_response
raise TwitterHTTPError(e, uri, self.format, arg_data)
twitter.api.TwitterHTTPError: Twitter sent status 401 for URL: 1.1/search/tweets.json using parameters: (oauth_consumer_key=**key**&oauth_nonce=**nonce**&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1438333973&oauth_token=**token**&oauth_version=1.0&q=%23test&oauth_signature=**signature**)
details: {'errors': [{'code': 32, 'message': 'Could not authenticate you.'}]}
I have tried checking my time (and changing the timezone).
I have tried putting in a callback URL into the app settings and regenerating the keys.
Any help appreciated
Thanks
Couldn't fix this so moved to tweepy library which works a treat!

Resources