I need to access a Twitter user's timeline as a JSON string and return the first 250 chars.
Twitter1.py:
import urllib.request, urllib.parse, urllib.error
import twurl
import ssl
TWITTER_URL = 'https://api.twitter.com/1.1/statuses/user_timeline.json'
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
while True:
print('')
acct = input('Enter Twitter Account:')
if (len(acct) < 1): break
url = twurl.augment(TWITTER_URL,
{'screen_name': acct, 'count': '2'})
print('Retrieving', url)
connection = urllib.request.urlopen(url, context=ctx)
data = connection.read().decode()
print(data[:250])
headers = dict(connection.getheaders())
print('Remaining', headers['x-rate-limit-remaining'])
An error related to urllib occurs in output:
Enter Twitter Account:jack
...
Traceback (most recent call last):
File "C:\Users\User\...\twitter1.py", line 18, in <module>
connection = urllib.request.urlopen(url, context=ctx)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\User\anaconda3\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
HTTPError: Bad Request
I cannot figure out the source of the issue. The syntax appears correct and the correct API information was entered into a separate python file 'hidden.py'. twurl and oauth were imported from twurl.py and oauth.py to access the data (included below). hidden.py simply returns my API info in JSON within a function oauth() and oauth is well known so it is also excluded here. Any guidance would be greatly appreciated.
twurl.py:
import urllib.request, urllib.parse, urllib.error
import oauth
import hidden
def augment(url, parameters):
secrets = hidden.oauth()
consumer = oauth.OAuthConsumer(secrets['consumer_key'],
secrets['consumer_secret'])
token = oauth.OAuthToken(secrets['token_key'], secrets['token_secret'])
oauth_request = oauth.OAuthRequest.from_consumer_and_token(consumer,
token=token, http_method='GET', http_url=url,
parameters=parameters)
oauth_request.sign_request(oauth.OAuthSignatureMethod_HMAC_SHA1(),
consumer, token)
return oauth_request.to_url()
Follow up: was resolved soon after I posted, the issue was regarding a domain being blocked by an antivirus filter.
Related
Update: Issue seems to be with Windows Powershell. Program works in Python IDLE.
So I have installed requests, urllib3 module properly. But whenever I try to use requests, I get HTTP 405 error. Please check the attached screenshot for my code and the error I get.
NOTE: I tried attaching images of my code and error but StackOverflow app gave me an error.
NOTE 2: I have tried GET method too but it doesn't work either, it throws the same HTTP 405 error.
My code:
from bs4 import BeautifulSoup
import requests
file = requests.post("https://w3schools.com/python/demopage.htm")
soup = BeautifulSoup(file,"lxml");
print(soup.prettify())
Error I get:
Traceback (most recent call last): File "requestspractice.py", line
1, in
import requests File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\site-packages\requests__init__.py",
line 43, in
import urllib3 File "C:\Users\Prasanna\Python1\urllib3.py", line 15, in
resp = urllib.request.urlopen(req) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 223, in urlopen
return opener.open(url, data, timeout) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 532, in open
response = meth(req, response) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 642, in http_response
'http', request, response, code, msg, hdrs) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 570, in error
return self._call_chain(*args) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 504, in _call_chain
result = func(*args) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 405: Method Not Allowed
I believe what you want to do is GET the page rather than POST anything to it.
file = requests.get("https://w3schools.com/python/demopage.htm")
Your URL is wrong should be at the end "html", but you're using: "https://w3schools.com/python/demopage.htm"
Getting HTTP 400 bad request from testrail server when i try to post testcase result using urllib in python3. Appreciate if someone help me on this. Thanks!
Below is code,
import urllib.request
import json
import base64
data = {'results':[{'case_id': '123','status_id': '1','comment': 'This test passed', 'version': '0.14.0-W9'}]}
headers = {}
post_data = urllib.parse.urlencode(data).encode()
auth = base64.b64encode(b'user:pass')
auth = auth.decode()
headers['Authorization'] = 'Basic %s' % auth
headers['Content-Type'] = 'application/json'
request = urllib.request.Request("http://testrail.com/index.php?/api/v2/add_results_for_cases/272374", data = post_data, headers = headers)
response = urllib.request.urlopen(request).read()
result = json.loads(response)
print(result)
And error output,
Traceback (most recent call last):
File "p3.py", line 13, in <module>
response = urllib.request.urlopen(request).read()
File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
Thanks Tomalak and Amiy for quick suggestions.
I have tried testrailAPI library of python3 and it works as expected.
I have a script(test.py) to test some api, like this:
def get_response(fct, data, method=GET):
"""
Performs the query to the server and returns a string containing the
response.
"""
assert(method in (GET, POST))
url = f'http://{hostname}:{port}/{fct}'
if method == GET:
encode_data = parse.urlencode(data)
response = request.urlopen(f'{url}?{encode_data}')
elif method == POST:
response = request.urlopen(url, parse.urlencode(data).encode('ascii'))
return response.read()
In terminal I call:
python test.py -H 0.0.0.0 -P 5000 --add-data
The traceback:
Traceback (most recent call last):
File "test.py", line 256, in <module>
add_plays()
File "test.py", line 82, in add_plays
get_response("add_channel", {"name": channel}, method=POST)
File "test.py", line 43, in get_response
response = request.urlopen(url, parse.urlencode(data).encode('ascii'))
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: BAD REQUEST
The data is {"name": "Channel1"}. I couldn't understand what is wrong. Please can someone give some tip or show whats's wrong?
When I call using curl, works:
curl -X POST -H "Content-Type: application/json" -d '{"name": "Channel1"}' http://0.0.0.0:5000/add_channel
I solved the problem change the test script:
The api was expected a JSON_MIME_TYPE = 'application/json', so I add a header in a request as follow bellow.
The scrit was using a wrong encode because some text in JSON couldn't be encode in Unicode, Eg:"Omö" encode in ascii launch the exception UnicodeEncodeError: 'ascii' codec can't encode character '\xf6' in position 1: ordinal not in range(128). So I changed to utf8.
Here is the fixed code:
def get_response(fct, data, method=GET):
"""
Performs the query to the server and returns a string containing the
response.
"""
assert(method in (GET, POST))
url = f'http://{hostname}:{port}/{fct}'
if method == GET:
encode_data = parse.urlencode(data)
req = request.Request(f'{url}?{encode_data}'
, headers={'content-type': 'application/json'})
response = request.urlopen(req)
elif method == POST:
params = json.dumps(data)
binary_data = params.encode('utf8')
req = request.Request(url
, data= binary_data
, headers={'content-type': 'application/json'})
response = request.urlopen(req)
x = response.read()
return x
I am trying to use an api to get some data about the prices of cryptocurrencies on an exchange site. When using urllib.request.urlopen, I keep getting errors.
import urllib
import urllib.parse
import urllib.request
import urllib.error
import json
def coin_price(coin):
url = 'https://yobit.net/api/3/ticker/'
pair = coin + '_btc'
final_url = url + pair
obj = urllib.request.urlopen(final_url)
jsonobj = obj.read().decode('utf-8')
data = json.loads(jsonobj)
item = data['ticker']
final = item['last']
print(final)
coin_price("ltc")
These are the errors I am getting
Traceback (most recent call last):
File "C:/Users/x/Downloads/PycharmProjects/test.py", line 20, in <module>
coin_price("ltc")
File "C:/Users/x/Downloads/PycharmProjects/test.py", line 12 incoin_price
obj = urllib.request.urlopen(final_url)
File "C:\Users\x\AppData\Local\Programs\Python\Python36 32\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\x\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\Users\x\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\x\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
When I run this code:
import http.client
hR = ["/index.html"]
conn = http.client.HTTPConnection("www.python.org", 80)
conn.connect()
conn.request("GET", hR)
response = conn.getresponse()
data = response.read()
print (data)
conn.close()
I receive the following error:
Traceback (most recent call last):
File "C:\Python32\files\fcon.py", line 5, in <module>
conn.request("GET", hR)
File "C:\Python32\lib\http\client.py", line 964, in request
self._send_request(method, url, body, headers)
File "C:\Python32\lib\http\client.py", line 992, in _send_request
self.putrequest(method, url, **skips)
File "C:\Python32\lib\http\client.py", line 877, in putrequest
if url.startswith('http'):
AttributeError: 'list' object has no attribute 'startswith'
Also, when I change the URL in line 3 to "http://python.org" I receive a different error:
Traceback (most recent call last):
File "C:\Python32\files\fcon.py", line 4, in <module>
conn.connect()
File "C:\Python32\lib\http\client.py", line 721, in connect
self.timeout, self.source_address)
File "C:\Python32\lib\socket.py", line 380, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed
The first error message tells you that hR should not be a list, but a string, this would work:
import http.client
hR = "/index.html"
conn = http.client.HTTPConnection("www.python.org", 80)
conn.connect()
conn.request("GET", hR)
response = conn.getresponse()
data = response.read()
print (data)
conn.close()
However you won't see any data, because python.org replies only with a http 301 respons redirecting to it's https page, which http.client does not automatically follow.
The second error you get because http://www.python.org is not a valid host name, www.python.org was correct here.
http.client is a rather low-level API, you should consider using urllib.request instead, or even betther the requests library.