I have done proxy settings using CNTLM and also have a script(google.py) to check proxy setting are working or not
import urllib.request
proxy_support = urllib.request.ProxyHandler({"http":"http://localhost:3128"})
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)
html = urllib.request.urlopen("http://www.google.com").read()
print(html)
when i run the script I'm getting following errors
C:\Users\asj5cob\Desktop>py -3.4 google.py
Traceback (most recent call last):
File "google.py", line 5, in <module>
html = urllib.request.urlopen("http://www.google.com").read()
File "C:\Python34\lib\urllib\request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 470, in open
response = meth(req, response)
File "C:\Python34\lib\urllib\request.py", line 580, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python34\lib\urllib\request.py", line 508, in error
return self._call_chain(*args)
File "C:\Python34\lib\urllib\request.py", line 442, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 588, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 502: Parent proxy unreacheable
Did you start CNTLM by executing in your administrator cmd window?
net start cntlm
Related
Update: Issue seems to be with Windows Powershell. Program works in Python IDLE.
So I have installed requests, urllib3 module properly. But whenever I try to use requests, I get HTTP 405 error. Please check the attached screenshot for my code and the error I get.
NOTE: I tried attaching images of my code and error but StackOverflow app gave me an error.
NOTE 2: I have tried GET method too but it doesn't work either, it throws the same HTTP 405 error.
My code:
from bs4 import BeautifulSoup
import requests
file = requests.post("https://w3schools.com/python/demopage.htm")
soup = BeautifulSoup(file,"lxml");
print(soup.prettify())
Error I get:
Traceback (most recent call last): File "requestspractice.py", line
1, in
import requests File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\site-packages\requests__init__.py",
line 43, in
import urllib3 File "C:\Users\Prasanna\Python1\urllib3.py", line 15, in
resp = urllib.request.urlopen(req) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 223, in urlopen
return opener.open(url, data, timeout) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 532, in open
response = meth(req, response) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 642, in http_response
'http', request, response, code, msg, hdrs) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 570, in error
return self._call_chain(*args) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 504, in _call_chain
result = func(*args) File "C:\Users\Prasanna\AppData\Local\Programs\Python\Python36\lib\urllib\request.py",
line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 405: Method Not Allowed
I believe what you want to do is GET the page rather than POST anything to it.
file = requests.get("https://w3schools.com/python/demopage.htm")
Your URL is wrong should be at the end "html", but you're using: "https://w3schools.com/python/demopage.htm"
Getting HTTP 400 bad request from testrail server when i try to post testcase result using urllib in python3. Appreciate if someone help me on this. Thanks!
Below is code,
import urllib.request
import json
import base64
data = {'results':[{'case_id': '123','status_id': '1','comment': 'This test passed', 'version': '0.14.0-W9'}]}
headers = {}
post_data = urllib.parse.urlencode(data).encode()
auth = base64.b64encode(b'user:pass')
auth = auth.decode()
headers['Authorization'] = 'Basic %s' % auth
headers['Content-Type'] = 'application/json'
request = urllib.request.Request("http://testrail.com/index.php?/api/v2/add_results_for_cases/272374", data = post_data, headers = headers)
response = urllib.request.urlopen(request).read()
result = json.loads(response)
print(result)
And error output,
Traceback (most recent call last):
File "p3.py", line 13, in <module>
response = urllib.request.urlopen(request).read()
File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
Thanks Tomalak and Amiy for quick suggestions.
I have tried testrailAPI library of python3 and it works as expected.
I m scraping and parsing the html content of secured website justdial.com into a csv file ,since i m using user agent also but still i m getting this error- raise HTTPError(req.full_url, code, msg, hdrs, fp) HTTPError: Forbidden .
My code is -
import urllib.request
import urllib
from urllib.request import urlopen
import bs4
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
url = 'https://www.justdial.com/Mumbai/311/B2b_fil'
req = urllib.request.Request(url, None, headers)
response = urllib.request.urlopen(req)
print(response.read())
html= urllib.request.urlopen(url).read()
soup = BeautifulSoup(html)
After printing response , i have to parse the content from html to csv file but it is giving this error
File "<ipython-input-21-c589d79bf43d>", line 1, in <module>
runfile('C:/Users/justdial.py', wdir='C:/Users')
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
execfile(filename, namespace)
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/justdial.py", line 21, in <module>
html= urllib.request.urlopen(url).read()
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
HTTPError: Forbidden
ENV
PhantomJS2.1.1;Windows10;selenium-3.8.0;python3
CODE
browser = webdriver.PhantomJS() //throw a UnicodeDecodeError
Problem
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position
2: invalid start byte
Details
Traceback (most recent call last):
File "d:/devtools/phantomjs-2.1.1-windows/bin/IndustryLeaderSpider.py",
line 20, in
browser = webdriver.PhantomJS('D:/phantomjs.exe')
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\phantomjs\webdriver.py",
line 58, in init
desired_capabilities=desired_capabilities)
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 154, in init
self.start_session(desired_capabilities, browser_profile)
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 309, in execute
response = self.command_executor.execute(driver_command, params)
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\remote_connection.py",
line 460, in execute
return self._request(command_info[0], url, body=data)
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\remote_connection.py",
line 522, in _request
resp = opener.open(request, timeout=self._timeout)
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 526, in open
response = self._open(req, data)
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 544, in _open
'_open', req)
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 806, in
meth(r, proxy, type))
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 814, in proxy_open
if req.host and proxy_bypass(req.host):
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 2739, in proxy_bypass
return proxy_bypass_registry(host)
File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 2706, in proxy_bypass_registry
fqdn = socket.getfqdn(rawHost)
File "D:\Continuum\Anaconda3\lib\socket.py", line 673, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 2: invalid start byte
Do someone have this problem? How to fix it?
Your main error seems to be :
File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 460, in execute return self._request(command_info[0], url, body=data) File "D:\Continuum\Anaconda3\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 522, in _request resp = opener.open(request, timeout=self._timeout) File "D:\Continuum\Anaconda3\lib\urllib\request.py", line 526, in open response = self._open(req, data)
Try the following steps :
Run CCleaner tool to wipe off all the OS chores.
Take a System Reboot.
While you initiate PhantomJS() provide the absolute path of the phantomjs binary as follows :
driver = webdriver.PhantomJS(executable_path=r'C:\path\to\phantomjs.exe')
Ensure that there is no unnecessary imports in your script. Example : urllib.request
Execute your Test.
Trying to use Python Twitter Tools to search for the tweeets containing a hashtag. (On a raspberry Pi with python3).
from twitter import *
token = "token"
token_key = "token_key"
con_secret = "con_secret"
con_secret_key = "con_secret_key"
t = Twitter(
auth=OAuth(token, token_key, con_secret, con_secret_key))
print(t.search.tweets(q="#test"))
But I always get a Authorization error.
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 319, in _handle_response
handle = urllib_request.urlopen(req, **kwargs)
File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 499, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 579, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Authorization Required
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "twitter-test.py", line 10, in <module>
print(t.search.tweets(q="#test"))
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 312, in __call__
return self._handle_response(req, uri, arg_data, _timeout)
File "/usr/local/lib/python3.4/dist-packages/twitter/api.py", line 345, in _handle_response
raise TwitterHTTPError(e, uri, self.format, arg_data)
twitter.api.TwitterHTTPError: Twitter sent status 401 for URL: 1.1/search/tweets.json using parameters: (oauth_consumer_key=**key**&oauth_nonce=**nonce**&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1438333973&oauth_token=**token**&oauth_version=1.0&q=%23test&oauth_signature=**signature**)
details: {'errors': [{'code': 32, 'message': 'Could not authenticate you.'}]}
I have tried checking my time (and changing the timezone).
I have tried putting in a callback URL into the app settings and regenerating the keys.
Any help appreciated
Thanks
Couldn't fix this so moved to tweepy library which works a treat!