Error of using urllib.request library in python - python-3.x

i try to read url for using urllib.request library in python.
but there are so many error in shell.
i wrote code following this.
import urllib.request
req=urllib.request
req.urlopen("http://www.naver.com")
but there are some error following this.
Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
req.urlopen("http://news.naver.com/")
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 162, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 465, in open
response = self._open(req, data)
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 483, in _open
'_open', req)
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 443, in _call_chain
result = func(*args)
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 1268, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\urllib\request.py", line 1243, in do_open
r = h.getresponse()
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\http\client.py", line 1174, in getresponse
response.begin()
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\http\client.py", line 282, in begin
version, status, reason = self._read_status()
File "C:\Users\kyu\AppData\Local\Programs\Python\Python35\lib\http\client.py", line 251, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
what's wrong??
i don't know why this happening
i use python 3.5.1 and windows8.1 64bit.

Related

Using urlib in a loop

I am trying to write a simple python script that checks the status code of a specific URL and take necessary actions based on the return code. I am using urllib module to achieve this. The issue with this code is that I have different departments defined within a list (I need to iterate over this list inside the urllib) I can't seem to find a way to substitute/insert elements of list in the for loop.
import getopt, sys
import urllib.request
depts = [ 'support', 'edelivery', 'docs']
for dept in depts:
res = urllib.request.urlopen('https://dept.oracle.com').getcode()
print(res)
I get the below error. Can we actually iterate over a list in urllib module in a loop?
#python3 reg_c_with_all.py
Traceback (most recent call last):
File "/usr/lib64/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib64/python3.7/http/client.py", line 1277, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1323, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1272, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1032, in _send_output
self.send(msg)
File "/usr/lib64/python3.7/http/client.py", line 972, in send
self.connect()
File "/usr/lib64/python3.7/http/client.py", line 1439, in connect
super().connect()
File "/usr/lib64/python3.7/http/client.py", line 944, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib64/python3.7/socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "reg_c_with_all.py", line 6, in <module>
res = urllib.request.urlopen('https://dept.oracle.com').getcode()
File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib64/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib64/python3.7/urllib/request.py", line 1393, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib64/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
When I run it individually it returns the response code as expected.
>>> urllib.request.urlopen('https://support.oracle.com').getcode()
200
>>> urllib.request.urlopen('https://docs.oracle.com').getcode()
200
>>> urllib.request.urlopen('https://edelivery.oracle.com').getcode()
200
>>>
Your code iterates over the departments but it does not use them to modify the URL. Try using an fstring to insert the value of dept in the URL:
import getopt, sys
import urllib.request
depts = [ 'support', 'edelivery', 'docs']
for dept in depts:
url = f'https://{dept}.oracle.com'
res = urllib.request.urlopen(url).getcode()
print(f'{url} => {res}')
Output:
https://support.oracle.com => 200
https://edelivery.oracle.com => 200
https://docs.oracle.com => 200

Bigcommerce API connection disconnects with no response

We've been using a Python script to pull orders from our BigCommerce store and reformat them into a .csv for use by our warehouse. This method has been working for the past 3 years yet since the 16 September 2020 our connection has started dropping without a response.
Full response is:
Downloading Order: 1541
WPy64-3720\python-3.7.2.amd64\python.exe : Traceback (most recent call last):
At D:\Administration\Scripts\re_orders\reau-orders.ps1:7 char:28
+ $OrdersProcessingOutput = (WPy64-3720\python-3.7.2.amd64\python.exe bcOrdersREAU ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 386, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 382, in _make_request
httplib_response = conn.getresponse()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\http\client.py", line 1321, in
getresponse
response.begin()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\http\client.py", line 265, in
_read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Administration\Scripts\re_orders\requests\adapters.py", line 438, in send
timeout=timeout
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 649, in urlopen
_stacktrace=sys.exc_info()[2])
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\util\retry.py", line 357, in increment
raise six.reraise(type(error), error, _stacktrace)
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\packages\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 386, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "D:\Administration\Scripts\re_orders\requests\packages\urllib3\connectionpool.py", line 382, in _make_request
httplib_response = conn.getresponse()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\http\client.py", line 1321, in
getresponse
response.begin()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\http\client.py", line 265, in
_read_status
raise RemoteDisconnected("Remote end closed connection without"
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed
connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bcOrdersREAU.py", line 1254, in <module>
checkPendingPayments(basePath + pendingPayFile)
File "bcOrdersREAU.py", line 915, in checkPendingPayments
getOrders(orderId,orderId)
File "bcOrdersREAU.py", line 249, in getOrders
orderJSON = getOrder(headers,orderNo)
File "D:\Administration\Scripts\re_orders\tenacity\__init__.py", line 333, in wrapped_f
return self(f, *args, **kw)
File "D:\Administration\Scripts\re_orders\tenacity\__init__.py", line 417, in __call__
do = self.iter(retry_state=retry_state)
File "D:\Administration\Scripts\re_orders\tenacity\__init__.py", line 360, in iter
return fut.result()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\concurrent\futures\_base.py", line 425,
in result
return self.__get_result()
File "D:\Administration\Scripts\re_orders\WPy64-3720\python-3.7.2.amd64\lib\concurrent\futures\_base.py", line 384,
in __get_result
raise self._exception
File "D:\Administration\Scripts\re_orders\tenacity\__init__.py", line 420, in __call__
result = fn(*args, **kwargs)
File "bc
OrdersREAU.py", line 1155, in getOrder
order = session.get(ApiPath2 + 'orders/' + str(orderNo), headers=headers)
File "D:\Administration\Scripts\re_orders\requests\sessions.py", line 531, in get
return self.request('GET', url, **kwargs)
File "D:\Administration\Scripts\re_orders\requests\sessions.py", line 518, in request
resp = self.send(prep, **send_kwargs)
File "D:\Administration\Scripts\re_orders\requests\sessions.py", line 639, in send
r = adapter.send(request, **kwargs)
File "D:\Administration\Scripts\re_orders\requests\adapters.py", line 488, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without
response'))
The code we're using runs daily and is essentially just iterating through new orders and pulling the details down, including the ordered products. Something like:
requests.get(ApiPath2 + 'orders/' + str(orderNo), headers=headers)
session.get(ApiPath2 + 'orders/' + str(orderNo) + '/products?page=' + str(page), headers=headers)
It doesn't fail every day but has started to be almost a daily occurrence now.
What we have tried:
new API credentials
different servers on a different network (Aus + Europe)
Running the script at different times
Any help would be greatly appreciated. Currently we just keep re running the code manually until it works.
Per python docs
self.send_header('Content-Type', 'blabla' )
self.end_headers()

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')

Anyone has an idea how I can solve this error?
I am trying to download data and I have connected a posgresql server with the database navigator as localhost. Files are successfully downloaded for like 3 minutes and than suddenly the download stops and returns this error message:
.
..
...
run_filing 201500419349300305
run_filing 201540439349301594
Traceback (most recent call last):
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/Georg/PycharmProjects/990-xml-database/irsdb/manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/base.py", line 369, in execute
output = self.handle(*args, **options)
File "/Users/Georg/PycharmProjects/990-xml-database/irsdb/return/management/commands/load_filings.py", line 123, in handle
self.run_filing(filing)
File "/Users/Georg/PycharmProjects/990-xml-database/irsdb/return/management/commands/load_filings.py", line 65, in run_filing
parsed_filing = self.xml_runner.run_filing(object_id)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/xmlrunner.py", line 111, in run_filing
this_filing.process(verbose=verbose)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/filing.py", line 232, in process
self._download(verbose=verbose)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/filing.py", line 70, in _download
stream_download(self.URL, self.filepath, verbose=verbose)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/file_utils.py", line 22, in stream_download
response = requests.get(url, stream=True)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Attached is a screenshot of the connectionpool.py code.

Python 3 issue with connecting web from program

Hi I have a program which just connects to web to read the website data. But when i run the program i am getting some complicated error regarding the web connection. Please see the program below,
from urllib.request import urlopen
html = urlopen("http://www.pythonscraping.com/pages/page1.html")
print(html.read())
The error message is shown below,
Traceback (most recent call last):
File "C:\Python34\lib\urllib\request.py", line 1174, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python34\lib\http\client.py", line 1090, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1128, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1086, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 924, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 859, in send
self.connect()
File "C:\Python34\lib\http\client.py", line 836, in connect
self.timeout, self.source_address)
File "C:\Python34\lib\socket.py", line 491, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "C:\Python34\lib\socket.py", line 530, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11002] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python34\scrapetest.py", line 2, in <module>
html = urlopen("http://www.pythonscraping.com/pages/page1.html")
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 455, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 473, in _open
'_open', req)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1202, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python34\lib\urllib\request.py", line 1176, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11002] getaddrinfo failed>
Please help on this.

Python - How to open urls (html files) which are located on an internal network with urlopen?

The addresses I am trying to open look like this:
"//usa-qa/QAEngr/QA_Team_VSripathi/Negar/Help files/broken/header_test/main.htm"
They are located on a company's internal network.
I tried this piece of code:
import urllib
import urllib.request as ur
address="//usa-qa/QAEngr/QA_Team_VSripathi/Negar/Help files/broken/header_test/main.htm"
info=ur.urlopen(address)
which results in the following error:
Traceback (most recent call last):
File "C:/Users/naref/Desktop/networktest.py", line 14, in <module>
info=ur.urlopen(address)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 458, in open
req = Request(fullurl, data)
File "C:\Python33\lib\urllib\request.py", line 279, in __init__
self._parse()
File "C:\Python33\lib\urllib\request.py", line 284, in _parse
raise ValueError("unknown url type: %s" % self.full_url)
ValueError: unknown url type: //usa-qa/QAEngr/QA_Team_VSripathi/Negar/Help files/broken/header_test/main.htm
The same happens when converting the path to url:
info=ur.urlopen(ur.pathname2url(address))
Then I tried to add an "http:" to the beginning of the address:
address="http://usa-qa/QAEngr/QA_Team_VSripathi/Negar/Help files/broken/header_test/main.htm"
info=ur.urlopen(address)
Which results in the following error:
Traceback (most recent call last):
File "C:/Users/naref/Desktop/networktest.py", line 14, in <module>
info=ur.urlopen(address)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 473, in open
response = self._open(req, data)
File "C:\Python33\lib\urllib\request.py", line 491, in _open
'_open', req)
File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
raise URLError(err)
urllib.error.URLError:<urlopen error [WinError 10061] No connection could be made because the target machine actively refused it>
It is interesting that this problem gets solved when I map the network drive on my computer which runs windows 8: (http://windows.microsoft.com/en-us/windows-8/create-shortcut-to-map-network-drive)
Also the file can be opened with the open function instead of the urlopen, however I prefer to use urlopen since for my purpose the address might be a website's url.

Resources