Python 3.7 - Download Image - Urllib.request.urlretrieve Error - python-3.x

I am really new to programming and currently learning python on Youtube ('The New Boston - Python 3' )
Was trying to download an image from internet based on the code show in the video but an error pop up.
Here is the code :
import random
import urllib.request
def download_web_image(url):
name = random.randrange(1,1000)
full_name = str(name) + '.gif' #str convert number to word
urllib.request.urlretrieve(url, full_name)
download_web_image ('https://images.freeimages.com/images/large-previews/ed3/a-stormy-paradise-1-1563744.jpg')
And the error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/isaactai/PycharmProjects/isaacfirst/IMAGEDOWNLOAD.py", line 10, in
download_web_image ('https://images.freeimages.com/images/large-previews/ed3/a-stormy-paradise-1-1563744.jpg')
File "/Users/isaactai/PycharmProjects/isaacfirst/IMAGEDOWNLOAD.py", line 8, in download_web_image
urllib.request.urlretrieve(url, full_name)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError:
Process finished with exit code 1
I am using PyCharm 2018.3 version
Please help me, thank you

Go to the folder where Python is installed. It should have a name like Python 3.x with x being whatever version of python you installed. Now double click on 'Install Certificates.command'. Had this error before and someone on stack helped me fix it also.
The path for me was as follows:
C:\Python33\Tools\Scripts
If that doesn't work here is another workaround using the ssl package:
pip install ssl
Do this before running the code. Then add this to your code.
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Essentially what this does is make your request "secure" so the HTTPS site will actually accept the request coming from python. You should always do this before trying to access a site with the https prefix.

Related

Attempting cifar-10 download via TF and get certicate error

I am starting out in machine vision, and have tried to download CIFAR-10 direct via python code and keep being stopped by a certificate error.
Not being a python expert, am not sure how to work aorund this :
I excute the python code :
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.layers import Input, Conv2D, Dense, Flatten, Dropout
from tensorflow.keras.layers import GlobalMaxPooling2D, MaxPooling2D
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.models import Model
cifar10 = tf.keras.datasets.cifar10
and then if fails with this error :
Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
Traceback (most recent call last):
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
self.connect()
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1422, in connect
server_hostname=server_hostname)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 423, in wrap_socket
session=session
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 870, in _create
self.do_handshake()
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\utils\data_utils.py", line 274, in get_file
urlretrieve(origin, fpath, dl_progress)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\utils\data_utils.py", line 82, in urlretrieve
response = urlopen(url, data)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
response = self._open(req, data)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
'_open', req)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1360, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Users\test1\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)>
I am not sure how to either download the dataset manually and get python to load from disk, or force it to ignore the error.
Any help would be appreciated. Thanks in advance.
The error occurs because you don't have the correct certificates installed, and the https query is failing to connect to the cifar-10 database.
You can bypass python and manually download it from: cifar-10-python.tar.gz. Then you will just need to point python to wherever you downloaded the file.
Or, if you are on a Mac then you can try installing the required certificates by running this file: /Applications/Python\ 3.6/Install\ Certificates.command and then running your script again
Alternatively,
Edit your local copy of this file cifar-10.py and rerun your script.
# line 78
origin = 'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz'
# and change it to be:
origin = 'http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz'
This file may exist in a few spots depending on how you installed tensorflow though.

Using urlib in a loop

I am trying to write a simple python script that checks the status code of a specific URL and take necessary actions based on the return code. I am using urllib module to achieve this. The issue with this code is that I have different departments defined within a list (I need to iterate over this list inside the urllib) I can't seem to find a way to substitute/insert elements of list in the for loop.
import getopt, sys
import urllib.request
depts = [ 'support', 'edelivery', 'docs']
for dept in depts:
res = urllib.request.urlopen('https://dept.oracle.com').getcode()
print(res)
I get the below error. Can we actually iterate over a list in urllib module in a loop?
#python3 reg_c_with_all.py
Traceback (most recent call last):
File "/usr/lib64/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib64/python3.7/http/client.py", line 1277, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1323, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1272, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1032, in _send_output
self.send(msg)
File "/usr/lib64/python3.7/http/client.py", line 972, in send
self.connect()
File "/usr/lib64/python3.7/http/client.py", line 1439, in connect
super().connect()
File "/usr/lib64/python3.7/http/client.py", line 944, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib64/python3.7/socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "reg_c_with_all.py", line 6, in <module>
res = urllib.request.urlopen('https://dept.oracle.com').getcode()
File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib64/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib64/python3.7/urllib/request.py", line 1393, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib64/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
When I run it individually it returns the response code as expected.
>>> urllib.request.urlopen('https://support.oracle.com').getcode()
200
>>> urllib.request.urlopen('https://docs.oracle.com').getcode()
200
>>> urllib.request.urlopen('https://edelivery.oracle.com').getcode()
200
>>>
Your code iterates over the departments but it does not use them to modify the URL. Try using an fstring to insert the value of dept in the URL:
import getopt, sys
import urllib.request
depts = [ 'support', 'edelivery', 'docs']
for dept in depts:
url = f'https://{dept}.oracle.com'
res = urllib.request.urlopen(url).getcode()
print(f'{url} => {res}')
Output:
https://support.oracle.com => 200
https://edelivery.oracle.com => 200
https://docs.oracle.com => 200

SSL: CERTIFICATE_VERIFY_FAILED with urllib

I'm running into trouble with the module urllib (Python 3.6). Every time I use the module, I get a page's worth of errors.
what's wrong with urllib and how to fix it?
import urllib.request
url='https://www.goodreads.com/quotes/tag/artificial-intelligence'
u1 = urllib.request.urlopen(url)
print(u1)
That block of code likes to spit out this mouthful of stuff:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 814, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/a-nguyen/Downloads/EzSorses/APAsauce.py", line 3, in <module>
u1 = urllib.request.urlopen(url)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)>
Seems like something's off with the module itself.
There is command line program that you can run on MacOsX that will install the certificates:
sudo /Applications/Python\ 3.6/Install\ Certificates.command
The sudo command will prompt for you password to elevate your privileges.
a dirty (not safe in terms of security) but fast hack like this do the job for me:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
I had the same issue. My python version is 3.6.5 and I'm on Mac OSX. When I tried to connect to the HTTPS sites I started getting the
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)>
exception. So, it was so obvious to me that the python on my machine has no certificates. I just ran the file from the path /Applications/Python 3.6 with a file name Install Certificates.command. This solved my issue.
I reffered to this post.
To fix the problem, open your bash and write:
cd /Applications/Python 3.6
then:
sudo Certificates.command

While using pandas got error urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

This is my code in python3 :
import pandas as pd
tables = pd.read_html("http://http.com/")
print(tables)
And I got this error :
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
server_hostname=server_hostname)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
self.do_handshake()
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/pandas/io/html.py", line 906, in read_html
keep_default_na=keep_default_na)
File "/usr/local/lib/python3.6/site-packages/pandas/io/html.py", line 743, in _parse
raise_with_traceback(retained)
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 344, in raise_with_traceback
raise exc.with_traceback(traceback)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>
This have been working very well till now, but I have this problem on this particular website. Also, I am new in this and don't know much about SSL Certificate, I searched it on google and still do not have much knowledge about it.
use -
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
For MAC
Click Applications => Python then double-click Install Certificates.command.

Python 3.5 - ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)

I am using Keras and trying to load mnist dataset.
Link to the website is: https://s3.amazonaws.com/img-datasets/mnist.pkl.gz
It works in a browser. But when I call 'mnist.load_data()', I get the exception:
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)
There are many solutions already provided, I tried almost all of them.
Below is the stacktrace:
Downloading data from https://s3.amazonaws.com/img-datasets/mnist.pkl.gz
Traceback (most recent call last):
File "/home/user/anaconda3/lib/python3.5/urllib/request.py", line 1684, in open
return getattr(self, name)(url)
File "/home/user/anaconda3/lib/python3.5/urllib/request.py", line 1894, in open_https
return self._open_generic_http(self._https_connection, url, data)
File "/home/user/anaconda3/lib/python3.5/urllib/request.py", line 1843, in _open_generic_http
http_conn.request("GET", selector, headers=headers)
File "/home/user/anaconda3/lib/python3.5/http/client.py", line 1083, in request
self._send_request(method, url, body, headers)
File "/home/user/anaconda3/lib/python3.5/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/home/user/anaconda3/lib/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
File "/home/user/anaconda3/lib/python3.5/http/client.py", line 911, in _send_output
self.send(msg)
File "/home/user/anaconda3/lib/python3.5/http/client.py", line 854, in send
self.connect()
File "/home/user/anaconda3/lib/python3.5/http/client.py", line 1237, in connect
server_hostname=server_hostname)
File "/home/user/anaconda3/lib/python3.5/ssl.py", line 376, in wrap_socket
_context=self)
File "/home/user/anaconda3/lib/python3.5/ssl.py", line 747, in __init__
self.do_handshake()
File "/home/user/anaconda3/lib/python3.5/ssl.py", line 983, in do_handshake
self._sslobj.do_handshake()
File "/home/user/anaconda3/lib/python3.5/ssl.py", line 628, in do_handshake
self._sslobj.do_handshake()
My 'openssl version' on console yeilds:
openssl version
OpenSSL 1.0.2e 3 Dec 2015
In python console:
>>> import OpenSSL
>>> OpenSSL.__version__
'0.15.1'
>>> import ssl
>>> print (ssl.OPENSSL_VERSION)
OpenSSL 1.0.2e 3 Dec 2015
I am using Python 3.5 that came with anaconda
I also upgraded OpenSSL using conda install OpenSSL
but the error persists.
I also tried a patch:
import ssl
from functools import wraps
def sslwrap(func):
#wraps(func)
def bar(*args, **kw):
kw['ssl_version'] = ssl.PROTOCOL_TLSv1
return func(*args, **kw)
return bar
ssl.wrap_socket = sslwrap(ssl.wrap_socket)
But still no resolution. How can I get rid of this error?

Resources