Scraping HTML Data with BeautifulSoup - python-3.x

Question

I'm doing a course on Python at Coursera.
There is this assignment where ive to scrape a html web page and use it in my code.
Here is the code:
import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = input('http://py4e-data.dr-chuck.net/comments_828036.html')
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html, 'html.parser')
# Retrieve all of the anchor tags
tags = soup('span')
sum = 0
for tag in tags:
sum = sum+int(tag.contents[0])
print (sum)
I'm using OnlineGDB as my compiler
On Compiling and running, a problem is arising:
Traceback (most recent call last):
File "main.py", line 11, in <module>
html = urllib.request.urlopen(url).read()
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 448, in open
req = Request(fullurl, data)
File "/usr/lib/python3.4/urllib/request.py", line 266, in __init__
self.full_url = url
File "/usr/lib/python3.4/urllib/request.py", line 292, in full_url
self._parse()
File "/usr/lib/python3.4/urllib/request.py", line 321, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: ''
Now, can anyone explain what this problem is and the required solution?

Answer 1

It would seem that the problem lies in this line:
url = input('http://py4e-data.dr-chuck.net/comments_828036.html')
input() in python allows you to have the user input something to use in your code. The parameter you pass to the input function (in this case, the url) will be the text to display when prompting the user for the input. Eg. age = input('Enter your age -> ') would prompt the user like so:
Enter your age -> #you would enter it here, then the age variable would be assigned the input
Anyways, it doesn't seem that you need an input at all. So all you have to do to fix the code is to remove the input from your code, and assign the url variable the url directly, like so:
url = 'http://py4e-data.dr-chuck.net/comments_828036.html'
Final code:
import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
url = 'http://py4e-data.dr-chuck.net/comments_828036.html'
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html, 'html.parser')
# Retrieve all of the anchor tags
tags = soup('span')
sum = 0
for tag in tags:
sum = sum+int(tag.contents[0])
print (sum)
Output: 2525
View and run it online
Furthermore, your code can be simplified a bit by using the requests module:
import requests
from bs4 import BeautifulSoup
url = 'http://py4e-data.dr-chuck.net/comments_828036.html'
html = requests.get(url).text
soup = BeautifulSoup(html, 'html.parser')
tags = soup('span')
sum = 0
for tag in tags:
sum += int(tag.contents[0])
print(sum)