Scraping dynamic website (CSS?) with python - python-3.x

Question

I want to know if a item is available in the local library. I can see this in the catalog with a green icon (available), or a red icon (loaned out/not available).
First I tried just beautifullsoup, this is the python code I tried:
try:
import urllib.request as urllib2
except ImportError:
import urllib2
from bs4 import BeautifulSoup
page = urllib2.urlopen("http://zoeken.mol.bibliotheek.be/?itemid=|library/marc/vlacc|9394694&undup=false")
soup = BeautifulSoup(page)
bal = soup.find(class_="avail-icon")
print(bal)
But while an element inspection in firefox gives:
<span class="avail-icon">
<i class="circle-icon avail-icon-none"></i>
<span class="hidden-text"></span>
</span>
class="circle-icon avail-icon-none" means the item is available (shows green icon on webpage),
class="circle-icon avail-icon-loanedout" means the item is loaned out (shows red icon on webpage).
I got:
<span class="avail-icon">
<i class="circle-icon avail-icon-loading"></i>
<span class="hidden-text">Toon beschikbaarheid voor</span>
</span>
class="circle-icon avail-icon-loading" means dynamic, I asume, so after some searching I found Selenium.
I tried the following code:
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://zoeken.mol.bibliotheek.be/?itemid=|library/marc/vlacc|9394694&undup=false")
html = driver.page_source
soup = BeautifulSoup(html)
bal = soup.find(class_="avail-icon")
print(bal)
Sadly, this gives me:
<span class="avail-icon">
<i class="circle-icon avail-icon-unknown"></i>
<span class="hidden-text">Toon beschikbaarheid voor</span>
</span>
Maybe I wasn't waiting enough before Selenium grabbed the webpage, so after some searching I changed the code to:
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Firefox()
driver.implicitly_wait(10) # seconds
driver.get("http://zoeken.mol.bibliotheek.be/?itemid=|library/marc/vlacc|9394694&undup=false")
html = driver.page_source
soup = BeautifulSoup(html)
bal = soup.find(class_="avail-icon")
print(bal)
Still the same result, class="circle-icon avail-icon-unknown" isn't what I'm looking for and I'm now out of ideas. Can someone throw me a hint?
PS: Maybe an idea, but I don't know how to do it:
In Firefox, in the element inspector, the right pane has a column called rules (dutch: regels). The red and green icon are loaded as one .png file (icon-sprite.png). Select the red/green icon to see what I mean.
background-position: -48px -16px; means available (green icon)
background-position: 0px -32px; means not available (red icon)
Can I somehow test for this?
PS2: I'm a novice programmer (skill level = low).

Answer 1

You can do this entirely by using selenium:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get("http://zoeken.mol.bibliotheek.be/?itemid=|library/marc/vlacc|9394694&undup=false")
#waiting until the icon is loaded...
WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.XPATH, """//*[#id="availabilityStatic"]/div/div/ul/li/ul/li/span/i""")))
circle_icon = driver.find_element_by_xpath("""//*[#id="availabilityStatic"]/div/div/ul/li/ul/li/span/i""")
icon_class = circle_icon.get_attribute("class")
if "loanedout" in icon_class:
print "item not available"
else:
print "item available"
driver.quit()