Scraping a hidden variable from a website (Bs4) - python-3.x

Question

I am trying to learn how to scrape a website. I am using Python3 and BS4.
I am stuck on a specific problem.
Example:http://www2.hm.com/en_in/productpage.0648256001.html
I am unable to scrape the "Sizes" available in the dropdown menu and whether they are sold out, in the above link. I went through the whole source code but couldn't figure out under which tags does the data exist. I am guessing it must be a hidden variable or something?

Answer 1

Okay, so I tracked XHR requests the website makes and I wrote the code below. Basically, it uses Selenium to get the value of productArticleDetails variable and the URL of the availability endpoint (I could have hardcoded it, but I found the variable it's in, so why not use it).
from itertools import chain
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
url = 'http://www2.hm.com/en_in/productpage.0648256002.html'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
browser = webdriver.Chrome()
browser.get(url)
details = browser.execute_script('return productArticleDetails;')
availability_url = browser.execute_script('return hm.options.product.productAvailabilityServiceUrl;')
browser.quit()
variants = {} # e.g one product can be available in different colors
for key, value in details.items():
# there is a lot of information in the details, not only product variants
try:
if 'whitePrice' in value:
variants[key] = value
except AttributeError:
pass
# 'http://www2.hm.com/en_in/getAvailability?variants=0648256001,0648256002,0648256003,0648256006,0648256007,0648256008'
payload = {'variants': ','.join(variants.keys())}
r = requests.get(urljoin(url, availability_url), params=payload)
available_sizes = r.json()['availability']
# r.json() contains:
# availability: ["0648256001001", "0648256001002", "0648256001007",…]
# fewPieceLeft: []
sizes = chain.from_iterable(variant['sizes'] for variant in variants.values())
for size in sizes:
availability = size['sizeCode'] in available_sizes
size['available'] = availability # True/False, feel free to implement handling "fewPieceLeft"
# Output
for variant in variants.values():
print(f'Variant: {variant["name"]}') # color in that case
print('\tsizes:')
for size in variant['sizes']:
print(f'\t\t{size["name"]} -> {"Available" if size["available"] else "Sold out"}')
Output:
Variant: Light beige/Patterned
sizes:
32 -> Available
34 -> Available
36 -> Sold out
...
Variant: Orange
sizes:
32 -> Available
...
The advantage of this approach is that you gain access to a lot of details such as 'whitePrice': 'Rs. 1,299',, 'careInstructions': ['Machine wash at 30°'], 'composition': ['Viscose 100%'], description and some more. You can take a look yourself:
import pprint
pprint.pprint(variants)
The disadvantage is that you need to use Selenium and download a driver, but to be fair I used Selenium only to get the variables, since extracting this nested JS object with regex seems impossible to me (correct me if I'm wrong) and browser.execute_script('return productArticleDetails;') is very clear and concise.
There are no hidden variables and it's totally possible to get the sizes with BeautifulSoup, each size is a <li>:
<li class="item" data-code="0648256001002">
<div class="picker-option"><button type="button" class="option"><span class="value">34</span></button></div>
</li>
You need to match the data-code attribute of the size to the data-articlecode attribute of the "product variant":
<li class="list-item">
<a title="Light beige/Patterned" data-color="Light beige/Patterned"
data-articlecode="0648256001">
...
</a>
</li>
I encourage you to implement this yourself, but I'll try to code it in the evening/tomorrow to make the answer complete. However, the website is rendered with JavaScript and in the response to the GET request, you won't get the entire HTML you see in the Elements tab of the DevTools. You can use Selenium to do that, but personally, I'd use Requests-HTML