Get an xhr document that loads when you visit a page - python-3.x

Question

I tried to obtain the elements that we can see below the photos on the following site or on others, equivalent:
https://www.nosetime.com/xiangshui/947895-oulong-xuecheng-atelier-cologne-orange.html
https://www.nosetime.com/xiangshui/705357-pomelo-paradis.html
https://www.nosetime.com/xiangshui/592260-cl-mentine-california.html
https://www.nosetime.com/xiangshui/612353-oulong-atelier-cologne-trefle.html
https://www.nosetime.com/xiangshui/911317-oulong-nimingmeigui-atelier-cologne.html
But I can't get it from the source code. It is supposed to download dynamically with a javascript script. In fact it seems to be in an xhr document:
So how can I get an xhr document that is downloaded when I visit a page?
I tried:
url = "https://www.nosetime.com/xiangshui/350870-oulong-atelier-cologne-oolang-infini.html"
r = requests.post(url, headers=headers)
data = r.json()
print(data)
Pero me devuelve:
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
<ipython-input-8-e72156ddb336> in <module>()
2
3 r = requests.post(url, headers=headers)
----> 4 data = r.json()
5
6 print(data)
3 frames
/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
355 obj, end = self.scan_once(s, idx)
356 except StopIteration as err:
--> 357 raise JSONDecodeError("Expecting value", s, err.value) from None
358 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Answer 1

Just add the right headers and there you have the data.
import requests
headers = {
"referer": "https://www.nosetime.com/xiangshui/350870-oulong-atelier-cologne-oolang-infini.html",
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36",
}
response = requests.get("https://www.nosetime.com/app/item.php?id=350870", headers=headers).json()
print(response["id"], response["isscore"], response["brandid"])
For some reason I can't paste the entire JSON output as SO thinks this is spam... o.O. Anyhow, this should get you the JSON response.
This prints:
350870 8.6 10091761
EDIT:
If you have more products, you can simply look over the product URLS and extract from the JSON what you need. For example,
import requests
product_urls = [
"https://www.nosetime.com/xiangshui/947895-oulong-xuecheng-atelier-cologne-orange.html",
"https://www.nosetime.com/xiangshui/705357-pomelo-paradis.html",
"https://www.nosetime.com/xiangshui/592260-cl-mentine-california.html",
"https://www.nosetime.com/xiangshui/612353-oulong-atelier-cologne-trefle.html",
"https://www.nosetime.com/xiangshui/911317-oulong-nimingmeigui-atelier-cologne.html",
]
for product_url in product_urls:
headers = {
"referer": product_url,
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36",
}
product_id = product_url.split("/")[-1].split("-")[0]
response = requests.get(
f"https://www.nosetime.com/app/item.php?id={product_id}",
headers=headers,
).json()
print(f"Product name: {response['enname']} | Rating: {response['isscore']}")
Output:
Product name: Atelier Cologne Orange Sanguine, 2010 | Rating: 8.9
Product name: Atelier Cologne Pomelo Paradis, 2015 | Rating: 8.8
Product name: Atelier Cologne Clémentine California, 2016 | Rating: 8.6
Product name: Atelier Cologne Trefle Pur, 2010 | Rating: 8.6
Product name: Atelier Cologne Rose Anonyme, 2012 | Rating: 7.7