Unable to get the contents of a webpage [closed] - get

Question

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 days ago.
Improve this question
Im writing a script and need the contents of for example this webpage:
https://pcb.inc.hp.com/webapp/#/nl-nl/contents/33128146?type=I&hierarchy=F&status=L&status=O
Im using scrapy and usually all works out but I am currently unable to get the html of this page with Requests or scrapy or any other module. Does someone know what could be going wrong?

Answer 1

The website is using AngularJS to dynamically generate content upon loading. You cannot scrape the contents directly from this website, instead I'd suggest using something like Selenium with Python to scrape data.
Or conversely, depending on what you need, you can check the Network tab in Chrome Dev Tools to see the requests being made, and scrape the data from those URLs.
For instance
Request URL: https://pcb.inc.hp.com/api/catalogs/nl-nl/nodes/0/children?status[]=O&status[]=L&hierParadigm=F
Response: {"baseProdname":"ROOT_NODE","oid":0,"level":0,"status":["O","L"],"cultureCode":"nl-nl","children":[{"baseProdname":"Solutions","oid":8176594,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Scanners/Copiers/Faxes","oid":15179,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Software","oid":8133386,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Ink/Toner/Paper/Printer Supplies","oid":12771,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Laptops and Hybrids","oid":321957,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Printers and Multifunction","oid":18972,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Point of Sale Systems","oid":7491307,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Desktops & Workstations","oid":12454,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Monitors","oid":382087,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Services","oid":8362107,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Accessories","oid":8386448,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"3D Materials and Consumables","oid":20063457,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Handhelds and Calculators","oid":215348,"level":1,"status":["L","O"],"cultureCode":"nl-nl"},{"baseProdname":"Industries","oid":20008722,"level":1,"status":["L"],"cultureCode":"nl-nl"},{"baseProdname":"Tablets","oid":5169094,"level":1,"status":["O"],"cultureCode":"nl-nl"},{"baseProdname":"Projectors","oid":3338965,"level":1,"status":["O"],"cultureCode":"nl-nl"},{"baseProdname":"Digital Cameras and Photo Studios","oid":382085,"level":1,"status":["O"],"cultureCode":"nl-nl"}]}