How can I retrieve data from a web page with a loading screen? - python-3.x

Question

I am using the requests library to retrieve data from nitrotype.com/racer/insert_name_here about a user's progress using the following code:
import requests
base_url = 'https://www.nitrotype.com/racer/'
name = 'test'
url = base_url + name
page = requests.get(url)
print(page.text)
However my problem is that this retrieves data from the loading screen, I want the data after the loading screen.
Is it possible to do this and how?

Answer 1

This is likely because of dynamic loading and can easily be navigated by using selenium or pyppeteer.
In my example, I have used pyppeteer to spawn a browser and load the javascript so that I can attain the required information.
Example:
import pyppeteer
import asyncio
async def main():
# launches a chromium browser, can use chrome instead of chromium as well.
browser = await pyppeteer.launch(headless=False)
# creates a blank page
page = await browser.newPage()
# follows to the requested page and runs the dynamic code on the site.
await page.goto('https://www.nitrotype.com/racer/tupac')
# provides the html content of the page
cont = await page.content()
return cont
# prints the html code of the user profiel: tupac
print(asyncio.get_event_loop().run_until_complete(main()))