BS4 does not find li items? - python-3.x

Question

I am trying to scrape articles data from trading economics with requests and bs4. Below is the script I wrote:
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = 'https://tradingeconomics.com/stream?c=country+list'
r = requests.get(url).text
parser = BeautifulSoup(r)
li = parser.find_all('li', class_='list-group-item')
print(li)
I can see from inspecting the webpage that each article is under a li item and has class='list-group-item'. Strangely, bs4 cannot find any li items, returning None. What can be the case?

Answer 1

The data you see on the page is loaded dynamically via JavaScript, so beautifulsoup doesn't see it. You can simulate the javascript request with this example:
import json
import requests
url = "https://tradingeconomics.com/ws/stream.ashx"
params = {"start": 0, "size": 20, "c": "country list"}
data = requests.get(url, params=params).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for item in data:
print(item["title"])
print(item["description"])
print(item["url"])
print("-" * 80)
Prints:
Week Ahead
Central banks in the US, Brazil, Japan, the UK and Turkey will be deciding on monetary policy in the coming week, while key data to watch for include US and China industrial output and retail sales; Japan and Canada inflation data; UK consumer morale; Australia employment figures and retail trade; New Zealand Q4 GDP; and India wholesale prices.
/calendar?article=29075&g=top&importance=2&startdate=2021-03-12
--------------------------------------------------------------------------------
Week Ahead
The US Senate will be debating and voting on the President Biden's $1.9 trillion coronavirus aid bill during the weekend, and send it back to the House to be signed into law as soon as next week. Elsewhere, monetary policy meetings in the Eurozone and Canada will be keenly watched, as well as updated GDP figures for Japan, the Eurozone, the UK and South Africa. Other important releases include US, China and India inflation data; US and Australia consumer sentiment; UK and China foreign trade; Eurozone and India industrial output; and Japan current account.
/calendar?article=29072&g=top&importance=2&startdate=2021-03-05
--------------------------------------------------------------------------------
Week Ahead
After President Biden's $1.9 trillion pandemic aid bill narrowly passed the House in the early hours of Saturday, all attention would move now to vote in Senate. Meanwhile, investors will also turn the eyes to US jobs report due Friday; GDP data for Australia, Brazil, Canada and Turkey as well as worldwide manufacturing and services PMI surveys and monetary policy action by the Reserve Bank of Australia. Other releases include trade figures for the US and Canada, factory orders for the US and Germany, unemployment rate for the Euro Area and Japan, and industrial output and retail sales for South Korea.
/calendar?article=29071&g=top&importance=2&startdate=2021-02-27
--------------------------------------------------------------------------------
...and so on.