Scrape Table With JavaScript "Show More Button" In Webpage - python-3.x

Question

I am attempting to pull a data table from a NYT open-access web article on the number of COVID-19 cases, which can be found here. The table shows the top 10 states with highest number of cases, and expands to all 50 states and U.S. territories upon clicking the "Show more" button.
The HTML portion of the table is as follows:
Using this tutorial, I have written the following code utilizing Selenium to try clicking this button, and pass this page off to BeautifulSoup to begin synthesizing for use in Pandas. My initial code looks as follows:
from bs4 import BeautifulSoup
import selenium
import time
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("/usr/bin/chromedriver", chrome_options=options)
driver.get("https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html")
At this juncture, I am not sure how to execute clicking the button (found in the HTML snippet: <button class="svelte-1tjczrs">Show more</button>), and stage it for BeautifulSoup.
Any help is greatly appreciated!

Answer 1

Try the following attempt. It should fetch you the required content unveiling show more button and put the same in a dataframe.
import pandas
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html"
with webdriver.Chrome() as driver:
driver.get(link)
datalist = []
show_more = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"#g-cases-by-state button[class^='svelte']")))
driver.execute_script("arguments[0].click();",show_more)
for elem in WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"#g-cases-by-state table[class^='svelte'] tr"))):
data = [item.text for item in elem.find_elements_by_css_selector("th,td")]
datalist.append(data)
df = pandas.DataFrame(datalist)
print(df)