Python/Selenium - How to process extracted data from App store - python-3.x

Question

I am using Selenium/Python to parse reviews from Apple app store. I used the following code to extract the data for the first five reviews:
URL: https://apps.apple.com/us/app/lemonade-insurance/id1055653645#see-all/reviews
wait = WebDriverWait(driver, 5)
response_ratings = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".we-customer-review")))
response_container = []
for e in response_ratings[:5]:
response_container.append(e.get_attribute('innerHTML'))
print(response_container[0])
Then, I print the first output
I expect to have star 5 out of 5, date July 6, 2019, title Convenient and Affordable!!!!, review The Lemonade app is so easy to use as well as having affordable rates!..., and Developer Response Thanks so much for your awesome review!! We're so happy to have you in... for the first review.
How do I get the above info? Thank you in advance for the help

Answer 1

You can use BeautifulSoup to parse the innerHTML and get what you're looking for.
One way of doing it would be:
import re
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
link = 'https://apps.apple.com/us/app/lemonade-insurance/id1055653645#see-all/reviews'
stars = re.compile(r"\d out of \d")
with webdriver.Chrome() as driver:
wait = WebDriverWait(driver, 10)
driver.get(link)
elements = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".we-customer-review")))
for elem in elements:
s = BeautifulSoup(elem.get_attribute("innerHTML"), "html.parser")
review_date = s.find("time").text
review_body = s.find("p").text
review_title = s.find("h3", {"data-test-customer-review-title": ""}).text.strip()
review_stars = ''.join(re.findall(stars, str(s.find("figure"))))
dev_response = s.find_all("p", {"data-test-bidi": ""})
print(f"{review_title} | {review_date} | {review_stars}")
print(review_body)
print(dev_response[1].text if len(dev_response) > 1 else "")
print("-" * 80)
This prints:
Convenient and Affordable!!!! | 07/06/2019 | 5 out of 5
The Lemonade app is so easy to use as well as having affordable rates! It took me all of 15 minutes to sign-up, pick a coverage and a deductible. Very nice customer service as well as very informative. At anytime of any day of the week I can log onto my account and check everything as well as make any necessary changes that I may need or want. The chat option feature works fantastic! Whenever I have a question I just go to the chat feature and within seconds someone is there to help and answer all my questions regarding their services and my plan coverage. I wished they had a referral feature cause I’ve already set up a couple of family members with the company as well. They were amazed that it only took about 10-15 minutes to setup and just how affordable it is!! I’ve gotten quotes from so many other companies but the monthly payments and deductible were too expensive, I was a little hesitant at first but I said hey I should at least give it a try and so far so good!!! I’m hoping that it’ll never come to the point where I’ll actually need to file a claim but if so I feel confident that the process will be easy and stress free considering how much stress I’m going to actually have due to a burglary or theft. I have faith that we’re going to have a very long relationship. Thanks to all the developers of the Lemonade App, the name of the company is nice too!!!!
Thanks so much for your awesome review!! We're so happy to have you in our Lemonade community!
--------------------------------------------------------------------------------
Made the best lemonade I’ve ever had! | 07/01/2018 | 5 out of 5
I have been telling EVERYONE about Lemonade. I don’t know how, but, getting insurance through your company is actually FUN! I have never had so much FUN doing a chore that typically involves a boring Q & A. The app really made me feel like I was getting insurance through a friend. I smiled with the shout out to my horoscope sign after entering my birthday, I loved the “making lemonade” process when getting the quote, and (because I have a future date for the policy to start) I absolutely adore the countdown. I look at it almost daily and become even more excited for my move (and I really don’t like having to move so this is really helping). The use of unclaimed funds going to charity actually makes my heart melt. By providing freedom to choose what organization you would like to contribute to truly makes me feel like I am giving back in some way, and it is beyond noble and inspiring for you to use that money to help others turn lemons into lemonade. Also, your customer service has been impeccable. Every question I have had has been answered quickly and by a friendly representative of the company. I don’t know much about the insurance world, other than we need to have it, but, you make me want to work for you!!! Where do I sign up?
--------------------------------------------------------------------------------