How to pull out a dta when it shows <-!--Content End--> after the JavaScript? - python-3.x

Question

I've just started to learn python. I want to extract glass temperature data every three hours for academic purpose. The website is below:
https://www.weather.gov.hk/wxinfo/ts/display_graph_grass_e.htm?kp&
I try to use BeautifulSoup to pull out the data using the below script. Here is the result:
Before I find the wanted data, there is a < !--Content End-- > after the JavaScript and I can't scrape the script behind it. Why would that happen and if there is any solution for that?

Answer 1

The data is stored as Javascript Array in the HTML page. We can use re and ast.literal_eval (doc) to retrieve it:
import re
import requests
from ast import literal_eval
url = 'https://www.weather.gov.hk/wxinfo/ts/display_graph_grass_e.htm?kp&'
html_text = requests.get(url).text
station_code = literal_eval(re.findall(r'StationCode\s*=.*?(\(.*?\))', html_text)[0])
station_name = literal_eval(re.findall(r'stnname\s*=.*?(\(.*?\))', html_text)[0])
station_height = literal_eval(re.findall(r'stn_height\s*=.*?(\(.*?\))', html_text)[0])
grass_temp = literal_eval(re.findall(r'grasstemp\s*=.*?(\(.*?\))', html_text)[0])
min_since_17 = literal_eval(re.findall(r'minSince17\s*=.*?(\(.*?\))', html_text)[0])
min_hour = literal_eval(re.findall(r'minHour\s*=.*?(\(.*?\))', html_text)[0])
min_minute = literal_eval(re.findall(r'minMinute\s*=.*?(\(.*?\))', html_text)[0])
rows = [*zip(station_code, station_name, station_height, grass_temp, min_since_17, min_hour, min_minute)]
headers = ['Station Code', 'Station Name', 'Station Height', 'Grass Temp', 'Min_since_17', 'Min Hour', 'Min Minute']
print(''.join('{: <20}'.format(d) for d in headers))
for row in rows:
print(''.join('{: <20}'.format(d) for d in row))
Prints:
Station Code Station Name Station Height Grass Temp Min_since_17 Min Hour Min Minute
kp King's Park 65 25.4 25.3 23 35
tkl Ta Kwu Ling 15 25.4 24.8 17 00
tms Tai Mo Shan 955 21.3 21.3 07 19