Extracting selective text using beautiful soup and write the result in CSV - python-3.x

Question

I am trying to extract selective text from website [https://data.gov.au/dataset?q=&groups=business&sort=extras_harvest_portal%20asc%2C%20score%20desc%2C%20metadata_modified%20desc&_organization_limit=0&organization=reservebankofaustralia&_groups_limit=0]
and have written code using beautiful soup:
`
wiki = "https://data.gov.au/dataset?q=&groups=business&sort=extras_harvest_portal+asc%2C+score+desc%2C+metadata_modified+desc&_organization_limit=0&organization=reservebankofaustralia&_groups_limit=0"
page= urllib.request.urlopen(wiki)
from bs4 import BeautifulSoup
import re
soup = BeautifulSoup(page)
data2 = soup.find_all('h3', class_="dataset-heading")
data3 = []
getdata = []
for link in data2:
data3 = soup.find_all("a", href=re.compile('/dataset/', re.IGNORECASE))
for data in data3:
getdata = data.text
print(getdata)
len(getdata)
`
My HTML is like :
<a href = "/dataset/banks-assets, class = "label" data-format = "xls">XLS<\a>
When I am running above code I am getting text that I want but 'XLS' word is coming alternatively, I want to remove 'XLS' and want to parse remaining text in csv in one column.My output is :
Banks – Assets
XLS
Consolidated Exposures – Immediate and Ultimate
Risk Basis
XLS
Foreign Exchange Transactions and Holdings of
Official Reserve Assets
XLS
Finance Companies and General Financiers
– Selected Assets and Liabilities
XLS
Liabilities and Assets –
Monthly XLS Consolidated Exposures – Immediate Risk Basis –
International Claims by Country
XLS
and so on.......
I checked whether above output is list. It was given list but it has only one element but as I have shown above my output is many texts.
Please help me out with it.

Answer 1

if the purpose is only to remove XLS rows from result column, then it can be reached, for example, ths way:
from urllib.request import urlopen
wiki = "https://data.gov.au/dataset?q=&groups=business&sort=extras_harvest_portal+asc%2C+score+desc%2C+metadata_modified+desc&_organization_limit=0&organization=reservebankofaustralia&_groups_limit=0"
page= urlopen(wiki)
from bs4 import BeautifulSoup
import re
soup = BeautifulSoup(page)
data2 = soup.find_all('h3', class_="dataset-heading")
data3 = []
getdata = []
for link in data2:
data3 = soup.find_all("a", href=re.compile('/dataset/', re.IGNORECASE))
for data in data3:
if data.text.upper() != 'XLS':
getdata.append(data.text)
print(getdata)
You will get a list with text you need. Then it can be easily transformed, for example, to DataFrame, where this data will appear as a column.
import pandas as pd
df = pd.DataFrame(columns=['col1'], data=getdata)
output:
col1
0 Banks – Assets
1 Consolidated Exposures – Immediate and Ultimat...
2 Foreign Exchange Transactions and Holdings of ...
3 Finance Companies and General Financiers – Sel...
4 Liabilities and Assets – Monthly
5 Consolidated Exposures – Immediate Risk Basis ...
6 Consolidated Exposures – Ultimate Risk Basis
7 Banks – Consolidated Group off-balance Sheet B...
8 Liabilities of Australian-located Operations
9 Building Societies – Selected Assets and Liabi...
10 Consolidated Exposures – Immediate Risk Basis ...
11 Banks – Consolidated Group Impaired Assets
12 Assets and Liabilities of Australian-Located O...
13 Managed Funds
14 Daily Net Foreign Exchange Transactions
15 Consolidated Exposures-Immediate Risk Basis
16 Public Unit Trust
17 Securitisation Vehicles
18 Assets of Australian-located Operations
19 Banks – Consolidated Group Capital
Putting to csv:
df.to_csv('C:\Users\Username\output.csv')