How do I put my scraped data into a data frame - text

Question

Please i need help, am having trouble trying to put my scraped data into a data frame that has 3 columns i.e. date, source and keywords extracted from each scraped website for further text analysis, my code is borrowed from https://stackoverflow.com/users/12229253/foreverlearning and is given below:
from newspaper import Article
import nltk
nltk.download('punkt')
urls = ['https://dailypost.ng/2022/02/02/securing-nigeria-duty-of-all-tiers-of-government-oyo-senator-balogun/', 'https://guardian.ng/news/declare-bandits-as-terrorists-senate-tells-buhari/', 'https://www.thisdaylive.com/index.php/2021/10/24/when-will-fg-declare-bandits-as-terrorists/', 'https://punchng.com/rep-wants-buhari-to-name-lawmaker-sponsoring-terrorism/', 'https://punchng.com/national-assembly-plans-to-meet-us-congress-over-875m-weapons-deal-stoppage/']
results = {}
for url in urls:
article = Article(url)
article.download()
article.parse()
article.nlp()
results[url] = article
for url in urls:
print(url)
article = results[url]
print(article.authors)
print(article.publish_date)
print(article.keywords)

Answer 1

I played around with it and here is how you can make it into a data frame. Assuming that you wanted to use pandas in the first place:
import nltk
import pandas as pd
from newspaper import Article
nltk.download('punkt')
urls = ['https://dailypost.ng/2022/02/02/securing-nigeria-duty-of-all-tiers-of-government-oyo-senator-balogun/', 'https://guardian.ng/news/declare-bandits-as-terrorists-senate-tells-buhari/', 'https://www.thisdaylive.com/index.php/2021/10/24/when-will-fg-declare-bandits-as-terrorists/', 'https://punchng.com/rep-wants-buhari-to-name-lawmaker-sponsoring-terrorism/', 'https://punchng.com/national-assembly-plans-to-meet-us-congress-over-875m-weapons-deal-stoppage/']
# create a data frame with the needed columns
saved_data = pd.DataFrame(columns=['Date', 'Source', 'KeyWords'])
# put into a data frame that has 3 columns i.e. date, source and keywords
def add_data_to_df(urls, saved_data):
for url in urls: # process each url separately
article = Article(url)
article.download()
article.parse()
article.nlp()
# create a row with the data you need using attributes
record = {'Date': article.publish_date, 'Source': url, 'KeyWords': article.keywords}
# append info about each url as a new row
saved_data = saved_data.append(record, ignore_index = True)
return saved_data
Now, when you run this function
add_data_to_df(urls, saved_data), you should see a data frame with contents similar to the ones I got below during testing:
Date Source KeyWords
0 2022-02-02 00:00:00 https://dailypost.ng/2022/02/02/securing-niger... [nigeria, securing, oyo, state, senator, prote...
1 2021-09-30 04:25:24+00:00 https://guardian.ng/news/declare-bandits-as-te... [shutdown, terrorists, nigeria, guardian, decl...
2 2021-10-24 00:00:00 https://www.thisdaylive.com/index.php/2021/10/... [terrorists, nigerian, declare, state, militar...
3 2021-10-05 14:41:48+00:00 https://punchng.com/rep-wants-buhari-to-name-l... [president, buhari, house, national, lawmaker,...
4 2021-07-31 00:30:47+00:00 https://punchng.com/national-assembly-plans-to... [plans, congress, deal, nigeria, nigerian, rig...
(Sorry for the format, I am showing the output as plain text since I am not allowed to attach screenshots, but you will have a nice pandas format)
Edit: adding a function to save the data frame to a csv file. Note that this one of the shortest ways of doing this and it will save the file to the current working directory, i.e., where you are executing your code:
# this function saves given data to csv
def save_to_csv(saved_data):
saved_data.to_csv('output.csv', index=False, sep=',')
# process the articles and create a csv
save_to_csv(add_data_to_df(urls, saved_data))