How do I combine paragraphs of web pages (from a text file containing urls)? - python-3.x

Question

I want to read all the web pages and extract text from them and then remove white spaces and punctuation. My goal is to combine all the words in all the webpage and produce a dictionary that counts the number of times a word appears across all the web pages.
Following is my code:
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as ureq
import re
def web_parsing(filename):
with open (filename, "r") as df:
urls = df.readlines()
for url in urls:
uClient = ureq(url)
page_html = uClient.read()
uClient.close()
page_soup = soup(page_html, "html.parser")
par = page_soup.findAll('p')
for node in par:
#print(node)
text = ''.join(node.findAll(text = True))
#text = text.lower()
#text = re.sub(r"[^a-zA-Z-0-9 ]","",text)
text = text.strip()
print(text)
The output I got is:
[Paragraph1]
[paragraph2]
[paragraph3]
.....
What I want is:
[Paragraph1 paragraph2 paragraph 3]
Now, if I split the text here it gives me multiple lists:
[paragraph1], [paragraph2], [paragraph3]..
I want all the words of all the paragraphs of all the webpages in one list.
Any help is appreciated.

Answer 1

As far as I understood your question, you have a list of nodes from which you can extract a string. You then want these strings to be merged into a single string. This can simply done by creating an empty string and then adding the subsequent strings to it.
result = ""
for node in par:
text = ''.join(node.finAll(text=True)).strip()
result += text
print(result) # "Paragraph1 Paragraph2 Paragraph3"
prin([result]) # ["Paragraph1 Paragraph2 Paragraph3"]