beautifulsoup4: how to parse a forum post? - python-3.x

Question

I have multiple occurences of the following (simplified) data structure as found in a forum software:
<li id="post12345" class="anchorFixedHeader" style="order: 1">
<div class="messagesidebar member" item-prop="author">
<div class="messageauthor">
<div class="messageauthorcontainer">
<a id="mac12">
<span class="username" itemprop="text">MostInnovativeUsernameEver</span>
</a>
</div>
</div>
</div>
<div class="messagecontent">
<div class="messagebody">
<div class="messagetext" itemprop="text">
Text before the quote.
<blockquote class="quotebox">
<div class="quoteboxcontent">
<p>
Hello, I'm a quote.
</p>
</div>
</blockquote>
Text after the class.
</div>
</div>
</div>
</li>
What I want to do for each occurence is to extract the username and for each username the corresponding messagecontent. I could do that succesfully, if there wasn't a single problem: the quote. When I print the extracted data in the console the data structure of the quote (naturally) gets messed up.
What I (seem) to need is the text before the quote, the quote itself and the text after the quote to deal with them separetely. I tried a bunch of stuff but don't quite find my way around in beautifulsoup just yet.
Ugh ... do you guys understand what I try to do?

Answer 1

Well, if I understood your question, here is a way to solve:
import re
import bs4
html = """<li id="post12345" class="anchorFixedHeader" style="order: 1">
<div class="messagesidebar member" item-prop="author">
<div class="messageauthor">
<div class="messageauthorcontainer">
<a id="mac12">
<span class="username" itemprop="text">MostInnovativeUsernameEver</span>
</a>
</div>
</div>
</div>
<div class="messagecontent">
<div class="messagebody">
<div class="messagetext" itemprop="text">
Text before the quote.
<blockquote class="quotebox">
<div class="quoteboxcontent">
<p>
Hello, I'm a quote.
</p>
</div>
</blockquote>
Text after the class.
</div>
</div>
</div>
</li>"""
forum_posts = []
re_break_line = re.compile(r'[^\n].*')
soup = bs4.BeautifulSoup(html, features='html.parser')
posts = soup.find_all('li')
for post in posts:
username = post.find('span', {'class': 'username'})
content = post.find('div', {'class': 'messagecontent'})
forum_post = {
'username': username.text,
'content': re_break_line.findall(content.text.replace(' ', ''))
}
forum_posts.append(forum_post)
print(forum_posts)
Output console:
[{'content': ['Text before the quote.', "Hello, I'm a quote.", 'Text after the class.'], 'username': 'MostInnovativeUsernameEver'}]