How to extract texture strings between two tags with different level with Python? - python-3.x

Question

I want to extract the all contents between two tags with different level.
After google, i can't find the effective solutions for my need.
I expect you have solultions with Python.
so please see the html code below:
<span id='info'>
<span>
<span class='cl'>text1</span>
text2 is Chinese string.
text3
</span>
<br 1>
<span class='cl'> text4</span> text5
<br 2>
<span class='cl'>
<span>text7</span>
text8
text9
</span>
<br 3>
<span class='cl'> text10</span> text11
<br 4>
<span class='cl'> text12</span> text13
<br 5>
</span>
And, before first <br 1> tag, the html in some pages is like below:
# one situation below:
<span id='info'>
<span>
<span class='cl'>text1</span>
text2 is Chinese string.
text2 is Chinese string.
text3
</span>
<br 1>
<span class='cl'> text4</span> text5
......other html....
</span>
# two situation below:
<span id='info'>
<span class='cl'>text1</span>
text2 is Chinese string.
text3
<br 1>
<span class='cl'> text4</span> text5
......other html....
</span>
I want to extract contents:
text1 text2 text3 .... text13
I tried Xpath and bs4 for very very more methods, but there were no ok for my need.
Could you tell me the right way to use Xpath or bs4 or other mehods to get the expected output above?
Thank you in advance!
Xpath I tried:
str = response.xpath("//span[#id='info']/descendant::span[contains(text(),'text1')]/following::br[1]/preceding-sibling::node()")
str = str.xpath('string(.)').extract()
print(str)
then i got(like below):
[' \n \n text1 \n \n tex2 \n tex3 \n \n]
Above is the contents before first <br > tag. Because the html tags is not stable in different pages eventhough they are from same website, in the area of <span id='info'>. So i have to extract the contents between two neighbouring <br > tags separately.
And soup didn't work, i did not study it.
For my want, just the <br> tags have the stable postions and status. So i want use <br> tags to position the informations ( one br, one infomation ).
So, how can i do?

Answer 1

Give this a try for a BeautifulSoup solution... It's almost completely stolen from this answer: https://stackoverflow.com/a/1983219/684776
from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
def tag_visible(element):
if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
return False
if isinstance(element, Comment):
return False
return True
def text_from_html(body):
soup = BeautifulSoup(body, 'html.parser')
texts = soup.findAll(text=True)
visible_texts = filter(tag_visible, texts)
return list(filter(lambda x: x != '', [t.strip() for t in visible_texts]))
html = """
<span id='info'>
<span class='cl'>
<span>text1</span>
text2
text3
</span>
<br 1>
<span class='cl'> text4</span> text5
<br 2>
<span class='cl'>
<span>text7</span>
text8
text9
</span>
<br 3>
<span class='cl'> text10</span> text11
<br 4>
<span class='cl'> text12</span> text13
<br 5>
</span>
"""
print(text_from_html(html))
Test in on Repl.it: https://repl.it/#mac9416/SO-57890610

Answer 2

Generally, don't use regex to parse HTML.
But if all the HTML you want to parse looks very similar to your example, then... do what works!
(?<=>)(?:\s*)[^(<|\s).]+(?=\s*<)
That will capture the text and leading whitespace. Then you can .trim() to the content you want.
See it in action on regex101.