How to select all nodes beetwen certain headers? - python-3.x

Question

Each <header> tag contains a Title of Conference.
Each <ul> tag contains the links of this conference.
When I'll to try to crawl the website, I'm try to associating the <header> tag with yours links in <ul> tags. But I don't know how I can only select the <ul> tags of are sibling two certain <headers>.
HTML:
<header>... 0 ... </header>
<ul class="publ-list">... 0 ...</ul>
<header>... 1 ... </header>
<ul class="publ-list">... 0 ...</ul>
<header>... 2 ... </header>
<ul class="publ-list">... 0 ...</ul>
<p>...</p>
<ul class="publ-list">... 1 ...</ul>
<header>... 3 ...</header>
<ul class="publ-list">... 0 ...</ul>
<ul class="publ-list">... 1 ...</ul>
<ul class="publ-list">... 2 ....</ul>
<ul class="publ-list">... 3 ....</ul>
<ul class="publ-list">... 4 ....</ul>
<header>... 4 ...</header>
Example:
<ul> tags are sibling of header[0] and header[1]
<ul class="publ-list">... 0 ...</ul>
<ul> tags are sibling of header[2] and header[3]
<ul class="publ-list">... 0 ...</ul>
<ul class="publ-list">... 1 ...</ul>
Some cases:
It's possible more than one ul tag between header tag
Sometimes has a p tag between ul tags
All tags are siblings!
All ul has class "publ-list"
My code:
TITLE_OF_EDITIONS_SELECTIOR = 'header h2'
GROUP_OF_TYPES_OF_EDITION_SELECTOR = ".publ-list"
size_editions = len(response.css(GROUP_OF_TYPES_OF_EDITION_SELECTOR))
i = 0
while i < size_editions:
# Get the title of conference
title_edition_conference = response.css(TITLE_OF_EDITIONS_SELECTIOR)[i]
# Get datas and links of <ul> tags "(.publ-list)"
TYPES_OF_CONFERENCE = response.css(GROUP_OF_TYPES_OF_EDITION_SELECTOR)[i]
TYPE = TYPES_OF_CONFERENCE.css('.entry')
types_of_edition = {}
size_type_editions = 0
for type_of_conference in TYPE:
title_type = type_of_conference.css('.data .title ::text').extract()
link_type = type_of_conference.css('.publ ul .drop-down .body ul li a ::attr(href)').extract_first()
types_of_edition[size_type_editions] = {
"title": title_type,
"link": link_type,
}
size_type_editions = size_type_editions + 1
editions[i] = {
"title_edition_conference": title_edition_conference,
"types_of_edition": types_of_edition
}
i = i + 1
Problem of My Code
Sometimes there are many ul tags
Sometimes has a <p> tag and it's break my xPath, and get only the previous <ul> tags.
I got it testing with JQuery on Console of Google Chrome, example:
"$($('header')[0]).nextUntil($('header')[1])"
But How I can select this using xPath or CSS Selector? Thank you!

Answer 1

Following combination of css selectors and python for loop can solve this task.
from parsel import Selector
html = """
<ul class="publ-list">p1</ul>
<header>h1</header>
<ul class="publ-list">p2</ul>
<header>h2</header>
<ul class="publ-list">p3</ul>
<header>h3</header>
<ul class="publ-list">p4</ul>
<p>p_tag_1</p>
<ul class="publ-list">p5</ul>
<header>h4</header>
<ul class="publ-list">p6</ul>
<ul class="publ-list">p7</ul>
<header>h5</header>
<ul class="publ-list">p8</ul>
"""
response = Selector(text=html)
tags = response.css("header, ul")
output = {}
key = False
for t in tags:
if key and "<ul" in t.css("*").extract_first():
output[key].append(t.css("::text").extract_first())
elif "<header>" in t.css("*").extract_first():
key = t.css("::text").extract_first()
if key not in output.keys():
output[key]=[]
else:
pass
print(output)
Output is:
{'h1': ['p2'], 'h2': ['p3'], 'h3': ['p4', 'p5'], 'h4': ['p6', 'p7'], 'h5': ['p8']}
This css selector: tags = response.css("header, ul") returns list of <header> and <ul> tags in the same order as in the html code.
After that we can iterate through received tags using for loop and select required data.

Answer 2

Try to use following-sibling like here:
>>> txt = """<header>..</header>
... <ul class="publ-list">...</ul>
... <header>..</header>
... <ul class="publ-list">...</ul>
... <header>..</header>
... <ul class="publ-list">...</ul>
... <p>...</p>
... <ul class="publ-list">...</ul>
... <header>..</header>
... <ul class="publ-list">...</ul>
... <ul class="publ-list">...</ul>
... <header>..</header>"""
>>> from scrapy import Selector
>>> sel = Selector(text=txt)
>>> sel.xpath('//header/following-sibling::*[not(self::header)]').extract()
[u'<ul class="publ-list">...</ul>', u'<ul class="publ-list">...</ul>', u'<ul class="publ-list">...</ul>', u'<p>...</p>', u'<ul class="publ-list">...</ul>', u'<ul class="publ-list">...</ul>', u'<ul class="publ-list">...</ul>']
So with //header/following-sibling::*[not(self::header)] we choose all header siblings, but not header.

Answer 3

This may be what you're looking for.
html = """
<ul class="publ-list">...</ul>
<header>..</header>
<ul class="publ-list">...</ul>
<header>..</header>
<ul class="publ-list">...</ul>
<header>..</header>
<ul class="publ-list">...</ul>
<p>...</p>
<ul class="publ-list">...</ul>
<header>..</header>
<ul class="publ-list">...</ul>
<ul class="publ-list">...</ul>
<header>..</header>
<ul class="publ-list">...</ul>
"""
Note I added a <ul>before the first and after the last <header>..</header> sets.
This expression
//ul[
preceding-sibling::header
and
following-sibling::header
]
should select all the <ul> tags, except those I added before and after, and none of the <p> tags which may be in the way.