Scraping by Xpath in Scrapy - python-3.x

Question

I want to scrape articles from web page (example article enter link description here). My code should scrape all of article text. I'm doing it by XPath. After pasting following XPath in Dev tools: (1.crtl+shift+i ///
2. ctrl+f)
//div[#class="item-page clearfix"]/*[self::p/text() or self::strong/text() or self::ol/text() or self::blockquote/text()]
It seems like it works and is able to find all text. Web page shows me that XPath is working properly. But my Python and Scrapy thinks otherwise. The code below in JSON is returning only first paragraph of article. I can't understand why. Why on web page it's working and in Python not? What I missed?
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from w3lib.html import remove_tags
class LubaczowSpider(CrawlSpider):
name = 'Lubaczow'
allowed_domains = ['zlubaczowa.pl']
start_urls = ['http://zlubaczowa.pl/index.php/']
rules = (
Rule(LinkExtractor(restrict_xpaths="//p[#class='readmore']/a"), callback='parse', follow=True),)
def parse(self, response):
yield {
"Text" : response.xpath('normalize-space(//div[#class="item-page clearfix"]/*[self::p/text() or self::strong/text() or self::ol/text() or self::blockquote/text()])').getall(),
"Url" : response.url
}
Thank you in advance for your suggestions and help!

Answer 1

When you use normalize-space in xpath version 1 (which I believe is used in scrapy), any trailing white space(s) is removed from the string before being returned see mdn. This has the effect that text nodes following each other will have the nodes after the first one replaced with a white space hence you only get the first paragraph back.
You can try to obtain all text data from the child nodes and then join them into one string. See sample code below
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from w3lib.html import remove_tags
class LubaczowSpider(CrawlSpider):
name = 'Lubaczow'
allowed_domains = ['zlubaczowa.pl']
start_urls = ['http://zlubaczowa.pl/index.php/']
rules = (
Rule(LinkExtractor(restrict_xpaths="//p[#class='readmore']/a"), callback='parse', follow=True),)
def parse(self, response):
all_text = response.xpath("//div[#class='item-page clearfix']//child::text()").getall()
text = ''.join([r.strip() for r in all_text]) # remove trailing spaces and combine into 1 string
yield {
"Text" : text,
"Url" : response.url
}
A sample screenshot showing the results of the above code is as shown below