Using Scrapy to Extract Wikipedia Links - python-3.x

Question

I am learning about scrapy and am trying to use it to scrape the below Wikipedia page:
https://en.wikipedia.org/wiki/List_of_sovereign_states_in_the_2020s
I would like to scrape each country and the hyperlink attached to that country and below is my code so far:
import scrapy
class CountrypopSpider(scrapy.Spider):
name = 'countryPop'
allowed_domains = ['en.wikipedia.org']
start_urls = ['https://en.wikipedia.org/wiki/List_of_sovereign_states_in_the_2020s']
def parse(self, response):
countries = response.xpath('//table//b//#title').extract()
for country in countries:
country_url = response.xpath('//table//b[contains(#href, 'Afghanistan')]').extract()
yield {'countries': country}
What it currently does is get all the countries from the main table and then I want it to loop through each of these countries, using the country name to get the url. I am having trouble though finding a way of using the country name to find the url, my latest attempt was using contains().
Any other comments on my scraping code would be appreciated.
Thanks

Answer 1

Try this
Approach 1
import scrapy
class CountrypopSpider(scrapy.Spider):
name = 'countryPop'
allowed_domains = ['en.wikipedia.org']
start_urls = ['https://en.wikipedia.org/wiki/List_of_sovereign_states_in_the_2020s']
def parse(self, response):
coutries=200
cnames=['Australia','Bhutan']
noduplicateset= set()
for cname in cnames:
for title in response.xpath('//table[1]//a[contains(#title,'+cname+')]'):
if cname not in noduplicateset:
yield {cname:'https://en.wikipedia.org'+title.css('a').get().split("\"")[1]}
noduplicateset.add(cname)
Approach 2
import scrapy
class CountrypopSpider(scrapy.Spider):
LOG_LEVEL = 'INFO'
name = 'countryPop'
allowed_domains = ['en.wikipedia.org']
start_urls = ['https://en.wikipedia.org/wiki/List_of_sovereign_states_in_the_2020s']
def parse(self, response):
coutries=200
cnames=['Australia','Bhutan']
for i in range(5,coutries):
for title in response.xpath('//*[#id="mw-content-text"]/div/table[1]/tbody/tr['+str(i+2)+']/td[1]/b/a'):
name=title.css('a ::text').get()
if name in cnames:
yield {name:'https://en.wikipedia.org'+title.css('a').get().split("\"")[1]}
If you output to json file, it will look like this