i am using proxy api to test and scrape movies from imdb but all i get is [scrapy.core.engine] DEBUG: Crawled (200) - python-3.x

Question

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scraper_api import ScraperAPIClient
client = ScraperAPIClient('hiding the key')
class MoviesSpider(CrawlSpider):
name = "movies"
def start_requests(self):
urls = ["https://www.imdb.com/search/title/?genres=drama&groups=top_250&sort=user_rating,desc"]
for link in urls:
yield scrapy.Request(client.scrapyGet(url=link,render=True),callback=self.parse_item)
rules = (Rule(LinkExtractor(restrict_xpaths='//h3[#class="lister-item-header"]/a'), callback="parse_item", follow=True),)
def parse_item(self, response):
yield {'link':response.url}
[scrapy.core.engine] DEBUG: Crawled (200) <GET https://api.scraperapi.com/?url=https%3A%2F%2Fwww.imdb.com%2Fsearch%2Ftitle%2F%3Fgenres%3Ddrama%26groups%3Dtop_250%26sort%3Duser_rating%2Cdesc&api_key=8ccf268c7e3c965da0777f5594598b9d&render=true&scraper_sdk=python%5C\> (referer: None)
2023-02-19 17:48:00 [scrapy.core.scraper] DEBUG: Scraped from <200 https://api.scraperapi.com/?url=https%3A%2F%2Fwww.imdb.com%2Fsearch%2Ftitle%2F%3Fgenres%3Ddrama%26groups%3Dtop_250%26sort%3Duser_rating%2Cdesc&api_key=8ccf268c7e3c965da0777f5594598b9d&render=true&scraper_sdk=python%5C\>
{'link': 'https://api.scraperapi.com/?url=https%3A%2F%2Fwww.imdb.com%2Fsearch%2Ftitle%2F%3Fgenres%3Ddrama%26groups%3Dtop_250%26sort%3Duser_rating%2Cdesc&api_key=8ccf268c7e3c965da0777f5594598b9d&render=true&scraper_sdk=python'}
i was expecting to get ['link': name of the link]

Answer 1

You can try using a standard spider instead of the crawlspider. If all you are trying to scrape is the urls then this is the way to go anyway since this way you only make a single request instead of one for each link.
class MoviesSpider(scrapy.Spider):
name = "movies"
def start_requests(self):
urls = ["https://www.imdb.com/search/title/?genres=drama&groups=top_250&sort=user_rating"]
for link in urls:
yield scrapy.Request(client.scrapyGet(url=link,render=True))
def parse(self, response):
for link in response.xpath("//h3/a/#href").getall():
full_link = response.urljoin(link)
yield {'link': full_link}