Scrapy Crawler gets terminated at random pages - python-3.x

Question

I'm new to Scrapy. I'm crawling the r/india subreddit using a recursive parser to store the title, upvotes and URLs of each thread. It works all fine but the Scraper ends unexpectedly with a weird error that shows:
2018-04-29 00:01:12 [scrapy.core.scraper] ERROR: Spider error processing
<GET https://www.reddit.com/r/india/?count=50&after=t3_8fh5nv> (referer:
https://www.reddit.com/r/india/?count=25&after=t3_8fiqd5)
Traceback (most recent call last):
File "Z:\Anaconda\lib\site-packages\scrapy\utils\defer.py", line 102, in
iter_errback
yield next(it)
File "Z:\Anaconda\lib\site-packages\scrapy\spidermiddlewares\offsite.py",
line 30, in process_spider_output
for x in result:
File "Z:\Anaconda\lib\site-packages\scrapy\spidermiddlewares\referer.py",
line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "Z:\Anaconda\lib\site-packages\scrapy\spidermiddlewares\urllength.py",
line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "Z:\Anaconda\lib\site-packages\scrapy\spidermiddlewares\depth.py", line
58, in <genexpr>
return (r for r in result or () if _filter(r))
File
"C:\Users\jayes\myredditscraper\myredditscraper\spiders\scrapereddit.py",
line 28, in parse
yield Request(url=(next_page),callback=self.parse)
File "Z:\Anaconda\lib\site-packages\scrapy\http\request\__init__.py", line
25, in __init__
self._set_url(url)
File "Z:\Anaconda\lib\site-packages\scrapy\http\request\__init__.py", line
62, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url:
2018-04-29 00:01:12 [scrapy.core.engine] INFO: Closing spider (finished)
And the error comes at random pages each time the spider is run, making it impossible for me to detect what's causing the problem. Here's my redditscraper.py file which contains the code(I've also used Pipeline and Items.py but that doesn't contain any problems I feel)
import scrapy
import time
from scrapy.http.request import Request
from myredditscraper.items import MyredditscraperItem
class ScraperedditSpider(scrapy.Spider):
name = 'scrapereddit'
allowed_domains = ['www.reddit.com']
start_urls = ['http://www.reddit.com/r/india/']
def parse(self,response):
next_page=''
titles=response.css("a.title::text").extract()
links=response.css("a.title::attr(href)").extract()
votes=response.css("div.score.unvoted::attr(title)").extract()
for item in zip(titles,links,votes):
new_item = MyredditscraperItem()
new_item['title']=item[0]
new_item['link']=item[1]
new_item['vote']=item[2]
yield new_item
next_page = response.css("span.next-
button").css('a::attr(href)').extract()[0]
if next_page is not None:
yield Request(url=(next_page),callback=self.parse)

Answer 1

As your exception says
ValueError: Missing scheme in request url:
thats means you try to scrap an invalid url - missing http:// or https:// in the url.
I guess the problem is not in the start_urls because otherwise the parse function won't be called. The problem is in the parse function.
When yield Request is called you need to check if next_page contains schema. it seems like the urls you parse are relative links, so you have two options to continue scrap those links without facing this exception:
Passing absolute url.
Skipping relative urls.