Force my scrapy spider to stop crawling
In the latest version of Scrapy, available on GitHub, you can raise a CloseSpider exception to manually close a spider. In the 0.14 release note doc is mentioned: “Added CloseSpider exception to manually close spiders (r2691)” Example as per the docs: def parse_page(self, response): if ‘Bandwidth exceeded’ in response.body: raise CloseSpider(‘bandwidth_exceeded’) See also: http://readthedocs.org/docs/scrapy/en/latest/topics/exceptions.html?highlight=closeSpider