Scraping ajax pages using python

First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.

Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

• open browser developer tools, network tab
• go to the target site
• click submit button and see what XHR request is going to the server
• simulate this XHR request in your spider

Also see:

• Can scrapy be used to scrape dynamic content from websites that are using AJAX?
• Pagination using scrapy

Hope that helps.