Do you need to scrape a page that is dynamically loading content as “infinite scroll” ? Using self.nxp +=1 the value passed to “pn=” in the URL gets incremented “pn=” is the query – in your spider it may be different, you can always use urllib.parse to split up the URL into it’s parts. Test […]