Categories
Python Code Scrapy

Scraping a JSON response with Scrapy

reference:
► https://stackoverflow.com/questions/44939247/scrapy-extract-ldjson#48131898

We can’t get the html we need using a normal selector so having located the ‘script’ section in the browser (Chrome/Developer Tools) we can load into a JSON object to manipulate.

 json.loads(response.xpath('//script[@type="application/ld+json"]//text()') to get the data from a page containing javascript 

Using json.loads

We extracted the output which was not available from just using a normal css or xpath selector in Scrapy.

See the JSON response in scrapy video

Leave a Reply

Your email address will not be published. Required fields are marked *