reference:
► https://stackoverflow.com/questions/44939247/scrapy-extract-ldjson#48131898
We can’t get the html we need using a normal selector so having located the ‘script’ section in the browser (Chrome/Developer Tools) we can load into a JSON object to manipulate.
json.loads(response.xpath('//script[@type="application/ld+json"]//text()') to get the data from a page containing javascript
Using json.loads
We extracted the output which was not available from just using a normal css or xpath selector in Scrapy.