Xpath for hidden values

This article describes how to form a Scrapy xpath selector to pick out the hidden value that you may need to POST along with a username and password when scraping a site with a log in. These hidden values are dynamically created so you must send them with your form data in your POST request.

Step one

Identify the source in the browser:

This is the xpath selector format you will need to use:

response.xpath('//div[@class="aspNetHidden"]/input[@name="__VIEWSTATE"]/@value').get()

This should get what you need…in Scrapy SHELL !

But…Not when you run it in your Spider…

Instead, you need to use :

response.xpath('//*[@id="Form1"]/input[@name="__VIEWSTATE"]/@value').get()