Categories
Python Code

Xpath for hidden values

This article describes how to form a Scrapy xpath selector to pick out the hidden value that you may need to POST along with a username and password when scraping a site with a log in. These hidden values are dynamically created so you must send them with your form data in your POST request.

Step one

Identify the source in the browser:

xpath-for-hidden-values
Ok, so we want to pass “__VIEWSTATE” as a key : value pair in the POST request

This is the xpath selector format you will need to use:

response.xpath('//div[@class="aspNetHidden"]/input[@name="__VIEWSTATE"]/@value').get()

This should get what you need…in Scrapy SHELL !

But…Not when you run it in your Spider…

Instead, you need to use :

response.xpath('//*[@id="Form1"]/input[@name="__VIEWSTATE"]/@value').get()

Scrapy-Hidden-Values-XPATH-Selector

Leave a Reply

Your email address will not be published. Required fields are marked *