This article describes how to form a Scrapy xpath selector to pick out the hidden value that you may need to POST along with a username and password when scraping a site with a log in. These hidden values are dynamically created so you must send them with your form data in your POST request.
Identify the source in the browser:
This is the xpath selector format you will need to use:
This should get what you need…in Scrapy SHELL !
But…Not when you run it in your Spider…
Instead, you need to use :