Categories
Python Code

How to web scrape iframes with scrapy

Web Scraping pages with iframes in can be done with Scrapy if you use a separate URL to access the data inside the iframe.

You need to identify the name of the page of the iframe and then append that to your base url to provide a 2nd URL for the Scrapy spider to visit.

Watch the video here on YouTube

In the video you see how to extract the elements from within the iframe using Scrapy

View/fork/copy the code from our GitHub repo: https://github.com/RGGH/iframes

See also our article on scraping iframes using Selenium :

https://redandgreen.co.uk/web-scrape-iframes-how-to-accept-cookies-privacy-pop-up-when-web-scraping-with-python-and-selenium/

Let us know if you would like any more information or discuss a project

We can provide a quote and initial analysis within 1 working day

For more details on iframes see : https://www.w3schools.com/html/html_iframe.asp