Selenium

April 9, 2021 / Last updated : February 13, 2023 admin Python Code

How to scrape iframes

If you are scraping a website with pop up messages asking you to agree to accept cookies. This can prevent your scraper from continuing to the pages you want to scrape. How do you get past these? Using Selenium you need to switch to the iframe (which you can identify using browser tools / inspect […]

October 7, 2020 / Last updated : October 7, 2020 admin Python Code

Combine Scrapy with Selenium

A major disadvantage of Scrapy is that it can not handle dynamic websites (eg. ones that use JavaScript). If you need to get past a login that is proving impossible to get past, usually if the form data keeps changing, then you can use Selenium to get past the login screen and then pass the […]