We’ve not included Javascript or Selenium and how others think about it may vary but time is finite, so concentrate on what gets the biggest benefit. We could have put Python at the centre but we’ve assumed that’s a given. How do you visualize your own process of web scraping?

If you use Scrapy, and CrawlerProcess then Python may not search the path that is required for Scrapy to find its “items.py” file. How do you fix this ‘ModuleNotFoundError‘ ? Read on… As we can see, Scrapy framework has the items.py file in a directory above the spiders directoy