& using Snippets This article will cover how to achieve the following, using Python : Reduce the need for print statements Reduce the need to type repetitive code Why not automate some of your coding to allow you to automate the ‘boring stuff’ – anything that you do regularly could be saved as a snippet […]
capture your start urls in your output with Scrapy response.meta Every web scraping project has aspects that are different or interesting and worth remembering for future use. This is a look at a recent real world project and looks saving more than one start url in the output. This assumes basic knowledge of web scraping, […]
Do you have a CSV file and a database to transfer the data into? The data needs to go into your database and you want to automate this? For instance : you have been web scraping and extracted publicly available data and now need to store it in a database table? This guide will show […]
Sooner or later you will need to resort to using regular expressions if you are scraping text. This article will show you some useful Python (version 3) examples which you can use over and over again.. We’ll be using “re.search” and “group” example 1 – parse a paragraph of text for the number of floors, […]
Back up your MySQL database (on a Raspberry Pi) Once in production you will need to ensure you have a copy of your data for many reasons which we can all identify with. So, before getting too far into a web scraping project using Scrapy with MySQL let’s spare a moment to get familiar with […]
How to make a dataframe from your SQL database and query the data This assumes you already have a sample database set up on your MySQL server and you have the username and password. In the example shown we are logging on to a Raspberry Pi running MariaDB and we are executing a query to […]
(and attachments) If you have seen any tutorials online, beware of many that use port 465, port 587 is the correct port. Here we show the code you can use to send email and an attachment with this simple Python code. It’s simple, as you don’t need to know MIME or “octet-stream” – you just […]
A common task is to track competitors prices and use that information as a guide to the prices you can charge, or if you are buying, you can spot when a product is at a new lowest price. The purpose of this article is to describe how to web scrape Amazon. Using Python, Scrapy, MySQL, […]
You may want to scrape Amazon for information about books about web scraping! We shorten what would have been a very very long selector, by using “contains” in our xpath : response.xpath(‘//*[contains(@class,”sg-col-20-of-24 s-result-item s-asin”)]’) The most important thing when starting to scrape is to establish what you want in your final output. Here are the […]