A common task is to track competitors prices and use that information as a guide to the prices you can charge, or if you are buying, you can spot when a product is at a new lowest price. The purpose of this article is to describe how to web scrape Amazon.
Using Python, Scrapy, MySQL, and Matplotlib you can extract large amounts of data, query it, and produce meaningful visualizations.
In the example featured, we wanted to identify which Amazon books related to “web scraping” had been reduced in price over the time we had been running the spider.
If you want to run your spider daily then see the video for instructions on how to schedule a spider in CRON on a Linux server.
Procedure used for price tracking
query = '''select amzbooks2.* from (select amzbooks2.*, lag(price) over (partition by title order by posted) as prev_price from amzbooks2) amzbooks2 where prev_price <> price'''
Visualize the stored data using Python and Matplotlib
The most important thing when starting to scrape is to establish what you want in your final output.
Here are the data points we want to extract :
‘title’
‘author’
‘star_rating’
‘book_format’
‘price’
Now we can write our parse method, and once done, we can finally add on the “next page” code.
The Amazon pages have white space around the Author name(s) so you this will be an example of when to use ‘normalize-space’.
We also had to make sure we weren’t splitting the partially parsed response too soon, and removing the 2nd Author, (if there was one).
Some of the results are perhaps not what you want, but this is due to Amazon returning products which it thinks are in some way related to your search criteria!
By using pipelines in Scrapy, along with the process_item method we were able to filter much of what was irrelevant. The great thing about web scraping to an SQL database is the flexibility it offers once you have the data. SQL, Pandas, Matplotlib and Python are a powerful combination…
The challenge was to scrape a site where the class names for each element were out of order / randomised. So the only way to get the data in the correct sequence was to sort through the CSS styles by left, top, and match to the class names in the divs…
The px varied slightly: see 710 and 711 above
The names were meaningless and there was no way of establishing an order, a quick check in developer tools shows the highlighter rectangle jumps all over the page in no particular order when traversing the source code.
So we began the code by identifying the CSS and then parsing it:
The idea is to extract the CSS style data, parse it for left, and top px, and sort through those to match the out of sequence div/classnames in the body of the page.
There was NO possibility of sequentially looping through the divs to extract ordered data.
After watching the intro to the challenge set on YouTube, and a handy hint from CMK we got to work.
From this article you will learn as much about coding with Python as you will about web scraping in specific. Note I used requests_html, as it provided me with the option to use XPATH.
BeautifulSoup could also have been used.
Identifying columns and rows based on “left px” and “top px”
Python methods used:
round
x = round(1466,-2)
print(x) # 1500
x = round(1526,-2)
print(x) # 1500
x = round(1526,-1)
print(x) # 1530
I needed to use round as the only way to identify a “Column” of text was to cross reference the <style> “left px” and “top px” with the class names used inside the divs. Round was required as there was on occasion a 2 or 3 px variation in the “left” value.
itemgetter
Item getterfrom operator import itemgetter
ls.sort(key=itemgetter('left','top'))
I had to sort the values from the parsed css style in order of “left” to identify 3 columns of data, and then “top” to sort the contents of the 3 columns A, B, C.
zip
zipped = zip(ls_desc,ls_sellp,ls_suggp)
Zipping the 3 lists – read on to find out the issue with doing this…
rows = list(zipped)
So to get the data from the 3 columns, A (ls_desc), B (ls_sellp), and C (ls_suggp) I used ZIP, but…….the were/are 2 values missing in column C!!
A had 77 values,
B had 77 values
C had 75 !
Not only was there no text in 2 of the blanks in column C, there was also NO text or even any CSS.
We only identified this as an issue after running the code – visually the page looked consistent, alas the last part of column “C” becomes out of sequence with the data in colmumn A and B which are both correct.
Solution?
Go back and check if column “C” has a value at the same top px value as Column “B”. If no value then insert an “x” or spacer into Column C at that top px value.
This will need to be rewritten using dictionaries, and create one dictionary per ROW rather than my initial idea of 1 list per column and zipping them!
Zipping the 3 lists nearly works..but 2 missing values in “Suggested Price” means that the data in Column C become out of synch.
special thanks to “code monkey king” for the idea/challenge!
Rewrite the section for “Column B” to check for presence of text in column “C” on the same row…
1 missing value halfway down column “C” means more error checking is required! – If you just want the “Selling Price” and “Description” then this is code is 100% successful! 👍
See the solution, and error on the YouTube Video
Conclusion:
For more robust web scraping where css elements may be missing use dictionaries/enumerate each row and check. It’s the old case of “you don’t know what you don’t know”
If you can ensure each list has the same number of items, then ZIP is ok to use.
