Data Analysis With Pandas
If you want to learn about Data Analysis with Pandas and Python and you’re not familiar with Kaggle, check it out! Time to read article : 5 mins TLDR; We show how to use idxmax and apply with Pandas Introduction Here we will look at some functions in Pandas which will help with ‘EDA’ – […]
EBAY API – Python Code
If you have been looking for EBAY API – Python code then this article can help. Rather than use web scraping techniques you can access live EBAY listings via the API with an SDK for Python. Time to read this article about EBAY API – Python Code : 5 mins TLDR; Watch the ebay api […]
PostgreSQL
PostgreSQL is a free, powerful SQL database which is frequently used with Python How to connect to postgres sudo -i -u postgres run psql postgres@rag-laptop:~$ psql psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1)) Type “help” for help. list databases postgres=# l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ———–+———-+———-+————-+————-+———————– gis […]
Extract links with Scrapy
Using Scrapy’s LinkExtractor method you can get the links from every page that you desire. What are Link Extractors? “A link extractor is an object that extracts links from responses.” Summary The above code gets all of the hrefs very quickly and give you the flexibility to omit or include very specific attirbutes Watch the video Extract Links | how to scrape website urls | Python + Scrapy […]
Read Scrapy ‘start_urls’ from csv file
How can the start_urls for scrapy be imported from csv? Using a list comprehension and a csv file you can make Scrapy get specific URLs from a predefined list use the .strip() method to remove newline characters Here you can see the line.strip() is performing the removal: [line.strip() for line in file] Demonstration of how […]
How to web scrape iframes with scrapy
Web Scraping pages with iframes in can be done with Scrapy if you use a separate URL to access the data inside the iframe. You need to identify the name of the page of the iframe and then append that to your base url to provide a 2nd URL for the Scrapy spider to visit. […]
How to scrape iframes
If you are scraping a website with pop up messages asking you to agree to accept cookies. This can prevent your scraper from continuing to the pages you want to scrape. How do you get past these? Using Selenium you need to switch to the iframe (which you can identify using browser tools / inspect […]
Comparing values in SQL against previously scraped data
If you have scraped data on more than one occasion and want to check if a value has changed in a column since the previous scrape you could use this: select col1, col2 from TABLENAME group by col1, col2 having count(col2) <2 This will compare and check if a value for col2 has changed since the previous scrape Let’s put some real names […]
Parsing Scraped Data with Pandas
Once you have some data you’ll need to find the relevant parts and use them for further analysis. Rather than use the proverbial Excel sheet you can write code to automate the task. Consider the following : The following code will match where the row contains “Djibouti” and return the value that is in the […]