PostgreSQL is a free, powerful SQL database which is frequently used with Python How to connect to postgres sudo -i -u postgres run psql postgres@rag-laptop:~$ psql psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1)) Type “help” for help. list databases postgres=# l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ———–+———-+———-+————-+————-+———————– gis […]
Using Scrapy’s LinkExtractor method you can get the links from every page that you desire. What are Link Extractors? “A link extractor is an object that extracts links from responses.” Summary The above code gets all of the hrefs very quickly and give you the flexibility to omit or include very specific attirbutes Watch the video Extract Links | how to scrape website urls | Python + Scrapy […]
How can the start_urls for scrapy be imported from csv? Using a list comprehension and a csv file you can make Scrapy get specific URLs from a predefined list use the .strip() method to remove newline characters Here you can see the line.strip() is performing the removal: [line.strip() for line in file] Demonstration of how […]
Web Scraping pages with iframes in can be done with Scrapy if you use a separate URL to access the data inside the iframe. You need to identify the name of the page of the iframe and then append that to your base url to provide a 2nd URL for the Scrapy spider to visit. […]
If you are scraping a website with pop up messages asking you to agree to accept cookies. This can prevent your scraper from continuing to the pages you want to scrape. How do you get past these? Using Selenium you need to switch to the iframe (which you can identify using browser tools / inspect […]
If you have scraped data on more than one occasion and want to check if a value has changed in a column since the previous scrape you could use this: select col1, col2 from TABLENAME group by col1, col2 having count(col2) <2 This will compare and check if a value for col2 has changed since the previous scrape Let’s put some real names […]
Once you have some data you’ll need to find the relevant parts and use them for further analysis. Rather than use the proverbial Excel sheet you can write code to automate the task. Consider the following : The following code will match where the row contains “Djibouti” and return the value that is in the […]
& using Snippets This article will cover how to achieve the following, using Python : Reduce the need for print statements Reduce the need to type repetitive code Why not automate some of your coding to allow you to automate the ‘boring stuff’ – anything that you do regularly could be saved as a snippet […]
capture your start urls in your output with Scrapy response.meta Every web scraping project has aspects that are different or interesting and worth remembering for future use. This is a look at a recent real world project and looks saving more than one start url in the output. This assumes basic knowledge of web scraping, […]
Do you have a CSV file and a database to transfer the data into? The data needs to go into your database and you want to automate this? For instance : you have been web scraping and extracted publicly available data and now need to store it in a database table? This guide will show […]
