Configure a Raspberry Pi for web scraping

Run Scrapy Spider

If you don’t already have a way to transfer the spider onto your Raspberry Pi you can simply use

git clone https://github.com/RGGH/scrapy6.git

Once all this is in place, you can run the spider with “nohup”

sudo nohup privatevpn

sudo python3 foodcom.py

Note: You also need to run the VPN with “nohup” as well, otherwise it will lose the connection when you exit from PuTTy / your Terminal session.

Check output

From your remote PC, eg a Windows PC, Mac, or Ubuntu, you can then copy the output file from the Pi, and open it, as you should have much more memory and of course a GUI!

To copy the results.csv from the Pi to my Ubuntu PC I used “rsync” but you could also use SCP.