Configure a Raspberry Pi for web scraping

Commands used to install the VPN:

Setting up the VPN (on the Pi Zero)

Important : Check disk free using “df -h

pi@pi_zero:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 4.7G 9.3G 34% /
devtmpfs 213M 0 213M 0% /dev
tmpfs 217M 0 217M 0% /dev/shm
tmpfs 217M 6.2M 211M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 217M 0 217M 0% /sys/fs/cgroup
/dev/loop0 80M 80M 0 100% /snap/core/9069
/dev/mmcblk0p1 43M 23M 20M 53% /boot
tmpfs 44M 8.0K 44M 1% /run/user/1000

Next, get the VPN specific files (the ones in bold are for “Private VPN” yours may be different).

sudo apt-get update

sudo apt-get http://privatevpn.com/client/ca.crt

sudo wget https://privatevpn.com/client/install.sh

sudo ./install.sh

sudo chmod a+x install.sh

sudo apt-get install openvpn curl

pi@pi_zero:~ $ sudo ./install.sh
PrivateVPN Linux OpenVPN Installer v0.3

  • Checking for OpenVPN – OK
  • Enter login details for PrivateVPN
  • [username]: *****
  • [password]: ********
  • Installing conf to default location (/etc/openvpn), write c to edit installpath.
  • Continue [yes/no/c] yes
    Fetching the server list from https://privatevpn.com/serverlist/
  • Installation Complete, run “sudo privatvpn” to connect to PrivateVPN service.

sudo micro /etc/privatevpn.conf # add server details

sudo nohup privatvpn # nohup means it will keep running if you quit PuTTy or Terminal from where you launch it.

To check it’s connecting to the VPN server use :

curl ifconfig.me and check if the ip address is the same as the server you saw in the Private VPN published list.

Python Code

Previous article

Scraping “LOAD MORE”