Raspberry Pi torrent box with AWS

This post is more or less a continuation of my previous blog post, Converting your Raspberry Pi into a light weight torrent box.

If you follow this post you will be able to develop your own fully functional and easily extendable torrent box for your personal use. Please note that there may be web sites which already have services similar and better than what I have implemented. We are here to learn how to do that aren’t we? Everyday is a school day!!!

BONUS!!! If my server is running now following link would allow you to check what I am downloading now. 😉 Copy and paste the below in your browser.

(For mozilla)

view-source:http://54.191.41.142:8080/getStatus

(For other)

http://54.191.41.142:8080/getStatus

Right click and select view source to display the status in a better format.

So far we have set-up the deluge torrent client in our RPi (RaspberryPi) and we have confirmed that it is operational. This step is very important to have a working torrent-box.

First I will go through the design or the architecture of the solution that I am going to develop.

Architecture

Architecture1.vsdx.jpg

Above image is the most simple depiction of the architecture of the overall system. Following are the intended capabilities of the system.

  1. I need to be able to submit magnet links of any torrent remotely to RPi
  2. I need to be able to remove any torrent remotely from the RPi
  3. I need to be able to schedule the Start and Stop times of any torrent
  4. I need to be able to see the status of current torrent list

In order to perform this I have to do some web based software development which I don’t have any experience. After discussing with my friends who are Software Engineers I decided to use following languages/packages/frameworks for this application.

  1. NodeJS – For server side web application
  2. Python – For RPi side client application

There are many other ways to do this. I did not evaluate any of them. If you have better idea please let me know via a comment. I have put the references list at the end of the blog post.

Let’s go another step deeper into the architecture.

Server Side Architecture

Server Architecture

I think the diagram above is clear to you. The EC2 web application will be listen to HTTP requests coming from a TCP port of our choice. Specifically it will open up 7 URIs (Which are like APIs) to the user and to RPi.

  • addTorrent – Used to add a torrent magnetic link to the server (GET request)
  • remTorrent – Used to remove a torrent from server (GET request)
  • getTorrent – Can be used by RPi to get the current torrent list from server (GET request)
  • submitStatus – Can be used by RPi to submit the current torrent states (POST request)
  • getStatus – Can be used by user to read current torrent status (GET request)
  • fullUpdate – Can be used by user to change full torrent list at once (POST request)
  • fullUpdatePi – Same as fullUpdate but is intended to be used by RPi

Now Software Engineers will ask why I use GET requests to add torrents. But I decided to do this because then I can easily submit torrents using the address bar of any browser. I want to make this as simple as possible.

Raspberry Pi Side Architecture

Rpi architecture.jpg

Above diagram is self explanatory and is exactly what I implemented in the Python scripts. To do all this you must have a working Internet connection to you Raspberry Pi. (duh!!!)

Implementation

Setting up the server

I used an Amazon AWS server for my application. But you can use any server you wish. Therefore, I will be explaining in general terms in this section.

  • I used Ubuntu as the operating system for my server because it is more familiar to me.
  • I decided to open port 8080 for communication, you can use any unreserved TCP port for this.

Please refer to these links on how to setup AWS server and open the TCP port.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html

Now we need to install NodeJS in the server. I used following commands to install the latest NodeJS package in my server. I will not explain what these commands do. Please refer to the man pages if you want further information.

ubuntu@ip-172-31-36-88:~$ curl -sL https://deb.nodesource.com/setup_5.x | sudo bash
ubuntu@ip-172-31-36-88:~$ sudo apt-get install nodejs

Check whether you have current versions. These versions are correct as per 30th April 2016. The versions that you get might be different. But you get the idea.

ubuntu@ip-172-31-36-88:~$ node -v
v5.11.0

ubuntu@ip-172-31-36-88:~$ npm -v
3.8.6

Setting up the web application

nodejs-new-pantone-black

Now let’s run our NodeJS script.

Create a project folder

ubuntu@ip-172-31-36-88:~$ mkdir ~/torrentbox_webapp
ubuntu@ip-172-31-36-88:~$ cd ~/torrentbox_webapp/

Install a dependency  package.

ubuntu@ip-172-31-36-88:~/torrentbox_webapp$ npm install express –save

Save the following NodeJS script. I have commented the code. Please go through the code if you want to learn how it is implemented.

The script is also available in the GIT repository under inside the Server folder as torrent_webapp.js (https://github.com/pradeepa-s/torrentbox.git)

You have to do a small update to the torrent_webapp.js script to setup your own password and port.

// Please setup your own password
var PASSWORD = ‘password’;

// Please setup the port of your choice
var LISTEN_PORT = 8080;

Find the above code snippet in the torrent_webapp.js file and add your own password and the post of your choice.

Let’s say you saved the NodeJS script with the name torrent_webapp.js. Now start the application in a different terminal. This is important if you are using an SSH terminal to configure the server. If you don’t use following commands your web application will terminate as soon as your session is end.

ubuntu@ip-172-31-36-88:~$ screen

Now you are in a new terminal screen. Something like another instance

ubuntu@ip-172-31-36-88:~$ cd torrentbox_webapp/
ubuntu@ip-172-31-36-88:~/torrentbox_webapp$ node torrent_webapp.js

Now press CTR+A and CTR+D to detach from the screen.

Later, if you want to reattach you can type

ubuntu@ip-172-31-36-88:~$ screen -r

To exit the screen, first reattach and then type

ubuntu@ip-172-31-36-88:~$ exit

Believe it or not server side is done!!!

Implementing client application

python-logo-master-v3-tm

I have four very small Python modules implemented for Raspberry Pi side.

  1. os_support.py – This module is used to check whether HDD is mounted
  2. server_comm.py – This module is used to send requests to the server
  3. torrentbox_update.py – This module is used to communicate with torrent client
  4. torrentbox.py – This is the main script which has the main execution loop

Please note that I am an Electronics Engineer, so my strength is in C and C++ development. I’m not sure whether the software practices are correct. If you find anything out of order please let me know via a comment. 🙂 Everyday is a school day!!!

You have to edit the server_comm.py script to point it to your server.

# Set the server address here
SERVER_ADD = ‘http://54.191.41.142:8080’

You have to edit the torrentbox_update.py script to setup the torrent download folder

# Set the output directory
output_dir = “/mnt/external/rpi_movies”

You have to edit the os_support.py script to tell the mount point of your HDD

# Update the mount location of your HDD
MOUNT_LOCATION = ‘/mnt/external’

The scripts are well commented. Therefore, I will not explain them here. Please go through the code and try to understand. If you have any question or any suggestion please let me know.

Please note that the server/client communication is happening once every 5 minutes. Once you add a torrent or remove it it will take maximum of 5 minutes to process.

These scripts are also available in the GIT repository inside the Client/RPi folder (https://github.com/pradeepa-s/torrentbox.git)

Download the scripts and copy them to a location of your preference. Let’s assume that you have copied the scripts to ~/torrent_box location.

pi@raspberrypi:~ $ mkdir torrent_box

Make the torrentbox.py script executable

pi@raspberrypi:~ $ cd ~/torrent_box/
pi@raspberrypi:~/torrent_box $ sudo chmod 755 torrentbox.py

We are almost there!!!

Implementing auto mount

In order to make sure that our storage is always available, it is better to setup auto-mount capability. In my previous post, I have used an USB flash drive as my storage. But here I will use an external HDD. The commands used to configure auto-mount are more or less the same irrespective of the type of storage you use.

pi@raspberrypi:~ $ sudo mkdir -v -p /mnt/external
pi@raspberrypi:~ $ sudo chmod 755 /mnt/external
pi@raspberrypi:~ $ ls -al /dev/disk/by-uuid/

term1.jpg

Note down the UUID value of your external hard disk. In my case it is the file which is linked to sda1

Now we need to edit the fstab to specify the storage mount location and mount options.

pi@raspberrypi:~ $ sudo cp /etc/fstab /etc/fstab.backup

If your HDD is ntfs formatted you need to install ntfs-3g

pi@raspberrypi:~ $ sudo apt-get install ntfs-3g
pi@raspberrypi:~ $ sudo nano /etc/fstab

Add the following line to the end of that file. Use a tab in between two words.

UUID=<The UUID of the disk>     /mnt/external/    ntfs-3g   uid=pi,gid=pi,umask=007   0   0

term2.jpg

Now save the file and exit.

pi@raspberrypi:~ $ sudo reboot

After the RPi is up make sure that your HDD is mounted in /mnt/external/.

Implementing start up scripts

Now we need to make sure that deluge daemon (ie: deluged) is automatically started when we power on the RPi. For this we will create a start up script. (Here I am assuming that deluge is already set up in RPi. If not setup please refer to this post.)

Add the startup script in /etc/init.d folder. Make sure that the script is an executable.

pi@raspberrypi:~ $ sudo chmod 755 torrentbox.sh

This script is also available in GIT as Client/RPi/torrentbox.sh https://github.com/pradeepa-s/torrentbox.git.

You need to edit the torrentbox.sh script to tell your username and to point to your Python script location

# Set the username
$USER_NAME = pi

# Python script location
$TORRBOX_LOC = ~/torrent_box

When you try to copy the script to /etc/init.d you should use sudo.

Then run the following command to install the script. This will create required symlinks in /etc/rcX.d folders.

pi@raspberrypi:~ $ sudo update-rc.d torrentbox.sh defaults

Testing

Now everything is setup. Let’s reboot Raspberry Pi and begin testing.

pi@raspberrypi:~ $ sudo reboot

First we’ll test whether the HDD is properly mounted.

pi@raspberrypi:~ $ cd /mnt/external/
pi@raspberrypi:~ $ touch a
pi@raspberrypi:~ $ rm a

If there were no errors your HDD is mounted properly.

Let’s see whether deluged is working.

pi@raspberrypi:~ $ deluge-console

You’ll see the console now. If it says connected your deluged is running properly.

term3.jpg

Usage

Adding torrent

In order to add a torrent you need to find the magnet link of the torrent. Magnet link is something similar to below,

magnet:?xt=urn:btih:56518B2A821FA34CB22B87DA1717AD6ABDAABDED&dn=Raspberry%20Pi%20Magazines%20-%20June%2029%202015%20%28True%20PDF%29&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a80&tr=udp%3a%2f%2fopen.demonii.com%3a1337&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969&tr=udp%3a%2f%2fexodus.desync.com%3a6969

In order to add this torrent to the server you need to populate the request. First go to this website http://meyerweb.com/eric/tools/dencoder/ and encode the magnet link. This is one issue of submitting links as an URI. If you use the above link the output will look like below,

magnet%3A%3Fxt%3Durn%3Abtih%3A56518B2A821FA34CB22B87DA1717AD6ABDAABDED%26dn%3DRaspberry%2520Pi%2520Magazines%2520-%2520June%252029%25202015%2520%2528True%2520PDF%2529%26tr%3Dudp%253a%252f%252ftracker.openbittorrent.com%253a80%26tr%3Dudp%253a%252f%252fopen.demonii.com%253a1337%26tr%3Dudp%253a%252f%252ftracker.coppersurfer.tk%253a6969%26tr%3Dudp%253a%252f%252fexodus.desync.com%253a6969

The request should have following format

http://<IP : PORT>/addTorrent/<Password>/<Magnet link>/<Start time>/<Stop time>

  • IP – Your ip address
  • PORT – The port your web application listens to
  • Password – The password you have setup in the web application
  • Magnetic Link – URI encoded magnetic link
  • Start Time – The time that you need to start downloading (24 hour format)
  • Stop Time – The time that you need to stop downloading (24 hour format)

Example:

Following link will add a torrent and start downloading the torrent at 12PM and finish at 8AM

http://10.1.1.1:8080/addTorrent/password/example link/0/800

Get torrent list

http://<IP : PORT>/getTorrent

This will give you a JSON array. You can see what are the current torrents in the server.

{"torrents":[{"Action":"Add","Values":"magnet:?xt=urn:btih:56518B2A821FA34CB22B87DA1717AD6ABDAABDED&dn=Raspberry%20Pi%20Magazines%20-%20June%2029%202015%20%28True%20PDF%29&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a80&tr=udp%3a%2f%2fopen.demonii.com%3a1337&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969&tr=udp%3a%2f%2fexodus.desync.com%3a6969","BTIH":"56518B2A821FA34CB22B87DA1717AD6ABDAABDED","Start":"1315","Stop":"1400","State":"NA"}]}

Removing a torrent

It’s almost same as above. But you need to give the index of the torrent to remove it. Get the torrent list and find the index of the torrent. First torrent in the JSON has the index 1, and so on. Be careful when you find the index. This is something I should improve.

http://<IP : PORT>/remTorrent/<Password>/<Index>

 Get torrent status

http://<IP : PORT>/getStatus

or

view-source:http://<IP:PORT>/getStatus

This will provide a string output which tells the current status of all the torrents in the RPi. You will have to view the source of the web site (Usually rightcick->view source) to get a good representation of the status.

Name: Raspberry Pi Magazines – June 29 2015 (True PDF)
ID: 56518b2a821fa34cb22b87da1717ad6abdaabded
State: Paused
Size: 0.0 KiB/0.0 KiB Ratio: -1.000
Seed time: 0 days 00:00:00 Active: 0 days 00:00:02
Tracker status: openbittorrent.com: Error:
Progress: 0.00% [~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]

You can use the following link to check my current download status. 😉

(For mozilla)

view-source:http://54.191.41.142:8080/getStatus

(For other)

http://54.191.41.142:8080/getStatus

Improvement plans

  1. Make it easy to remove torrents
  2. Provide a web form to submit torrents
  3. Develop an android app to submit torrents
  4. Provide better torrent status display web site

References

  1. http://www.w3schools.com/jsref/default.asp
  2. http://www.tutorialspoint.com/nodejs/
  3. http://www.tutorialspoint.com/python
  4. http://docs.python-requests.org/en/master/
  5. http://www.tutorialspoint.com/unix/
  6. http://expressjs.com/
  7. http://blog.scphillips.com/posts/2013/07/getting-a-python-script-to-run-in-the-background-as-a-service-on-boot/
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s