Distribution of Big Files

I had a problem a while ago, with the use of VMs in classrooms and labs.  Let me tell you, no matter where you are, if you have 25+ computers that need to have a local VM (because you don’t have a VDI in place, let alone budgeted), you’re going to experience a bottleneck, and your network admin is going to come running down the hall with screams of distress (yes, I’m talking to you, Josh).

There are very few things that can help in this case.  SCCM can do the job, but it’s not very expedient or elegant.  Pure Windows File Transfer is almost guaranteed to cause issues for anyone trying to connect to the file server.  PDQ Deploy is good, because of queuing, but it’s not really meant for files over 5gb in size.

However, there is one protocol that we’ve learned over the years is perfect for large file transfers to multiple computers: BitTorrent.  Yes, that strange, copyright enticing protocol that has caused ripples across the tech industry and law makers alike.  Please remember, it’s merely a tool, and it matters how you use it.

Let’s continue with the example of Virtual Machines, as those are roughly 10-20gb in size.

First, create your torrent file, preferably on the file server, so it can act as the primary seed.  If you add a tracker, use the file server, since you want this only in your local intranet.

Second, I suggest setting up a Uniform Server or XAMPP for serving your new .torrent file, and BTStorm (the MySQL branch) to act as your tracker.  You can use a file share, but this takes out any authentication issues.

Next, via scripting on your client machines, use Aria2 to grab the .torrent file from your server, and start pulling the files.

Selec All Code:
ShellExecuteWait("C:\aria2\aria2c.exe", "http://webserver/file.torrent --file-allocation=none --bt-enable-lpd=true --bt-tracker=<trackerURL> --on-bt-download-complete=ariapskill.cmd --allow-overwrite=true --dir=C:\ -q")

This simple line of AutoIt script will execute the Aria2 CLI, hidden from the users’ view, and begin downloading the torrent data.  It also uses local peer discovery, so when you’re doing massive numbers at one time, things will go much faster, and no longer bog down your network, or bottleneck at your server.

When the file transfer is complete, instead of running until the computer is rebooted, there’s a configuration line that executes a command to kill the aria2c.exe process (–on-bt-download-complete=ariapskill.cmd).  I used PSKill, from SysInternals, which might be the best way to go about it.

All in all, the use of Aria2 is an easy way to distribute large amounts of data in a short amount of time, and keep your servers from having I/O issues.

Posted in Workings and tagged , , .