I maintain several personal websites, and felt they should be backed up in case my Xen host has issues.

Step 1

Create an Amazon S3 account at http://aws.amazon.com/s3/.  Once your account is created, you will need to create ‘credentials’, which will allow us to authenticate with S3.  You can access this by going to the “Amazon -> Account -> AWS Identity and Access Management” then clicking ‘Security Credentials’ on the left, and then creating a ‘Access Key’.  These keys are composed of 2 parts: a public portion, called the ‘Access Key ID’, and a private portion (never to be shared) called the ‘Secret Access Key’.

Step 2

We need to install a program called ‘s3cmd’.  This will allow us to interface with Amazon S3 via the command line.  On Ubuntu:

sudo apt-get install s3cmd

Step 3

Now we need to setup s3cmd to save settings about our setup.  Make sure you have the Access Key ID and the Secret Key.  Run the following command to get started:

s3cmd --configure

From here you will get an interactive prompt:

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: 231231232
Secret Key: 213123123

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: ubuntu
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: yes

New settings:
  Access Key: 231231232
  Secret Key: 213123123
  Encryption password: ubuntu
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]

I chose to pick “Use HTTPS protocol”, which will upload it via a secure method. This is a good idea, although will slightly impact performance and may use slightly more traffic. In addition, s3cmd also will encrypt the files using gpg, which means that if someone broke into your s3 account, they would still need that pass phrase to decrypt your data.

Step 4
We can now test s3cmd and try to upload a file. You will need to create a ‘bucket’, which is where our files for this project are stored. You can have many buckets, so if you want to separate your projects you could create additional ones for each one. When we make a bucket name, they are globally visible in S3, so you will want to pick something not likely to be taken:

s3cmd mb s3://sharms.org-wordpress-blog

If that command runs successfully, we now have a new bucket called ‘sharms.org-wordpress-blog’. If not, pick a different name and try again. Now we can test uploading a file:

s3cmd put /home/sharms/testfile.txt s3://sharms.org-wordpress-blog

# Verify its where we think it is
s3cmd ls s3://sharms.org-wordpress-blog

Step 5
Using bash, we can automate this, and backup all of our files, daily, weekly, monthly etc. Here is an example, which I put at ‘/usr/local/bin/backup_blog_to_s3.sh’:

bucket="s3://sharms.org-wordpress-blog"

logger -t backup_blog_to_s3.sh "Backing up sharms.org blog to S3"
cd /var/www
tar -cf sharms.org.tar blog
bzip2 -9 sharms.org.tar
s3cmd put sharms.org.tar.bz2 ${bucket}
rm /var/www/sharms.org.tar.bz2

logger -t backup_blog_to_s3.sh "Backing up MySQL database to S3"
mysqldump sharms-wordpress -u databaseuser -p databasepassword -a -r sharms-wordpress.sql
bzip2 -9 sharms-wordpress.sql
s3cmd put sharms-wordpress.sql.bz2 ${bucket}
rm sharms-wordpress.sql.bz2

You can see from the example that we backup all of the files in the ‘blog’ directory, and export all of our data out of a MySQL database. You can even change the file names so they have the date when they were backed up:

tar -cf sharms.org-wordpress-$(date +%d%m%y) blog

Running Automatically
If we wanted to backup the system every day, this is very easy:

sudo cp /usr/local/bin/backup_blog_to_s3.sh /etc/cron.daily
sudo chmod 755 /etc/cron.daily

Security Notes
When considering this setup, you are most vulnerable to someone obtaining access to your server, and getting your Amazon keys. You can always revoke them from the Amazon Webservices Control Panel, but you don’t want an attacker using your S3 account for nefarious means. Beyond the scope of this document, you could setup a user called ‘backups’, and make the file ‘~backups/.s3cmd’ with the permissions ’600′, to stop other users from looking at it’s contents.

IOCTL is a function call that allows you to interface with kernel drivers, allowing you to adjust settings or set parameters from code without compiling a new module.

From a programming perspective, having the linux kernel source is a prerequisite. In this example, I cloned the main kernel:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux

Once I had the source, I specifically wanted to see exactly how I could interface with the driver called usblp. I was using a USB to parallel port converter, and wanted to see if there was any way to force it to operate differently as we needed a non-printer device to work with it.

After opening the kernel source, I found the driver file itself which was under /drivers/usb/class/usblp.c. In this file I found a section of information about which IOCTLs it supported, and put them in a header file for my program called usblp-hack.h:

#ifndef USBLP_HACK_H
#define USBLP_HACK_H

#include <linux/ioctl.h>

/* ioctls: */
#define IOCNR_GET_DEVICE_ID     1
#define IOCNR_GET_PROTOCOLS     2
#define IOCNR_SET_PROTOCOL      3
#define IOCNR_HP_SET_CHANNEL        4
#define IOCNR_GET_BUS_ADDRESS       5
#define IOCNR_GET_VID_PID       6
#define IOCNR_SOFT_RESET        7
/* Get device_id string: */
#define LPIOC_GET_DEVICE_ID(len) _IOC(_IOC_READ, 'P', IOCNR_GET_DEVICE_ID, len)
/* The following ioctls were added for http://hpoj.sourceforge.net: */
/* Get two-int array:
 * [0]=current protocol (1=7/1/1, 2=7/1/2, 3=7/1/3),
 * [1]=supported protocol mask (mask&(1<<n)!=0 means 7/1/n supported): */
#define LPIOC_GET_PROTOCOLS(len) _IOC(_IOC_READ, 'P', IOCNR_GET_PROTOCOLS, len)
/* Set protocol (arg: 1=7/1/1, 2=7/1/2, 3=7/1/3): */
#define LPIOC_SET_PROTOCOL _IOC(_IOC_WRITE, 'P', IOCNR_SET_PROTOCOL, 0)
/* Set channel number (HP Vendor-specific command): */
#define LPIOC_HP_SET_CHANNEL _IOC(_IOC_WRITE, 'P', IOCNR_HP_SET_CHANNEL, 0)
/* Get two-int array: [0]=bus number, [1]=device address: */
#define LPIOC_GET_BUS_ADDRESS(len) _IOC(_IOC_READ, 'P', IOCNR_GET_BUS_ADDRESS, len)
/* Get two-int array: [0]=vendor ID, [1]=product ID: */
#define LPIOC_GET_VID_PID(len) _IOC(_IOC_READ, 'P', IOCNR_GET_VID_PID, len)
/* Perform class specific soft reset */
#define LPIOC_SOFT_RESET _IOC(_IOC_NONE, 'P', IOCNR_SOFT_RESET, 0)
#endif

There are plenty of hints on usage here, and I was able to grab a little more info after searching for some of those defines in google code search. Using these IOCTLs I wanted to know exactly which modes the USB to parallel converter supported, and try to set it in mode 3, which I hoped would give me more options for talking to the device.

Here is my usblp-hack.c which was able to probe and update these settings, although in my case mode 3 didn’t work, but shows an example of exactly how to do it:

#include "usblp-hack.h"
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>		/* open */
#include <unistd.h>		/* exit */
#include <sys/ioctl.h>		/* ioctl */

main()
{
	int fd;
	int twoints[2];

	fd = open("/dev/usblp0", O_RDONLY | O_NONBLOCK);
	if (fd < 0) {
		printf("Cannot open device.\n");
		exit(-1);
	}

	if(ioctl(fd, LPIOC_GET_PROTOCOLS(sizeof(int[2])), &twoints) >= 0)
		printf("Great success: %d / %d\n", twoints[0], twoints[1]);
	else {
		printf("Fail!\n");
		exit(-1);
	}

	if(ioctl(fd, LPIOC_SET_PROTOCOL, 2) >= 0)
		printf("set protocol to version 2\n");
	else {
		printf("Fail!\n");
		exit(-1);
	}

	if(ioctl(fd, LPIOC_GET_PROTOCOLS(sizeof(int[2])), &twoints) >= 0)
		printf("Great success: %d / %d\n", twoints[0], twoints[1]);
	else {
		printf("Fail!\n");
		exit(-1);
	}

	if(ioctl(fd, LPIOC_SOFT_RESET) >= 0)
	{
		printf("Success reset device\n");
	} else {
		printf("Could not reset device\n");
		exit(-1);
	}

	close(fd);
	exit(0);
}

You can compile the above example just using gcc:

gcc -o usblp-hack usblp-hack.c

You can see in the example above I was able to probe the device to see what mode it was currently in and supports (LPIOC_GET_PROTOCOLS), attempt to set the device mode (LPIOC_SET_PROTOCOL), and soft reset the device (LPIOC_SOFT_RESET).

So in the end, nothing here too complex or mind blowing, but if you are just getting your feet wet it might take you a minute to dig this sort of information up.

I was reading a post about PyGoocanvas, and decided to take a look at some code and see what fun could be had. I haven’t done much yet, but this screenshot of it is pretty entertaining in it’s own right: