Remote backup of SimpleDB domains

So Amazon Web Services is pretty good but they're still lacking some of the small stuff. I'd like to see automated usage reports, warnings about suspicious traffic surges, and better backup options. I really don't understand why they don't assign a few dudes to build these things out because I know people have been begging for them for years.

Anyway, I coded a really simple Python script that will back up my domains. In no way is this a scalable solution - if you have a couple million items in your database, this is likely not a cost effective option. I've got less than a thousand entries today and I'm adding only a few hundred per week. It's hosted on WebFaction, which is a great hosting company - for only a few bucks per month you get access to your own very capable virtual server (no more mooching off friends or using inferior free services). It also uses boto, which is a very mature Python library for all things AWS. I've added it to my crontab to run a few days per week and it sends me an email when it's done. Of course, all usernames, passwords, email addresses removed. So here it is:

#! /usr/bin/env python2.7

from boto.sdb.connection import SDBConnection
from datetime import date
from smtplib import SMTP
from email.mime.text import MIMEText
import os
import settings

conn = SDBConnection('access_key','secret_key')

# backup account domain
account_filename = 'backup/account-' + date.today().isoformat() + '.xml'
f = open(account_filename, 'w')
domain = conn.get_domain("account")
domain.to_xml(f)
f.close()
account_size = os.stat(account_filename)

# backup timelog domain
timelog_filename = 'backup/timelog-' + date.today().isoformat() + '.xml'
f = open(timelog_filename, 'w')
domain = conn.get_domain("timelog")
domain.to_xml(f)
f.close()
timelog_size = os.stat(timelog_filename)

msg = MIMEText("wrote " + account_filename + " (" + str(account_size.st_size) + " bytes)\nwrote " + timelog_filename + " (" + str(timelog_size.st_size) + " bytpes)")
msg['Subject'] = 'Payroll Nanny backup report for ' + date.today().isoformat()
msg['From'] = 'my_app_address'
msg['To'] = 'my_email_address'

s = SMTP()
s.connect('smtp.webfaction.com')
s.login('my_webfaction_username', 'my_webfaction_password')
s.sendmail('my_app_address', 'my_email_address', msg.as_string())
s.quit()

Google App Engine is pretty great (if you go all the way)

So I'm starting to bang around another project. As I describe it on my Careers 2.0 profile:

Project is still in its infancy but it will have server (GAE-hosted Python program evaluating JSON feeds of stock data with Cloundant [CouchDB] backend) and mobile (Android app to visualize/manipulate the data from Cloundant) components. There really isn't a point behind this program and I don't necessarily condone algorithmic trading - it's just a fun hack.

Or so I thought. Doesn't look like I'll be using Google App Engine because I don't want to do it the "GAE way" (not that there's anything wrong with that). Deploying an app to GAE is super simple but I ran into problems when starting to incorporate 3rd-party libraries (couchdbkit and restkit to be exact). These libs are too small to be part of the standard runtime environment. It's interesting to note that even Django is implemented in a slightly different way on GAE.

I'll spare the details (though they are available in my stackoverflow question) but I was having importing couchdbkit into my script. It was choking on things likes sockets and resources, which made me think I was running afoul of the sandbox rules:

An App Engine application cannot:

• write to the filesystem. Applications must use the App Engine datastore for storing persistent data. Reading from the filesystem is allowed, and all application files uploaded with the application are available.
• open a socket or access another host directly. An application can use the App Engine URL fetch service to make HTTP and HTTPS requests to other hosts on ports 80 and 443, respectively.
• spawn a sub-process or thread. A web request to an application must be handled in a single process within a few seconds. Processes that take a very long time to respond are terminated to avoid overloading the web server.
• make other kinds of system calls.

I put a question on the Cloudant discussion board and got not 1 but 2 answers from Cloudant techs within an hour (these dudes are good - I don't even have a paying account and they are very responsive). Anyway, they tipped me off to this telling Quora answer. It looks like the Django/Python hosting space is about to blow up with Heroku-like solutions.

Both the Cloudant tech and Django responder (a Django co-inventor) had good things to say about WebFaction. It is not a free hosting service ($9.50 per month and goes down with longer term contracts). After a quick Q&A session with a sales rep, I took the plunge. 1 hour later, my program was working perfectly on my WebFaction account. Success! The only thing that makes me nervous is having to do any sys admin stuff at all - getting my script and installations to reference the right Python version (WebFaction comes everything from 2.4 [default] up through 3.2) is about all I care to handle.