"Multiple dex files" error with Eclipse + ADT

I haven't done much Android development in the last year but this week wanted to update my app a bit.  So I installed Eclipse Juno (4.2), grabbed the latest Android Developer Tools (r21), and the latest AWS SDK for Android (1.4.4) since my app persists data on AWS.

These are all pretty big jumps from what I used to have - Eclipse Indigo, probably a single digit ADT, and AWS SDK 0.2.X IIRC.  I've only been using my System76 laptop for grad school work of late, not "real" coding obviously.  Almost everything worked fine.  Thanks go out to these teams for honoring backward compatibility!  The one problem I had was "Unable to execute dex: Multiple dex files define X" (X being something in the AWS package).

So off to StackOverflow.  I tried all the suggestions:

• Open/close Eclipse a few times, doing project cleans
• Reinstalling ADT
• Deleting my /bin and /gen directories, then cleaning
• Made sure my build path was legit - several people traced it back to having /bin in the build path

The AWS JARs I imported were core, debug, and sdb (my app uses SimpleDB - don't ask).  Since my testing only consisted of moving the .apk to dropbox and making sure it worked on my Galaxy S3, I didn't need the debug JAR.  Once I removed it, everything worked okay.

Kinda perturbed this is still a problem - many SO posts and blogs I've seen are by people who appear to know what they're talking about so I don't think it's always a silly oversight by junior developers.  I'm thinking it's a dependency problem - maybe AWS SDK 1.4.4 wasn't developed with ADT r21?  If that was the case, Eclipse and SDK providers make it really hard to grab older versions like you would with a POM file.  If I needed to debug in this instance, I'd be in real trouble.

AWS Free Usage Tier experience

I've had an app in the wild for about 1.5 years.  I'm in pure maintenance mode so other than replying to user emails, I check out error reports, reviews, and download stats every couple months.  Occasionally, I'll peak into the persistence to see what the raw data is looking like (shout out to sdbtool!).  But it's been some time since I looked at how much resources I'm consuming on AWS.  I built this app thinking I would always fall in the Free Tier pricing level.

My AWS usage report tells me 2 things:

• I'm a very small fry.
• The AWS Free Usage Tier is pretty generous.

Before getting to the AWS stats, here's my use case.  I probably have only a couple hundred active users today.  I'm using AWS for SimpleDB persistence (no graph processing here!) so all the compute time is dedicated to CRUD with the database.  SimpleDB rows are attribute-value pairs (Strings) and I average around half-a-KB of data per row.  I have about 25k rows of data in my SimpleDB instance.

My SimpleDB usage for July 2012:

• 0.052 Compute-Hours (25 free)
• 0.000529 GB-Month of storage (1 GB-Month free)

I was confused by the GB-Month number.  I thought that meant how much data I'm storing in my SimpleDB instance but it must refer to the amount of new data.  So that's half a MB of new data coming in which translates to around 1,600 new items ('rows' in SimpleDB parlance) in my domains (SimpleDB's word for 'tables') in July.  In fact, the total amount of disk space is represented by the TimedStorage-ByteHrs field of the detailed usage report.

My AWS usage for July 2012:

• 0.001 GB data transfer in (1 GB free)
• 0.002 GB data transfer out (1 GB free)

Since I serve up much more data than I ingest, the data out number is my only concern.  That's 2 MB of data leaving AWS and my app is fielding about 18K Select requests for items (SimpleDB's term for 'rows') each month.  Note the request might not be for an entire item - it could be for 1 or more attributes ('columns').

Add these numbers up and I obviously have room to scale!  In fact, I could get almost a 500x increase in activity and still fall within the free usage tier.  Since I only coded this app for fun and experience, I probably won't grow but it's fun to know where I'm at.  It's really exciting to consider all the backend options out there for developers (GAE is similarly generous).

Remote backup of SimpleDB domains

So Amazon Web Services is pretty good but they're still lacking some of the small stuff. I'd like to see automated usage reports, warnings about suspicious traffic surges, and better backup options. I really don't understand why they don't assign a few dudes to build these things out because I know people have been begging for them for years.

Anyway, I coded a really simple Python script that will back up my domains. In no way is this a scalable solution - if you have a couple million items in your database, this is likely not a cost effective option. I've got less than a thousand entries today and I'm adding only a few hundred per week. It's hosted on WebFaction, which is a great hosting company - for only a few bucks per month you get access to your own very capable virtual server (no more mooching off friends or using inferior free services). It also uses boto, which is a very mature Python library for all things AWS. I've added it to my crontab to run a few days per week and it sends me an email when it's done. Of course, all usernames, passwords, email addresses removed. So here it is:

#! /usr/bin/env python2.7

from boto.sdb.connection import SDBConnection
from datetime import date
from smtplib import SMTP
from email.mime.text import MIMEText
import os
import settings

conn = SDBConnection('access_key','secret_key')

# backup account domain
account_filename = 'backup/account-' + date.today().isoformat() + '.xml'
f = open(account_filename, 'w')
domain = conn.get_domain("account")
domain.to_xml(f)
f.close()
account_size = os.stat(account_filename)

# backup timelog domain
timelog_filename = 'backup/timelog-' + date.today().isoformat() + '.xml'
f = open(timelog_filename, 'w')
domain = conn.get_domain("timelog")
domain.to_xml(f)
f.close()
timelog_size = os.stat(timelog_filename)

msg = MIMEText("wrote " + account_filename + " (" + str(account_size.st_size) + " bytes)\nwrote " + timelog_filename + " (" + str(timelog_size.st_size) + " bytpes)")
msg['Subject'] = 'Payroll Nanny backup report for ' + date.today().isoformat()
msg['From'] = 'my_app_address'
msg['To'] = 'my_email_address'

s = SMTP()
s.connect('smtp.webfaction.com')
s.login('my_webfaction_username', 'my_webfaction_password')
s.sendmail('my_app_address', 'my_email_address', msg.as_string())
s.quit()

Android Market observations

It's been a few weeks since my last post - I've been super busy launching my first app on Android Market. The prototype was built a couple months ago and it's mature enough to be released into the wild. Homepage here and Android Market entry here.

Anyway, the work necessary to get my app on Android Market was suprisingly minimal. Here are some general observations:

• Google does a great job tying into Eclipse: You can code with any IDE you want but for my money, I have to use Eclipse because it works so seemlessly with Google stuff. Even signing the app is simple.



• Be very distrustful about network connections: I've been using my app for a couple months and never got presented with a "Force Close" (a crash). However, some of the errors being logged are the result of devices dropping network connection and my code was not catching these conditions sufficiently. I'm pushing an update this week which catches these exceptions, presents an alert dialog that states the network isn't working, and takes the user to a safe activity.



• Be careful about security: There as a good article this week warning users of Amazon Web Services (which I am one) to get better with protecting credentials. I spent a good amount of time developing authentication middleware and don't use the VMs this article focuses on but it scared me enough to take a closer look at how I use IAM. I'm rolling credentials more, beefing up my personal password security, and paying more attention to my server logs.



• Testing is hard: I got caught not checking for some basic weird user behavior, like putting spaces in usernames then putting that string in a GET request. Now that I have a bunch of users, I'm fixing these shortcomings.



• Android Market exhibits some weird behavior for publishers: When trying to push my update, I was presented with "An unexpected error occurred. Please try again later." after clicking the "Upload" button. It seems like this is a catch-all error message that the Market displays whenever there's a problem. Back in April 2011, some servers were down and this was getting displayed all the time. I finally resolved it by logging out of Android Market and back in.

NoSQL databases - now with SQL!

So I'm writing an app and wanted to take the opportunity to see what the NoSQL buzz was all about. By no means would I need to scale to the level most programmers expect when they employ NoSQL (like Netflix) but I think I've worked with RDBMS enough (though it's been awhile). Ted Dziuba has a pretty funny critique of the NoSQL craze here.

Amazon Web Services (AWS) SimpleDB was a perfect fit - not only is it free for small fish like me but the new AWS Android SDK includes SimpleDB support. I'm not interested in writing a bunch of HTTP libraries - the AWS Android API does everything for me.

The big knock against NoSQL is its data consistency. You're not always guaranteed to get the data you're expecting. SimpleDB counters with a Consistent Read option which I'm employing. You give up a little speed but for a small app like mine it's a no-brainer. But there are no transactions to track so there's still some risk.

Another surprise was SimpleDB's support of SQl through SelectRequest. You can't do stuff like JOINs (these kinds of "advanced" operations are handled in application code) but it's convenient when you need to pull a targeted set of data.

Special shout out to the folk(s) who made sdbtool, a Firefox plug in that let's you interact with your SimpleDB account.

Here are a couple snippets from my class which handles SimpleDB interaction. Connecting is as easy as:

    BasicAWSCredentials credentials;
    Properties properties = new Properties();  
    try {            
        properties.load(getClass().getResourceAsStream(AWS_PROPERTIES));
                  
        String accessKeyId = properties.getProperty("accessKey");
        String secretKey = properties.getProperty("secretKey");

        // some boring error checking

        credentials = new BasicAWSCredentials( properties.getProperty( "accessKey" ), properties.getProperty( "secretKey" ) );
        // note mDB is an AmazonSimpleDBClient
        mDB = new AmazonSimpleDBClient(credentials); 
    }

Executing queries and putting results in a List is fairly easy:

    SelectRequest selectRequest = new SelectRequest("select * from accounts where m_username = '" + username + "'").withConsistentRead(true);
    SelectResult selectResult = mDB.select(selectRequest);  
    List resultList = selectResult.getItems();

I'm able to really take advantage of the API simplicity when adding stuff to the database. NoSQL architectures seem to be pretty great for these types of actions (as long as the data gets there!).

    List attributes = new ArrayList(1);
    attributes.add(new ReplaceableAttribute().withName("m_username").withValue( username));
    PutAttributesRequest request = new PutAttributesRequest("accounts", username, attributes);
    mDB.putAttributes(request);

You know, if data consistency is critical, you could keep querying the database until your data is positively there...

Amazon AWS and Database.com

Much of my application development has been "data structure centric". Future projects might be in the same vein so while tinkering with Android coding, I'm looking into 2 major database products with hooks into Android (among others devices, languages, and platforms, of course).

Amazon's AWS has been a major web services provider for awhile. Their data storage product is Amazon S3 but the interesting part is the AWS SDK for Android which provides S3 storage, database creation and messaging. In fact, Amazon might have made it TOO easy to deploy mobile enterprise ("cloud") apps has this PCWorld article suggests! Once I've secured a moderate amount of Android development knowledge, I'm definitely going to open an AWS account and play around with this API (the AWS Free Usage Tier is free for a year).

Salesforce has moved into the database-as-a-service market with database.com, which as this InfoWorld article states aims to be the "back end for the entire Internet". While other companies are playing catch up, Salesforce appears heavily leveraged into the cloud computing and ASP worlds so they might be well positioned to be a database of choice for mobile developers. Unfortunately, database.com is not really live yet (their FAQ page states the full developer preview is coming out in 2011) so I guess this is the end for now.