A solution in search of a problem

I don't believe in algorithmic trading methods though I am aware that most stocks bought and sold every day are done so because a computer decided to pull the trigger.

Be that as it may, the real reason for my foray into this world was to write a program with server-side Python which utilizes a CouchDB storage solution and an Android app which visualizes the data on a canvas. Oh yeah, and lots and lots of JSON.

I don't really have a name for these programs - I call the collection of Python scripts ZapDome. The Android app is called WebFrenzy. I've open sourced the Python stuff but like I told a startup guy this week, please don't judge me by the excellence of this code - I'm a Python noob. I'll post the WebFrenzy code after I clean it up a bit. It's a real mess right now. Perhaps most importantly, it's yielded a pretty clever (in my opinion) lightweight Java CouchDB library for Android devices specifically built for interacting with Cloudant servers called BarcaJolt. I'll blog about that later.

Anyway, my algorithm is to look for significant volume trends throughout the day and compare those intraday prices with the closing price. So I collect data at the top of the hour 10AM - 3PM EST then sweep up the closing price at 5PM EST. If the volume is >= 125% or <= 75% what SHOULD be according to it's moving volume average at that time, I flag it.

For example, if a stock is averaging 1,000,000 shares of volume a day and at noon EST it's already racked up 900,000 shares traded, that obviously meets my definition of significant volume. Let's say at noon EST, this stock's price is $2.00 and at the end of the day, it closes at $3.00. What if this happens often? That is, hours when the volume is "high", it almost always posts a profit according to it's closing price?

Honestly, I don't really care what the algorithm is. I'm never going to use this data - I'm a value investor!. But it's a fun hack and gets me experience in things I want to learn more about (Python, CouchDB, JSON, Android apps).

So here's a couple screen caps of my Android app showing the behavior of 2 stocks I've been tracking over the last week (I've only pulled a week's worth of data). Obviously you can't pull any trends out of such a small data sample. The data is on an X/Y graph (Vollume/Price). So the top right quadrant shows instances where the stock's volume was high and the stock price rose before the close. More to follow.

A CouchDB library for Android

So I really liked this article that was featured on Hacker News a week ago and it got me to thinking that maybe I'm using too many 3rd party libraries to help with areas I don't have much experience in (couchdbkit, for example, is awesome). So instead of using Ektorp to help me with Java/CouchDB/JSON stuff, I think I'm going to write my own library to help me with the Android portion of an algorithmic trading system I'm playing around with.

There already is a well-known CouchDB API build specifically for Android called droidcouch but I think this space can use more than 1 option and I have some ideas.

The most striking change between 2003, when I typed my last professional line of code, and now is the influx of 3rd party libraries. The infrastructure necessary to get a lot of adopters is already built out with github/Google Code/sourceforge.

Some basic couchdbkit 'views' (get it?)

We had to go to LA this weekend and I made my wife drive so I could learn up on CouchDB views. I had to re-read the Design Document and View sections of my O'Reilly book a couple times before it really clicked. Thankfully, the book is detailed enough that I didn't need to be looking at a database through Futon to get the point.

So today I hacked together a simple view in a Python script. This returns all my documents with a .symbol of 'C'. I'm not good at JavaScript (yet) so this is about as advanced as I want to get as I focus on Python and CouchDB:

design_doc = {
    '_id': '_design/test',
    'language': 'javascript',
    'views': {
        'allcitis': {
            "map": """function(doc) { if (doc.symbol == "C") { emit(doc._id, doc); }}"""
            }
        }
    }
frenzydb.save_doc(design_doc)

No problem - worked the first time and I saw it my Cloudant database. Parsing the view results through couchdbkit to a little longer to figure out. The Getting Started page ends with:

greets = Greeting.view('greeting/all’)

It took me a little while to figure out 'greets' wasn't a list, it's a couchdbkit.client.ViewResults. As it says in the comments for couchdbkit/client.py, "It return an ViewResults object on which you could iterate, list, ...". Okay great, throw it in a list. Here's a trivial function to get the average of 'price' using the aforementioned view:

citis = list(frenzydb.view('/test/allcitis'))

sum = 0
for citi in citis:
    sum += float(dict(citi['value'])['price'])
print round(sum / len(citis), 2)

Google App Engine is pretty great (if you go all the way)

So I'm starting to bang around another project. As I describe it on my Careers 2.0 profile:

Project is still in its infancy but it will have server (GAE-hosted Python program evaluating JSON feeds of stock data with Cloundant [CouchDB] backend) and mobile (Android app to visualize/manipulate the data from Cloundant) components. There really isn't a point behind this program and I don't necessarily condone algorithmic trading - it's just a fun hack.

Or so I thought. Doesn't look like I'll be using Google App Engine because I don't want to do it the "GAE way" (not that there's anything wrong with that). Deploying an app to GAE is super simple but I ran into problems when starting to incorporate 3rd-party libraries (couchdbkit and restkit to be exact). These libs are too small to be part of the standard runtime environment. It's interesting to note that even Django is implemented in a slightly different way on GAE.

I'll spare the details (though they are available in my stackoverflow question) but I was having importing couchdbkit into my script. It was choking on things likes sockets and resources, which made me think I was running afoul of the sandbox rules:

An App Engine application cannot:

• write to the filesystem. Applications must use the App Engine datastore for storing persistent data. Reading from the filesystem is allowed, and all application files uploaded with the application are available.
• open a socket or access another host directly. An application can use the App Engine URL fetch service to make HTTP and HTTPS requests to other hosts on ports 80 and 443, respectively.
• spawn a sub-process or thread. A web request to an application must be handled in a single process within a few seconds. Processes that take a very long time to respond are terminated to avoid overloading the web server.
• make other kinds of system calls.

I put a question on the Cloudant discussion board and got not 1 but 2 answers from Cloudant techs within an hour (these dudes are good - I don't even have a paying account and they are very responsive). Anyway, they tipped me off to this telling Quora answer. It looks like the Django/Python hosting space is about to blow up with Heroku-like solutions.

Both the Cloudant tech and Django responder (a Django co-inventor) had good things to say about WebFaction. It is not a free hosting service ($9.50 per month and goes down with longer term contracts). After a quick Q&A session with a sales rep, I took the plunge. 1 hour later, my program was working perfectly on my WebFaction account. Success! The only thing that makes me nervous is having to do any sys admin stuff at all - getting my script and installations to reference the right Python version (WebFaction comes everything from 2.4 [default] up through 3.2) is about all I care to handle.

Python + Couchdbkit = nice

So I'm working on a CouchDB project and you can read more about what I'm doing on the mobile side here.. There will be a server-side component and I want to get my feet wet with Python. I'm currently working through the tutorial but like most people I learn more by doing.

There are several CouchDB libraries for Python but I went with Couchdbkit. This decision was influenced by a post on the Cloudant discussion board stating Couchdbkit and Cloudant should work well together. So far, I have to agree.

The installation instructions for Couchdbkit are pretty straight forward. For the record, I'm using Python 2.6.6. I did find I needed to use restkit for HTTP resources. I set my sights low - I just wanted to get the program of Couchdbkit's Getting Started page to work. Username/password redacted to protect the innocent:

#! /usr/bin/env python

import datetime
from couchdbkit import *
from restkit import BasicAuth

class Greeting(Document):
    author = StringProperty()
    content = StringProperty()
    date = DateTimeProperty()

server = Server('https://[username].cloudant.com/', filters=[BasicAuth('[usename]', '[password]')])
db = server.get_or_create_db("greeting")

Greeting.set_db(db)
greet = Greeting(
    author="Benoit",
    content="Welcome to couchdbkit world",
    date=datetime.datetime.utcnow()
)
greet.save()


There are more exercises on the Getting Started page including an introduction to views. This worked as advertised with no need to change the supplied code.

Rocking and rolling with Ektorp and Cloudant

My first Android app was developed using AWS' SimpleDB. While I like the Android SDK, I'm not terribly satisfied with the performance I'm getting. As far as NoSQL databases go, CouchDB is getting really great press. Cloudant is a well-respected YC-aligned startup which hosts CouchDB databases. I decided to give it a try, as I'd like to get deeper into JSON as well (generally, SimpleDB uses XML while CouchDB uses JSON).

Ektorp is one of several Java APIs for CouchDB but it doesn't look like many people (anyone as far as I can tell) are using it with Cloudant. So I decided to give it a try just for fun.

First off, Ektorp requires a lot of external JARs (9 or 10 from what I see). Importing the 1 org.ektorp JAR and 8-9 other JARs into an Eclipse project quickly proved tiresome. Ektorp recommends using Maven, which manages all dependencies. Better yet, m2eclipse offers a way to integrate Maven with Eclipse. The setup instructions and videos are very self-explanatory. Dropping the supplied Ektorp entry into my pom.xml was simple. All required dependencies were quickly and correctly imported. Piece of cake.

Okay, now to connect to Cloudant. Getting a free Cloudant account is easy. The database UI is clean and simple. I want to be a good coder and use SSL, which Cloudant can certainly support. Ektorp supplies a good API tutorial which accomplishes what I want to start off with:

HttpClient httpClient = new StdHttpClient.Builder()
.host("localhost")
.port(5984)
.build();

CouchDbInstance dbInstance = new StdCouchDbInstance(httpClient);
CouchDbConnector db = new StdCouchDbConnector("mydatabase", dbInstance);

db.createDatabaseIfNotExists();

Initial attempts proved unsuccessful as I couldn't figure out what .host() to use. The supplied documentation wasn't clicking - on to StackOverflow where I'm sure someone's dealt with something similar before. Not being able to add an "ektorp" tag to my question leads me to believe the library isn't as prevalent as I thought. No worries, someone was able to kindly point out I shouldn't be including "http://" or "https://" in my host variable. Oh yeah. Read my StackOverflow question for the details. So I was able to get it working using http as follows:

HttpClient httpClient = new StdHttpClient.Builder()
.host("[username].cloudant.com")
.port(5984)
.username("[username]")
.password("[password]")
.build();

Getting SSL to play nice proved difficult. Using port 443 didn't readily work until someone from Cloudant wisely told me to RTFM. Okay, adding .enableSSL(true) and .relaxedSSLSettings(true) is a good point but I was still getting an IllegalStateException and the stack trace indicated it was trying to use http after initializing my object with https. Huh? Finally, after scouring the Ektorp discussion board, I found this post which indicated Ektorp 1.1.1 included SSL-related bug fixes. So I updated my pom.xml to look for 1.1.1 vice 1.1.0. Recompile Maven, run Java application, success! Database added to my cloudant account via https.

I was getting an annoying but non-fatal exception relating to SLF4J:

Failed to load class "org.slf4j.impl.StaticLoggerBinder"

This is fairly common. Manually adding slf4j-simple-X.jar (slf4j-simple-1.6.1.jar to be exact) to the project resolved the issue.

So this is my super-complex program:

HttpClient httpClient = new StdHttpClient.Builder()
.host("[username].cloudant.com")
.port(443)
.username("[username]")
.password("[password]")
.enableSSL(true)
.relaxedSSLSettings(true)
.build();

CouchDbInstance dbInstance = new StdCouchDbInstance(httpClient);
CouchDbConnector db = new StdCouchDbConnector("httpsdatabase", dbInstance);

db.createDatabaseIfNotExists();

and remember to make ektorp's dependency look for version "1.1.1" in your pom.xml.

Ext4 delayed allocation data loss

My Ubuntu (Maverick Meerkat) netbook crashed this week and completely wiped an open .odt document. The behavior was very similiar to what happened here. I'm not a file system guru so I wasn't aware until now of the long standing issue with Ext4 file system delayed allocation. From LinuxInsight:

In Ext4 file system, Delayed Allocation causes some extra risks of the data loss if your system gets crashed before all the data is written to hard drive.

This was also covered on slashdot over 2 years ago.

It appears that the transition to Ext4 requires applications properly use fsync() to prevent data loss. Theodore T'so, who played a major role in developing Ext4, includes a good summary here. In fact, Android devices which run on Ext4 are rolling out and preventing data loss has been a major concern.

I can't imagine the Open Office that ships with Ubuntu 10.10 doesn't adhere to the Ext4 spec and use fsync() properly but I've never had this problem in other apps. Very aggravating and perhaps another reason to use Google Docs or give Libre Office a try!