Using geohashing in Solr

LatLonType might be sufficient for many use-cases regarding Solr spatial searching.  It's a pretty good option and performs decently in benchmarking I've seen.  However, I'm interested in geohashing.  While there isn't a great option out of the box (yet), SOLR-2155 is interesting and I've given it a try.  It was written by David Smiley, whose Solr 3 book I use almost every day.

I grabbed the v1.0.5 JAR from SOLR-2155's github repo.  It's a little confusing where to drop it but I decided on /usr/local/solr/solr/lib/ based on David's advice (see bottom of thread).  The instructions on how to configure schema.xml and solrconfig.xml on the github page are spot-on.  Obviously you have to restart Solr for these changes to take effect.

Then feeding in data is a snap.  I'm using Lily so I created a STRING field in Lily and wired it up to point to the solr2155.solr.schema.GeoHashField Solr field.  When creating Lily records, I send in "lat + ',' + lon" values.  I haven't been able to benchmark Solr (or Lily) indexing but it's no noticable drop-off from my previous attempts with double, tdouble, or LatLonType.

Searching performance is great.  Again, no benchmarks but everything feels snappy (great technical term, I know).  Bounding box queries like this come back as quick as text or string searches which is really thrilling:

http://s1:8983/solr/select?q=*:*&fq=geoloc:[1,-60 TO 5,-55]&shards=s2:8983/solr,s1:8983/solr,s3:8983/solr,s4:8983/solr

One issue I've seen is precision values in Solr result sets.  It looks like a Java double (ie, 3.89999233 is really 3.9).  The underlying data (3.9 in this case) gets preserved but Solr returns the modified value (like 3.89999233).  So just be careful about what you display.

I'll try to get some benchmarks but here's a good brief with some numbers.

What to do when your free app doesn't take off

You write a great app, put it on iOS App Store and/or Android Market, err Google Play, quickly mature to a pay app, sit back and enjoy your steady stream of secondary income.

The majority of cases don't turn out this way. So what happens when your idea doesn't work?

This isn't a post about talent acquisitions or well poisoning. Nor do I have enough experience to comment on pivoting startups or apps/sites that are not free. It's scoped specifically to individuals who independently offer free apps or web sites that, ahem, don't work out. I've got a little experience in this area, having published a free Android time management app which stores user data remotely. It didn't take off but I have some active users who seem to like it enough.

If your product is a standalone widget and you want to kill it off, sundowning immediately shouldn't be too controversial. However if you've actively sought users who have entrusted you with their data, the answer isn't so clear. Not only have they invested time to learn your app, they've actually uploaded their data and expect to be able to access it.

Let's look at the options:

1. Immediately shut down and get out of Dodge: At best this is rude and at worst you've really screwed your former users as they've lost their data forever. Not a valid option for responsible developers.
   
   
2. Hang a going out of business sign: Even if everything else is free, there is an opportunity cost is keeping a 'failed' idea going (discussed more in #3). It's perfectly reasonable for someone to want to move on. The crucial ingredients here are sufficient notice and data reclamation options.
• Build in at least one month of lead time before services get shut off. Two or three months would be more considerate.
• Stop accepting new users (unpublish for apps, no more signups for web sites).
• Notify users via email and/or web site banner.
• If you have an app, consider publishing a final version that includes a conspicuous popup every time the app is launched informing them of the impending shutdown. Drive the point home any way you can.
• In the last couple weeks, review your logs/database for people still uploading data and send them a friendly reminder.
• Give users lots of options for downloading their data to convenient formats (CSV, etc). Even if you didn't have this feature before, you're going to need to build it to shutdown gracefully.
• Remain available even after shutdown for stragglers.
  
  
3. Keep your product available in perpetuity: If its not costing you anything, you could keep it going for the long-term. There are disadvantages and risks! APIs change. Users will still seek support. You could get a bill one day for a service you consume.
   
   Additionally, there should be an expectation that someone is minding the store. Developers who go this route can never completely disconnect. Consider not accepting new users and being up front about your intent to stop iterating.

Failed ideas don't have to be all bad - just be responsible and considerate to your users.

OpenShift test drive

I'm looking for an all-in-one PaaS solution. Currently, my little Android app hosts its data on AWS SimpleDB (I need to swap this out with something relational) and uses GAE for authentication services. While it's been interesting to use these systems, I'd like a single solution.

OpenShift had a booth at PyCon 2012 plus they were handing out free tee shirts. Sold! The only code I'm interested in right now is Java and Python which they of course support (among others). OpenShift also comes with MySQL and MongoDB. I don't believe OpenShift is in production status so I don't know what the pricing structure will be. Whatever it is, I'm sure I'll stay firmly in the 'free' range just as with AWS and GAE. Maybe one day I'll code something that does real volume and I'll have to pay! ;-)

Getting started was a snap. Their control panel UI is still pretty new and can use some polish but OpenShift is very straightforward so it wasn't confusing. git does all the heavy lifting in terms of deploying code, at least for an Express app. They provide a very nice Django example which was much appreciated and worked out of the box. I'm interested in checking out Tornado which is easily stood up in OpenShift. I use the requests module all the time (requests is a urllib2 replacement) and bringing it into my OpenShift app took like 10 characters of typing. I can even ssh to my virtual app, write to the filesystem and fire up daemons at will!

I'm leaning towards porting my Android app to run on OpenShift. If I don't, I'm sure I'll find something else to use it for (probably something for my graduate studies). Check it out.

UPDATE 5/5/12: I used OpenShift to host my semester project for a grad school class I'm taking.  It's available (for now, probably not forever) at http://tornado-ryancutter.rhcloud.com/.  The app attempts to process large amounts of Google+ content and rank attached links intelligently.

The only issue I had with OpenShift was a lack of support for python multiprocessing.  The issue is documented here.  When I discovered the problem, I hopped on IRC and was able to immediately talk to an OpenShift rep (on a Saturday, no less).  Talk about customer service!  Hopefully it'll be fixed one day.  I was able to use python threading to do what I needed to do but I can see how this would be a showstopper for some people.

Overall, very impressed with OpenShift.

Running Jython apps using HBase

Jython is a great tool to have in your arsenal - currently using it for prototyping but it could do much more I'm sure. After yum installing jython onto a Centos 6 machine (not recommended as much newer versions exist), I used this great writeup to set up Jython and HBase:

$ export HBASE_HOME=/usr/bin/hbase

$ export JYTHON_HOME=/usr/bin/jython

$ export CLASSPATH=`$HBASE_HOME classpath`

$ alias pyhbase='HBASE_OPTS="-Dpython.path=$JYTHON_HOME" HBASE_CLASSPATH=/usr/share/java/jython.jar $HBASE_HOME org.python.util.jython'

$ pyhbase

Jython 2.2.1 on java1.6.0_30

>>> from org.apache.hadoop.hbase.client import HTable

>>>

Once the environment is set up, you can write an app. Let's say you have a table called 'test' with a column family named 'stuff' and column named 'value'.

import java.lang
from org.apache.hadoop.hbase import HBaseConfiguration, HTableDescriptor, HColumnDescriptor, HConstants
from org.apache.hadoop.hbase.client import HBaseAdmin, HTable, Get
from org.apache.hadoop.hbase.util import Bytes

conf = HBaseConfiguration()
admin = HBaseAdmin(conf)

tablename = "test"
table = HTable(conf, tablename)

row = 'some_row_key'
g = Get(Bytes.toBytes(row))
res = table.get(g)

val = res.getValue(Bytes.toBytes('stuff'), Bytes.toBytes('value'))

print Bytes.toString(val, 0, len(val))

The only weird behavior I couldn't explain is the need to overload that last Bytes.toString() call. I posted a question on StackOverflow but didn't get any great feedback. I'm thinking it's something in Jython maybe(?).

Setting up Duplicity with GnuPG

Really enjoy the functionality of Duplicity. On CentOS:

sudo yum install duplicity

If you get "No package duplicity available.", you need to install EPEL. For CentOS 6:

sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Then try yum again.

To make a key using GnuPG:

gpg --gen-key

The defaults are fine. When the key is complete, make sure you copy down the key (made bold) because you'll pass it to duplicity:

gpg: checking the trustdb

gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model

gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u

pub 2048R/12345678 2012-01-26

.....

You might need to export the key if another user will use it. In my case, I had to create the keys with one user but another user would execute the backups.

gpg --output secret --export-secret-keys

gpg --output public --export

Then the other user needs to:

gpg --import /path/to/secret

gpg --import /path/to/public

You can verify the keys are there by:

gpg --list-keys

If when using the key you get these errors:

gpg: : There is no assurance this key belongs to the named user

gpg: [stdin]: sign+encrypt failed: Unusable public key

You should (as the user experiencing this error):

gpg --edit-key [key]

> trust

// decide how much to trust it

> save

Now to actually use duplicity, it'll most likely be cron'd so a shell script would work nice. I like the way Justin Hartman did it so there's really no need to re-invent what he did. Just ignore the AWS stuff if you're not backing up there.

Issues with new NIC on Centos 6.0 server

A Dell Poweredge went down with an E1410 error which I couldn't clear. The Motherboard/NIC had to be replaced. This resolved the error but brought about an annoying networking situation.

I really like this nixCraft Howto for Red Hat simple networking issues. After putting the new MAC address on the HWADDR line in /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 (just added 1 to eth0's MAC addr), I ran "/etc/init.d/network restart" expecting everything to come up. No such luck - got Fatal "Device eth0 does not seem to be present" errors. Verified the MAC address, swapped the addresses between eth0 and eth1, restarted the server, no joy.

Eventually someone smarter than me to me to look at eth2 and eth3. Turns out the NIC card was binding to eth3. To resolve this, I commented out the HWADDR line in ifcfg-eth0 and ifcfg-eth1 and restarted the server. Running "ifconfig eth0" showed the NIC now attached to eth0. I wanted this to always be the case so now I uncommented the HWADDR line in ifcfg-eth0 and ifcfg-eth1 and restarted again. I probably could have just restarted network not the server and got the same results but everything was good.

Parallels config - CentOS Server 5.0 (guest) on Mac OS X (host)

Not much documentation on getting CentOS Server 5.0 to work as a guest OS on Mac OS X so here's some gouge:

Installing Parallels Tools:

• To enable networking, be in "Shared Network" and run "/sbin/dhclient eth0".
• Mount the CDROM: "mount /dev/cdrom /mnt/cdrom". You might need to create /mnt/cdrom directory.
• Install the prereqs: "yum install gcc kernel-devel".
• The kernel and kernel-devel package must be exactly the same. To check this, run "uname -a ; rpm -qa kernel\* | sort". For me, this returns 2.6.18-274.3.1.e15 for kernel, kernel-devel, and kernel-headers (don't think headers are required but not sure). If you need to update the kernel, "yum install kernel" and reboot.
• /mnt/cdrom/install should work
• Once installed, my Mac OS X home directory was available at /media/psf/Home.