Manually editing Solr's clusterstate.json on Zookeeper

There will probably come a time when you want to do something not readily covered in Solr's APIs.  Manually editing the clusterstate.json is easy to do but should be approached with caution.

I wanted to drop all shards from a single host from my Solr cluster (didn't care about losing the data) and do a little spring cleaning.  To open a command-line interface with ZK:

/path/to/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181

Then simply:

get /clusterstate.json

Place that content into a local file.  After backing up the original content, make your edits.  For instance, to drop the dead nodes, delete JSON elements with "down" states:


To upload your new clusterstate.json (no need to halt services):

/path/to/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181 set /clusterstate.json "`cat /local/path/to/clusterstate.json`"

Mismatched Solr 4.x versions with SolrCloud

I wanted to run a Solr 4.0 client against a more modern Solr server (like Solr 4.3.1 - shard splitting!).  While we should match client and server versions, it's not unheard of to float a little bit.  4.0 to 4.3.1 is more than a little bit but I wanted to get Lily to work against 4.3.1 - notsomuch.

You'll likely get an error like this in the Solr ClusterState code:

org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper... 
Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map 

So something changed with the way the cluster is recorded in ZK/Solr.  I encountered this again by accident when I forgot to update the Solrj version in my app's POM (again, trying to run a 4.0 client against a 4.3+ server).

This is probably only an issue with SolrCloud so remaining Solr classic users likely aren't affected.  But if you're not using SolrCloud yet, you should take a look.

My Weekend Project: trellminder (Trello email notifications)

Love trello!  My wife and I use it to coordinate what needs to be done around the house.  And I agree with the folks at trello that email notifications (and email in general) are lousy.  I don't want to see each time a card changes - the only thing I want to be alerted when a task is about to be due or is past due.  Mainly so I don't get in trouble at home ;-)  So I thought it would be a fun weekend project.

I use CentOS at work so I'm most comfortable with Red Hat and I've played with OpenShift, their PaaS option, before.  This feels like a cron job, which OpenShift supports.  Trello has a nice API so this wasn't too hard to code in Python.  I didn't want to make my Trello board public of course but you can generate a token which allows read-only access to a board, once approved.

Once you get an account on OpenShift, create a python-2.6 app and add the cron cartridge:

rhc app create trellminder python-2.6
rhc cartridge add cron-1.4 -a trellminder

I use requests whenever possible - urllib2 should be avoided at all costs.  To do so, add "install_requires=['requests']" to setup.py in your root dir.

There was a very helpful question on the OpenShift forum which clued me into how this script should run.  Mainly, I should use a shell script to activate the Python 2.6 virtenv then call the python script.  The use of a 'jobs.deny' file would force cron to call the .sh and ignore the .py.

The actual source code is posted on my github repo.

Getting replicated Ehcache and iptables to play nice

Struggled with this a bit and thought others might find this useful.  If you're using RMI Replicated caching with Ehcache, you need to put a little thought into port security/strategy.  The sample ehcache.xml includes:


Using this config means you're going to have to poke holes in iptables for ports 40001 and 40002.  All that is pretty simple - the gotcha is if you're using automatic peer discovery.  It needs multicasting to work.  The docs call this out but it took me awhile to realize I had to specifically allow it in iptables, as it is likely prohibited by default in most environments.

IBM has a nice post about how to do this.  So in this example (replicated Ehcache with automatic peer discovery), you'll need this in iptables on each host you expect to participate:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 40001 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 40002 -j ACCEPT
-A INPUT -m pkttype --pkt-type multicast -j ACCEPT

Using Python and paramiko to move bits around

Automating the movement of files around a stack is one of those unglamorous jobs that just has to get done.  It's certainly possible to do this in bash but Python makes it a little more easy and fun.

paramiko is an actively maintained Python module that assists with secure connections to remote hosts.  This use cases favors the SFTPClient class which easily allows an app to connect to and put files on a remote machine.  paramiko is LGPL and only requires Python 2.5+ and PyCrypto 2.1+.

It's probably best to create an account on the remote host with a very limited scope.  That is, it only has permission to modify the directory which you intend to write to.  Those creds will go in the Python script.  Don't run this as root or your normal user account!

In this simple example, all files in a particular local directory will be recursively copied to a remote host.  Enjoy!

Assembling ordered Lists using HashMap and Collections.sort()

LinkedHashMap won't work if that's what you were thinking....

I was presented with a use case having to assemble an ordered List of objects out of tens/hundreds of thousands of ordered elements.  Here's the List<EventObject> I need to do something with:

• Keys are not unique.
• The List<EventObject> needs to become a List<SummarizedObject>.
• 1 or more EventObjects make up a SummarizedObject.
• The order of List<EventObject> must be matched in the resulting List<SummarizedObject>.  Meaning, the first EventObject will definitely be in the first SummarizedObject and the last EventObject will definitely be in the last SummarizedObject.
• Typically, for every 10k EventObject values 100 unique keys and 200 unique SummarizedObjects are created (give or take 1 order of magnitude).
• I'm not concerned about concurrency at this time.

LinkedHashMap<String, List<SummarizedObject>> wouldn't work for me even though it maintains order because a single key could have several SummarizedObjects (hence the List of SummarizedObjects) whose order could interweave with values in different keys.  Bummer.

I kept coming back to the HashMap.  Being able to quickly determine a key's existence in the Map and (if it does exist) going straight to that key's List<SummarizedObject> was ideal.  So I dropped an order field in SummarizedObject, which looked like:

The pseudo-code for getting the EventObjects into a HashMap<String, List<SummarizedObject>> is:

So my last problem is to assemble all SummarizedObjects into a single, properly ordered List.  Enter Collections.sort(). Here's real code:

Great performance and not that much code. And it can be optimized further by finding a better way to find the proper SummarizedObject in Step 5 of the above pseudo-code. Hope this helps someone!

Marshaling POJOs with Jersey

When a web service has multiple consumers, you obviously don't want to duplicate a lot of code.  So what if you want to push data to/from a Java API and a web site like jQuery?  Jersey can help you do both with JSON.  A lot of what I'll cover is in the Jersey docs.

• Turn JSON support on in your servlet's web.xml:

<init-param>
  <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
  <param-value>true</param-value>
</init-param>

• This example also uses compression so add the gzip filters too.
• Annotate your POJO with JAXB.  Make sure you have an empty constructor.  I like to make the member variables private and force use of public getters.  I also elect to ignore unknown properties.

@XmlRootElement(name = "foo")
@XmlAccessorType(XmlAccessType.PUBLIC_METHOD)
@JsonIgnoreProperties(ignoreUnknown = true)
public class Foo {
    private String blah;

    Foo() {}

    
    @XmlElement()
    public String getBlah() { return blah; }
}

• Set up your web service.  Since you're returning JSON, jQuery will be happy with basically no further consideration.  Note I didn't define a setBlah() above but I hope you get the picture.  

@GET @Path("/")
@Produces({MediaType.APPLICATION_JSON})
public Foo getFoo() {
  Foo foo = new Foo();
  foo.setBlah("Hello!");
  return foo;
}

• Connect your WebResource properly.  Note the addition of the JacksonJsonProvider class.  The gzip filter is not required but you should probably use it.

ClientConfig config = new DefaultClientConfig();
config.getClasses().add(JacksonJsonProvider.class);
Client client = Client.create(config);
client.addFilter(new GZIPContentEncodingFilter(true)); 
WebResource webResource = client.resource(hostname);

• Now request your POJO response (note the use of gzip here):

Foo foo = webResource.path("/")
               .header("Accept-Encoding", "gzip")
               .get(new GenericType<Foo>() {});

• You can also pass POJOs to the web service using POSTs.  This snippet assumes you annotated class Bar is a similar manner to Foo and the "/bar" POST endpoint accepts a Bar object.

Bar bar = new Bar("stuff");

Foo foo = webResource.path("/bar").post(new GenericType<Foo>() {}, bar);

Edited on 2/10/13 to include compression in example.