Thoughts on ZeroMQ

I’ve been doing a bit of ZeroMQ work in Clojure at work, and I thought I’d share my thoughts on it.  It’s worth understanding that what I was building was a globally distributed system, written for the JVM, involving long-running queries and peer-to-peer communication.  If your use case is different from this, you might have a radically different experience.

The Good

ZeroMQ basically gives you an interface at the C API level that looks very like a POSIX socket.  It then implements a packet protocol on top, so you never need to worry about receiving partial messages.  It’s also got some nice magic in there to allow you to connect to a server that is currently down and have the messages queued.  This is pretty nice in terms of reducing the fragility of the system.  Also, as a developer, it means that you can stop worrying about whether you started the client or the server first when checking something out.  (Also, doing the demo where you bring up a missing server and communication resumes is great for boss wow points.)

What it doesn’t do is tell you how to serialize your objects.  This only sounds like a good idea if you think that local proxy objects are a good idea.  The rest of us would like control over our wire object format, thank you very much.  So, for instance, for small objects you can use Protocol Buffers or Thrift.  If you’re sending a lot of data you can knock yourself out and gzip the entire thing.  All of this without having to figure out where the hooks in the serialization API are.

So, ZeroMQ basically takes your socket programming and simplifies it.  Well, it would except for a couple of things. 

The Bad

The documentation looks much better on a skim through than when you dive into it

• There’s an awful lot to learn for even simple cases.
• Most of the examples aren’t available in Java.  (Or most of the many other languages for which ZeroMQ has wrappers.)
• The guide keeps repeating itself.
• The Javadocs for JZMQ aren’t available anywhere online.
• The wire protocol is completely undocumented.
• The guide keeps repeating itself.
• JZMQ has a wilfully different API from the C API.*

One or two of these would be fine.  All together, it’s a bit of a perfect storm for a newbie.

Next, for all that they talk about supporting multi-threading, asynchronous programming is very much regarded as the advanced option.  Most of the simple use cases involve blocking calls and enforced send/receive ordered pairs.  There’s some very very clever code going on in ZeroMQ to make it do what it does, but all of the REQ/REP stuff does is make it easy to write blocking IO calls in an asynchronous environment.  It’s kind of like node.js in reverse.  Frankly, I’ve spent years getting up to speed with thinking asynchronously.  Sticking a synchronous layer in the way is just going to complicate matters for me.

The pub/sub stuff is only appropriate if you want every client to receive every message.  There’s no real subscription API.  This would require a whole extra layer on top of the socket layer, so it’s understandable why they haven’t done it.

*And sometimes the implementation is a bit… odd.  For instance, there’s two different ways to set a poll timeout.  Only one works.

The Ugly

This bit may make no sense unless you’ve spent a couple of days with ZeroMQ.  I should say right now that I never did succeed in getting router to router communication working.  No idea why, my code looked pretty similar to the C code and the req to router stuff worked fine.  I did, finally, figure out that I could solve my problem using dealer to router. This also improved the design a bit.

With that said, seriously, have you looked at the examples of router to router communication?  There’s a sleep command in there.  That’s not just lazy coding, that’s the only way to avoid dropped messages.  The only general way to handle this is to manually send heartbeats to your peers.  Pretty much most of the benefits of ZeroMQ just start to disappear with router to router communication.  It comes with a health warning in the guide but frankly, the typeface isn’t nearly large enough.

A Few Thoughts More

It seems like I’ve ragged on ZeroMQ quite extensively.  It’s been an intensely frustrating experience getting it to work, but it does achieve a fair bit of dull and tricky stuff in the background that I no longer need to worry about.  I think it’s got a slightly odd attitude to asynchronous processing. (To be fair, it was pretty much the state of the art when I read Tenenbaum.)  However, it’s fast and it solves problems.  What more did you want?

I’m not sure I’d use it for intra-process communication the way that Mongrel does, Retlang/Jetlang/Rx all seem to solve the problem with significantly less fuss.  No, none of these are available in C, but it wouldn’t be that hard to port Retlang to anything with function pointers, mutexes and finally clauses.

Finally, I’ve only been playing with it for a couple of weeks.  A ZeroMQ expert could undoubtedly introduce you to many more advantages of it than I could.