Author: Julian Birch

Anyone who looks out from .NET from time to time will know that our libraries are missing quite a lot that other platforms take for granted.  A decent embeddable web server is one.  Thankfully, the Nancy team have addressed this issue with the Nancy self-hosting engine.  It’s far from the fastest thing on the planet, it’s no substitute for IIS, and it’s not going to give the authors of Jetty any sleepless nights, but it does hit one sweet spot that is close to my heart: live diagnostics.

You see, even if you do your logging and your alerting well, these diagnostics are after the fact and rarely contain detailed enough information to solve certain kinds of problems.  These are the problems where you need to know what’s going on right now and why?  There’s two reasons for this:

• Typically you don’t have live access to the in-memory structure of the application
• There’s just too much information. You need some way to drill down.

The too much information problem is serious.  Quite simply, you can’t sensibly log the level of detail for every last transaction you sometimes wish you had for just one or two.  The in-memory problem is so large many don’t even notice it’s there.  Rather than take a look at what their program is doing, they accept that they’re just going to have to theorize about what state it is in and how it got there.  An embedded web server solves both problems.  You might be thinking that this kind of quality control can be achieved by unit testing, and you’d be right, except that you wouldn’t have captured the amount of time it takes to develop a test when you’re not sure what’s wrong.  I recently (successfully) put an automated trading system live, and I would estimate that 40% of the test suite was developed with the aid of the live diagnostics the system provided.

Next time you’re writing a back-end service, try embedding a small web server that just prints up a single page of information about what’s happening.  Once you’ve had a taste, I’m sure you’ll find it addictive.

I’ve used VS2012 for a couple of days now and have been pleasantly surprised.  It’s a lot faster than VS2010, which was an appalling piece of engineering.  The dark theme, my personal preference, is basic but pleasant.  It doesn’t quite make its way everywhere: property pages, for instance are not only dark theme unaware but 100% as ugly as they were in previous version.  The new event log is gorgeous and I’m hoping it’s going to prove useful.

On the other hand, it doesn’t exactly feel finished.  Autocomplete on symbols pops under the function documentation tooltip, a bug that seems to have sailed straight past beta testing.  They’ve also added a well-meaning feature to look up help on exceptions that just doesn’t reflect the way that exceptions are actually implemented in the .NET Framework.  For instance, if you’re having a problem connecting to SQL Server, it’ll send you straight to the general SqlException page.  To fix this, they either need to make the feature smarter, or make exceptions in .NET a lot more specific. 

Oh yes, and unless you read English in a different way from the rest of the human race, you’ll want to read the contents of this page pretty much straight after booting it for the first time.

Richard Stallman has long said that open source software isn’t the same thing as free software.  Indeed, he’s spent a lot of time educating people about the distinction.  A distinction about which, frankly, few of us care.  Most of us admire his principles, but then get back to producing software for corporations and, as long as it’s open source, it’s fine.

Part of the point is that it’s free at the point of use.  Yes, there are many other advantages, but the simple fact that you just have to say to your manager “can we use this” rather than having to deal with a purchasing decision makes a huge difference to the time to market of anything you build with open source.  It’s not really about whether the price is $100 or $10,000, it’s just plain harder to get commercial software adopted.  Of course, people are always trying to find a loophole in the terms of these licences.  TiVo is the obvious example, which GPL3 addressed.  Even then most firms didn’t worry about the GPL.  Despite the FUD, if you just wrote internal applications and weren’t selling the software to third parties, the terms didn’t make much of a difference.

Then the web opened the mother of all loopholes.  You could run GPL software on the server, and it wasn’t considered “distribution”.  Who’s doing this?  Well, Google for one.  So, the AGPL was born,* that includes the products of your code.  So, you can’t use an AGPL PDF generation software component in your web stack without everything becoming subject to the AGPL’s requirements.**

Again, there are people trying to make money out of open source software.  However, the AGPL provides a new opportunity: the terms are so unacceptable to most firms they might well prefer to pay for the software.  The AGPL is the first license that’s likely to be free as in freedom but not free as in beer.  The phrase “open source and free software” is turning into “free software versus open source”.

People trying to develop commercial open source have a legitimate beef with the community.  So it’s not a surprise to see that some have decided to monetize their reputation*** and have been quite upfront about their reasons for doing so.  Now, you might think “what’s the problem?  No-one’s forcing you to use it.”  And you’d be right, both RavenDB and Lamson can be quietly ignored.  Nonetheless, it’s a pity open source has lost the likes of Zed Shaw.

Somewhat more problematic are the projects that have migrated licenses, like nServiceBus.  nServiceBus isn’t even a library, it’s a framework that most likely extends into most parts of your business critical code.  If you want to upgrade, and the chances are than any successful project will eventually, you’re going to be paying.  The first, and the second one, is free.  The third… well, it’s not gonna cost you anything like as much as BizTalk.  Maybe your manager won’t notice that the “free” library now costs money, but if he does, he’s likely to be asking much harder questions the next time you want to use a “free” library.

You could say that some of us have been leeching off the amazing work done by amazing developers and expecting a free ride for too long, and you’d be right.  Maybe open source software is fundamentally subject to the tragedy of the commons.  But RMS warned us: open source is not the same as free software. 

*I should point out for the sake of accuracy that the original AGPL in fact predates GPL3.

**This isn’t an idle example, iText has moved to AGPL.

***For the record, I think the arguments in the Oracle vs Google case suggest Oren’s understanding of “derivative work” is defective.  Unless Oracle wins, in which case he’s right on the money.

This is a bunch of ideas for macros that I know I’m never going to get around to implementing, but I would find insanely useful if anyone else did.

Magic Partial

Partial in Clojure works, but it’s slow.  You can tell this by the number of times you see functions that return other functions, rather than applying partial.  What you really want is to declare a function once and get the partials declared for you.  e.g.

(defn-partial wrap-authenticated [handler request]
    (handler (assoc request :user (get-user request)))

gets rewritten as

(defn wrap-authenticated
   ([handler request] (handler (assoc request :user (get-user request))
   ([handler] (fn [request] (handler (assoc request :user (get-user request))))

A better solution would be, of course, to fix the performance of partial, but that’s a lot harder.

There’s a harder version of this: to evaluate the function up to the point it needs the unprovided parameters.  But just the basic version would be a performance win.

Generators

C# implements generators like those in Python.  However, it’s worth noting that the feature is entirely supported by the compiler, there’s no MSIL level implementation.  The function gets written as a state machine.  To go purely functional, you’d probably want to do it by constructing a sequence of (value, continuation) pairs and then calling (map first coll) on that.  Obviously, you’d want to support all Clojure special forms.

Await

Await is a new feature of C#, but that’s no reason Clojure can’t have it.  It’s already been done for CoffeeScript.  If you really want to be flash, implement generators and await using the same code.

I was honoured enough to do another lightning talk at SkillsMatter.  I’m afraid I had easily the least interesting talk of the evening, as you can see from the hastily prepared slides on Google Docs.

Neale Swinnerton gave a great talk on Paredit.  I’m still trying to nail paredit, and this was a great help.  The slides are a thing of beauty, watch them in full screen mode. 

Nick Rothwell gave a lightning talk that I wish had been longer, on teaching Clojure to artists.  He’s done some amazing and thankless work on embedding Clojure in MaxMSP.  As always, he gave a great demo.

Malcolm Sparks has already written up the main talk, which was excellent.  I’m not convinced I agree with him about excluding executable code from configuration (I have form on this).  Ultimately, I believe that “configuration” is just a way we describe code properties that different between environments.  There’s no need for it to be XML, or an INI file, or RDF.  The only requirement really is for it to be findable and editable at short notice.

Neale is doing another talk on 6th March on Clojurescript One.

Anyone learning Clojure like I am is recommended to read the source code to Jay Field’s Expectations library.  I was particularly interested to read his article on scenarios.  There’s some very interesting ideas there, but at the moment I wanted something a bit more basic.  “with-redefs” already provides the ability to replace one function with another, specifically for mocking purposes.  So really all I wanted was the ability to capture function invocations.

So, here’s a working mechanism.  You use with-captures exactly the way you would use with-redefs, only you can call the “captures” function on the redefined function and it will return the list of previous function invocations.

The most interesting aspect of the code is the use of a record implementing IFn.  Implementing records well remains a bit verbose (well, by Clojure standards).  I’d highly recommend checking out defrecord2 by David McNeil, which is looking to simplify working with records.

(ns hendrix.test.capture)

(defn ignore
  "Ignores the inputs and returns the outputs.  Useful as mock target."
  [& args] nil)

; Capture

(defn capture-invoke [{:keys [f captures]} args]
  (let [r (apply f args)]
    (swap! captures conj args)))

(defrecord capture-t [f captures]
  clojure.lang.IFn
  (invoke [this] (capture-invoke this []))
  (invoke [this a] (capture-invoke this [a]))
  (invoke [this a b] (capture-invoke this [a b]))
  (invoke [this a b c] (capture-invoke this [a b c]))
  (invoke [this a b c d] (capture-invoke this [a b c d]))
  (applyTo [this args]
    (clojure.lang.AFn/applyToHelper this args)))

(defn new-capture [f]
  (new capture-t f (atom [])))

(defn to-capture [[v f]]
  (new-capture (if (= f :v) (var-get v) f)))

(defn to-capture-map [h]
  (zipmap (keys h) (->> h (map to-capture))))

(defn captures [c]
  (-> c :captures deref))

(defn with-captures-fn [bindings action]
  "Like with-redefs-fn, only you can call 'captures' on the redefined functions."
  (-> bindings
      to-capture-map
      (with-redefs-fn action)))

; Code ripped off from with-redefs
(defmacro with-captures
  "Like with-redefs, only you can call 'captures' on the redefined functions."
  [bindings & body]
  `(with-captures-fn ~(zipmap (map #(list `var %) (take-nth 2 bindings))
                              (take-nth 2 (next bindings)))
     (fn [] ~@body)))

(defn add-two [x] (+ x 2))

(defn example []
  (with-captures [identity :v
                  add-two ignore]
    (identity 3)
    (identity 6)
    (add-two 7)
    {:ignore (captures add-two)
     :passthrough (captures identity)}))

I was interested to read Paul Ingles article on uSwitch’s handling of SSL.  Particularly interested because I remember writing the code he is replacing (It was about seven years ago, I believe…).  I thought makes a good case study in how your language choice affects your code.  To recap his article, uSwitch’s architecture offloaded SSL to hardware.  This meant that detecting whether you were running under SSL was impossible without a custom header.  Furthermore, the SSL/Non-SSL functionality was a bit more complex than a simple “redirect insecure pages”.

However, after you’ve added the custom header to your load balances from a programming perspective your problems are only just starting.  The uSwitch code base at the time was ASP.NET 2.0.  To access the request, you had to use the Request object, which was sealed.  So you didn’t really have any other option than to define a method that tested for SSL and shout at anyone who forgot to use it.  uSwitch had a large number of developers and high turnover, which made things like this an on-going issue.

Solving in Ruby

Paul’s solution to this problem is pretty much the right thing to do in Ruby.  He modifies the behaviour of Rack in order to support his use case.  No-one else working on an unrelated part of the code base ever needs to know about this subtlety of the SSL implementation.  Thus, a dynamic typed language with mutable classes demonstrates massively better separation than the statically-typed C#.

So Ruby has taken the pain away in a page of code.  However, can we improve on this if we move to a dynamically typed language with immutable state?  Like my favourite language, Clojure.  Well, are there any problems left?  Yes, although admittedly they’re not huge:

• We’ve moved the problem of “make people call this custom function” to “make people call this standard function”.  If they’re used to old versions of the rack and examine env directly, they’ll circumvent it.  (Admittedly, it’s my belief that Forward’s developers are typically of higher quality than old-school uSwitch’s.)
• We’ve modified Rack to get work done, which is fine as long as no-one else does the same thing.
• We’ve actually had to duplicate some of the functionality from Rack to make sure it behaves the same way.

Solving in Clojure

Now, the only way to remove the first problem is to actually modify the request.  In languages with mutable state this is usually regarded as a bad thing, because it’s very hard to track exactly what could be affected by your change.  Immutability solves that.  Modifying the request also prevents us from having to modify the underlying architecture, so any solution we build will be fully composable with anything else we choose to use.

(defn non-standard-https [handler request]  
    (handler (if (get-in request :headers "HTTP_USWITCH_HTTPS")  
                 (assoc request :scheme :https)  
                 request)))  
                 
(def handled-routes (partial non-standard-https routes))

What has the Clojure version given us?  Because the request is immutable, only the routes we choose to affect take the functionality.  (Not really an issue in this particular case, but still a nice property to have.)  We’ve not had to modify Ring: its middleware is fully composable and because of this property, we haven’t had to duplicate Ring functionality in our own code.  It’s not configurable, but at four lines, it hardly needs to be.

To recap:

• Handling this simple piece of functionality at uSwitch in C# was a constant code-quality issue.
• Handling it in Ruby pretty much fixes the problem.
• Handling it in Clojure makes it look like a non-problem.

I’m genuinely of the belief that the only thing Clojure’s web solution lacks is maturity.

TL;DR For all that people think of Clojure as a “hard” “propeller-head” language, it’s actually designed right from the start not for intellectual purity, but developer productivity.

I thought I’d share my recent experiences of developing a commercial Clojure application.  It’s currently racking in about 800 lines of Clojure and 2000 lines of CoffeeScript.  I avoided ClojureScript for the simple reason that I had no experience in it, so I’m afraid I have no great insights there.  I guess it’s worth pointing out that although this is the first time I’ve really used Clojure in anger, I’ve been studying it for two years and have spent weeks of time practicing.  I can recommend pretty much all of the main teaching tools, labrepl, Clojure koans and 4clojure.

So, what does the application do?  It’s a web tool for calculating and analysing various costs my bank incurs.  The calculation itself and the UI is pretty much entirely in CoffeeScript using Backbone and jQuery.  The 800 lines of Clojure perform the following functions:

• A REST back end for the data.
• Windows authentication
• Serving the front page, which is the entire application.
• User permissioning
• A quick asset pipeline that compiles the CoffeeScript on demand.
• Charting

Adding Excel export shouldn’t make the code base much larger, maybe another 100 lines.  I should add that most of those 800 lines aren’t long and a fair number of them are blank.  To give you an idea of just how expressive this is, I had an earlier back end written in C#.  800 lines wouldn’t get you to the end of the entity definitions and NHibernate mappings.

More Than Just Concise

I suspect even Clojure’s greatest detractors would agree that it’s terse.  However, the real win is just quite how productive it is.  The more I’ve worked with it, the more surprised I’ve been and just how effective it can be.  Whilst certain languages on the functional side such as OCaml, Racket and Haskell have communities that have many competing priorities, of which developer productivity is only one, Clojure has a laser focus on beating the averages.  The first and most significant of the productivity features is its status as a JVM language.  This gives you access to a huge number of libraries straight out of the box:

• Waffle
• POI
• JTDS (which I used for SQL Server and Sybase connectivity)
• JFreeChart

Without POI, Excel export would be just a CSV file.  Without Waffle, Windows Authentication would be impossible: that is to say, there’s no way I could justify the expense of getting it to work and I’d have been left with implementing my own authentication system.  JFreeChart wasn’t absolutely essential: I could probably have got away with drawing the images myself, but still my productivity was raised by having it there.  Finally, JTDS is rock solid and fast.  The database connectivity story of Java is light years ahead of anything you’ll see in Node,* never mind Haskell.

There is a catch to all of this: you have to read and work with Java.  This isn’t particularly hard, but it means you become very familiar with the wall of inanity.  (I keep thinking “seriously, how many layers of abstraction do you need?”.)  However, there are thoughtful features that enhance the Java experience.  .. gives you the ability to chain method calls with less ceremony than Java.  “doto” gives a the advantages of jQuery style chain syntax without any need to change the underlying methods.  Finally, the REPL is a real help.  Getting the exactly correct chart settings is much easier when you can adjust it on the fly.  (You’ll note that this is a scenario that TDD doesn’t help you with at all.)

Better Than A Better Java

After the ability to just use Java libraries, there comes the excellent Clojure libraries.  Leiningen is an excellent build tool.  clj-webdriver takes the pain out of Selenium programming.  Andrew Brehaut has already written a pretty definitive guide to the Clojure web stack.  I’ve had the ability to edit any file in my server and have the changes reflected immediately (providing the file compiled).**  It’s amazing how few web frameworks can actually do this.  For instance, ASP.NET can do it for views, but not models or controllers.

More than just the fact that they’re good is the sheer pleasure of interoperability.  A lot of this comes from the basic philosophy of composability.  When everything’s a function, sequence or a hashmap, connecting one module to another is pretty easy with very few dependencies.  It’s also eminently testable.  However, Clojure goes deeper than that with extremely thoughtful features.  For instance, I’m using incanter to generate charts.  They return JFreeChart objects.  Compojure defines a protocol called Renderable that allows you to specify how return types get converted into Ring responses.  Now, you could apply an adapter pattern in most languages to link these two up.  In Clojure, you can declare that a type implements a protocol and it just works.  You don’t need to extend the type; you don’t need to monkey patch it; you don’t need to introduce a proxy object under your control.

There are other features that make developing easier as well, such as macros, dynamic binding, software transaction memory and multimethods, that I haven’t even needed to use.  I’m left with a very readable code base that is sensibly structured and easily expandable.  Surprising as it may seem, I’ve actually found it even faster to use than node.js.

*Although Brian Carlson’s node-postgres is quite superb.

**Some of this I wrote myself, but frankly, it only took few hours, and that included figuring out how to integrate with Leiningen.

I did a lightning talk for the LCUG earlier this month.  The idea was to give an idea of the things you can do with closures and was principally aimed at Java programmers who are interested in Clojure.  These are the slides, these are the speaker’s notes and here’s the video taken by skillsmatter.  It come in at under six minutes.  I was the first “support act” for Sam Aaron, whose talk isn’t up since it was a dry run for his Clojure Conj talk.  However, you can also see Philip Potter talking about labrepl and Nick Rothwell talking about DJing with Clojure.

David Havelin was the guy who broke this news: you can now use CoffeeScript directly on windows.  Here’s what you need to do:

• git clone git://github.com/jashkenas/coffee-script.git
• cd coffee-script/bin
• Download the latest node.exe (http://nodejs.org/dist/v0.5.7/node.exe) into the same directory.  Note the is an “unstable” version, but you can’t get away with 0.4.x
• ./node coffee –compile c:myfile.coffee

Finally, throw away all of your over-complex coffeescript on windows solutions.  Hopefully soon there will be a batch file.