Flexible language

I've been learning Lisp for few years now, and every Lisp book I read keeps saying that Lisp is a flexible language that you can extend to the degree when it fits naturally to your domain. It's easy to say, but what exactly does this phrase mean? After all, when you program in your non-Lisp language, don't you modify it for your domain problem? I've been thinking about it for a long time, and only recently I started to understand what flexibility really means. There is a difference between using the language and changing the language to solve a problem. In this post I will try to show the difference based on a simple example.

Problem

Suppose you have a process that listens to a message queue. The messages are just ordinary maps. If the map contains certain keys, one or more handlers must be invoked. Here is a matrix that shows which handler is invoked for which key

For example, if the map has key a, then DocHandler and AlertHandler need to be called. If it has key b, then NoteHandler and AlertHandler are called. In reality there might be more keys and more handlers, but for simplicity we limit our example to three keys and three handlers.

Java

Let's see how this can be implemented in Java. I chose Java just as an example of non-Lisp language. You can pick any other non-Lisp language instead.

public class SimpleMessageListener {

    public List onMessage(Map message) {
        List result = new LinkedList();
        if (isDoc(message)) {
            result.add(handleDoc(message));
        }
        if (isNote(message)) {
            result.add(handleNote(message));
        }
        if (isAlert(message)) {
            result.add(handleAlert(message));
        }
        return result;
    }

    // Decision makers

    private boolean isDoc(Map message) {
        return message.containsKey("a") || message.containsKey("c");
    }

    private boolean isNote(Map message) {
        return message.containsKey("b") || message.containsKey("c");
    }

    private boolean isAlert(Map message) {
        return message.containsKey("a") || message.containsKey("b");
    }

    // Handlers

    private String handleDoc(Map message) {
        return String.format("Document:%s:%s", message.get("a"), message.get("c"));
    }

    private String handleNote(Map message) {
        return String.format("Note:%s:%s", message.get("b"), message.get("c"));
    }

    private String handleAlert(Map message) {
        return String.format("Alert:%s:%s", message.get("a"), message.get("b"));
    }
}

The internals of handle- methods might be very different in reality. Consider the fact they have the same structure as a coincidence. What is not coincidence though is the structure of is- methods. Those methods are identical indeed.

Is this code clean? I would say, no. The main issue is that it's split in three separate but closely related parts. If tomorrow I introduce another message key and a new handler, I have to change three places in the code. Another problem is the code duplication in two spots: a series of if-statements and a group of is- methods.

The last thing to notice about this code is that it's hard to see what kind of problem it's trying to solve. If I didn't provide a matrix which maps message keys to handlers, it would take even more time to figure out what the code is doing. Can we make this code better?

Let's rewrite it as follows:

public class FunctionalMessageListener {

    private interface Handler {
        void handle(Map message, List acc);
    }

    private class DocHandler implements Handler {
        private boolean isDoc(Map message) {
            return message.containsKey("a") || message.containsKey("c");
        }
        public void handle(Map message, List acc) {
            if (isDoc(message)) acc.add(String.format("Document:%s:%s", message.get("a"), message.get("c")));
        }
    }

    private class NoteHandler implements Handler {
        private boolean isNote(Map message) {
            return message.containsKey("b") || message.containsKey("c");
        }
        public void handle(Map message, List acc) {
            if (isNote(message)) acc.add(String.format("Note:%s:%s", message.get("b"), message.get("c")));
        }
    }

    private class AlertHandler implements Handler {
        private boolean isAlert(Map message) {
            return message.containsKey("a") || message.containsKey("b");
        }
        public void handle(Map message, List acc) {
            if (isAlert(message)) acc.add(String.format("Alert:%s:%s", message.get("a"), message.get("b")));
        }
    }

    private List<Handler> handlers() {
        return Arrays.asList(new DocHandler(), new NoteHandler(), new AlertHandler());
    }

    public List onMessage(Map message) {
        List result = new LinkedList();
        for (Handler handler : handlers()) {
            handler.handle(message, result);
        }
        return result;
    }
}

In this version we eliminated ugly if-series, and group together decision making logic and message handling. From that perspective the code became cleaner, but not necessarily clearer. Now it actually takes more effort to understand what the code is doing. Also, the duplication inside the is- methods is still there. We can fix it by extracting it to some abstract class or utility method. We can also use Java reflection within handlers() method to build a collection of handlers without explicitly specifying them. All these manipulations arguably make the code cleaner, but... one thing we'll never be able to fix is the separation between decision making logic and message handling. Whatever you do, there will always be the if-statement that checks if you need to process the message, and the message processing logic itself. Those two things will always be separate. Here is the point where we hit the language constraints.

Clojure

Now let's try to solve the same problem in Lisp and see if we can fix the language to eliminate the last issue from the paragraph above. Here is the direct translation of the previous Java snippet to Clojure dialect of Lisp

(defn- doc-handler [msg]
  (let [a (msg :a), c (msg :c)]
    (when (or a c)
      (format "Document:%s:%s" a c))))

(defn- note-handler [msg]
  (let [b (msg :b), c (msg :c)]
    (when (or b c)
      (format "Note:%s:%s" b c))))

(defn- alert-handler [msg]
  (let [a (msg :a), b (msg :b)]
    (when (or a b)
      (format "Alert:%s:%s" a b))))

(defn- handlers []
  [doc-handler note-handler alert-handler])

(defn on-message [msg]
  (letfn [(handle [acc h]
            (if-let [res (h msg)]
              (conj acc res)
              acc))]
    (reduce handle [] (handlers))))

This code is already easier to read, but we can do even better. The separation between decision making logic and message handling is still there. At this point we should ask the question: what kind of code do we want to see there? And my answer is: I want to replace the -handler methods above with the following code

(handler doc [a c]
  (format "Document:%s:%s" a c))

(handler note [b c]
  (format "Note:%s:%s" b c))

(handler alert [a b]
  (format "Alert:%s:%s" a b))

You see: no conditionals. Handlers are self-sufficient entities which know when they have to be applied and how. In their signatures they explicitly declare what kind of parameters they expect, and in the body they just use those parameters. No boilerplate, clean and simple. The beauty of Lisp is that you can actually implement that code. The way you do it is by creating a macro which will generate the appropriate functions. Creating a macro is not a simple task, I spent quite some time to get this one working, but it's worth of doing, because now the code is clean and clear.

We can make one additional step further by moving the handler declarations inside the handlers() method. (We need one small macro for that.) And here is the final solution

(defmacro handler [name args & body]
  `(fn [~'msg]
     (let [~@(interleave args (map (fn [x] `(get ~'msg ~(keyword x))) args))]
       (when (or ~@args)
         ~@body))))

(defmacro build-handlers [& body]
  `(defn- handlers []
     [~@body]))

(build-handlers

  (handler doc [a c]
    (format "Document:%s:%s" a c))

  (handler note [b c]
    (format "Note:%s:%s" b c))

  (handler alert [a b]
    (format "Alert:%s:%s" a b)))

(defn on-message [msg]
  (letfn [(handle [acc h]
            (if-let [res (h msg)]
              (conj acc res)
              acc))]
    (reduce handle [] (handlers))))

As I said, the first macro might be cryptic, but look at the highlighted part. This is the essence of our problem, and it cannot be done any simpler. Suppose, we need to implement a new handler which should be called if key c is present in the message. Here what we would need to add to build-handler's body to implement this new requirement

  (handler new [c]
    ...)

Simple, right? And what if a new key is added to the message which should be processed by document handler? Here is what we need to change

  (handler doc [a c d]
    ...)

We just add a new key to the function's parameter list. That's it — one-word change.

Summary

Lisp is the most powerful programming language. By that I mean you can change the language in such a way that the solution to any particular problem can be expressed in the simplest possible way. By changing the language, you can remove all the barriers between the language and the problem domain. I hope I demonstrated this in my simple example.

Resources

Clojure source code for this blog, along with the unit tests.

PyCon Canada 2012

This weekend I attended PyCon Canada, the first conference in Canada dedicated to Python ecosystem. As you might find from my blog, I'm not a Python guy. I've been using Python mostly as a scripting language. I went to this conference for fresh ideas, or, as Michael Feathers said, for cross-polination from Python community. This blog post is not a detailed review of the conference — I just want to share my impressions in general.

Organization

Considering how little time the organizers had for preparing this conference, 5 months I believe, they did amazing job. They invited great speakers. They kept people well informed using mailing list and Twitter. The official web site was clear and easy to navigate. The location was good. The food was decent. The only complaint I had is about the temperature in the rooms on the first day. It was so freezing cold inside that I had to wear my jacket all the time. But on the second day the problem was fixed.

Keynotes

Keynotes were absolutely fantastic. There were three of them. Jessica McKellar was talking about Python community. How they foster it, how they attract new people to programming in general and to Python in particular. She shared her experience from organizing Boston Python user group, the biggest Python user group in the world. The takeaway from her talk: Python community is big, welcoming, and well supported by Python Foundation.

Second keynote was Michael Feathers' Why You Should(n't) be Using a Functional Programming Language Instead. The main idea of his talk is: Don't lock yourself inside one language. Go outside of your community to see what other languages exist out there, how they solved the problems. Study those languages, learn their idioms and techniques, and then go back to your language and start using the ideas you've learnt. I completely agree with that, and that's why I went to this conference in the first place. He gave bunch of examples of functional programming in Haskell. Then he showed his Ruby code written in functional style, where you could see the influence of Haskell. I liked his presentation because he verbalized the ideas I myself have been thinking about for a while. When I started programming in Groovy my Groovy code was basically a Java code without semicolons. Now my Java code looks more like Groovy.

The closing keynote was by Fernando Pérez, the scientist from University of California, Berkeley, and the creator of IPython. The talk, titled Science and Python, was really mind blowing. When I was a student I did all my computations using mainly Fortran and some proprietary software I don't even remember the name of. Later, I played with Mathematica and Octave a little bit. But I didn't know that you can do very sophisticated scientific calculations using Python. Fernando gave some examples from neuroscience, astrophysics and biology, and it's really impressive. The discovery of Supernova PTF11kyl is especially astonishing. From now on, if I need to do some math, I'll be using Python libraries; no more proprietary expensive software. Another theme of the presentation was IPython. Initially I thought it's just a shell on top of standard Python, but it's actually the whole ecosystem. I cannot explain in few words how amazing it is. Just google for "ipython notebook" or read Fernando's blog.

Talks

As it happens on every conference, there were some great talks and some lousy talks, interesting talks and boring talks, geeky talks and academic talks. It's all normal and fine. The good thing about this conference though is that signal-noise ratio was pretty high; congratulations to the organizers for choosing talks. Another thing I like is the diversity of formats. There were 45-min presentation, 20-min talks, 5-min lightning talks, 90-min tutorials, and 3-hour workshop (there are also two full day coding sessions but I'm not attending them). This is a really good approach. Switching between different formats during the day helps your brain functioning more productive, in my opinion.

Pleasant discoveries

I found many projects presented at the conference are using RabbitMQ, and that's great. I wish in Java world people would use AMQP more frequently instead of blindly choosing JMS for every new project.

Many people are using MongoDB properly. Nowadays NoSQL is a very popular buzzword, and many projects are using various NoSQL databases for no other reason but fashion, even if it makes no sense for the project at all. It was nice to see that there are developers out there who do their homework and adopt NoSQL because it fits their domain.

Unpleasant discoveries

There seems to be a trend in Python community to despise Java. I actually see this trend in many communities outside Java, so it's not Python specific, but at this conference I've heard too many jokes about Java so it's not funny anymore, especially hearing them from the people who don't write a line in Java.

Another thing surprised me is the fanatic admiration of Mercurial and hate of Git from some Python programmers. I know lots of people who hate Git, mainly because they are confused and scared by Git. But dislike it for the reason not being written in Python is something new to me.

Problems in Python

Package and distribution management in Python is in pretty bad shape. Every person I talked to admitted that it's complete mess at the moment. I myself feel that pain every time I need to install a new library. Which tool should I use: pip, easy_install, pysetup? Some libraries installed using those tools don't work, or work partially. Many programmers use rpm or deb packages instead of Python tools, because OS packages usually work. I came to the same conclusion on my Mac OS. The only flawlessly working Python environment I have is that installed via mac ports. In Java we don't have those problems. Maven solved it once and for all long time ago. Now every JVM language benefits from it. Python community should clean up this mess and standardize their tools. I was told that with introducing PyPi and PEPs the situation is getting better, well, let's see if it resolves all the issues.

What I've learnt

Here is the list of things I found pretty interesting, in no particular order.

Python libraries to use

numpy, matplotlib, pandas, scipy, sympy, quantities, collections. Thanks to the people who told me about these libraries.

Cool Python stuff

RunSnakeRun — GUI for Python profiler. Check out the screenshots on their web site. I wish Java profilers could draw such nice graphs.

bpython — Python REPL for geeks written in Urwid. Thanks to Ian Ward for really nice presentation.

Interesting ideas

Print log statements in JSON format so that you can analyze them using powerful tools. You can also save logs in MongoDB, either offline or asynchronously, and do statistic analysis using MapReduce.

Write stored procedures in PostgreSQL in Python (and some other languages). They look much better in Python than plSQL.

Things to learn

Here are some technology and tools that have a great potential, in my opinion, and worthy of learning: ZeroMQ, IPython, OpenStack. Those were mentioned multiple times during the conference, and I need to check them out in more details.

Summary

The conference was great. I'm glad I attended it. The organizers did a great job. The conference was beneficial not only to Python community but to Toronto programming community in general. Thanks to all who made it happen.

P.S. Videos from the conference are available here.

Simple web application in Clojure

This blog post is a micro-tutorial on how to build a simple web application in Clojure. The reason I call it micro will be clear when I introduce the framework we are going to use. This tutorial will be interesting to programmers relatively new to Clojure, but who have some experience with web frameworks in other languages, for instance Spring MVC. The goal of this tutorial is to help you get started with web development in Clojure. Also I want to share my approach to web development in general and in Clojure in particular. This approach is by no means a paragon of web development, but because I like to watch how other people write the software, I thought somebody might be interested to see how I do it.

Problem

So what are we going to develop? I don't want to build a simplistic web application for the sake of building the application. On the other hand, I want to constrain myself to small feature set to prevent this tutorial from sinking in too many details. After thinking a while I found a problem which looks pretty simple, but at the same time there is a good chance people (including myself in the first place) might actually use the program I'm going to create. And here is the problem.

I have a bunch of articles and e-books sitting in some directory on my home server. To be able to read those books from any computer in my home network, I run the simple Python web-server, which exposes the content of the directory via HTTP. If you are curious, here is the command I'm using:

$ cd /path/to/your/ebook/dir
$ python -m SimpleHTTPServer 3030

And here is how the "library" looks like in the browser

This library application is good enough for me, mainly because it's functional. I can easily find the book by skimming the page or using Find command in a browser. But for the purpose of this tutorial I want to make it slightly better. For example, I can split the file names and display the books in a table view, where I can see clearly what the name of the book is, who the author is, and when it was pablished. I can even add sorting as a bonus feature. Basically, I want something like this

As you can imagine, it shouldn't be hard to do this. The file names are already in the form that is easy to parse. So the question is really how to show the table data in a browser. Simple problem, minimum requirements. Let's see how to solve it in Clojure.

Tools

Clojure, being a Lisp descendant, is a powerful language. That means you can create your own web framework during a weekend, which many people actually do. But I think it's much better to take existing library, promote it, enhance it, fix the bugs if you like it. In Clojure I found such a framework, it's called Noir. This framework is very small, so small that their developers call it micro-framework, that's why I'm calling this tutorial micro-tutorial. Probably we shouldn't even call it framework at all, library would probably be a better name. The closest analog to Noir in other languages I know is probably Webmachine in Erlang, or Spring MVC in Java. I wouldn't compare it to Grails or Rails because those guys are huge.

Noir is not only small, it's also simple. You can look at their source code and understand how it works without any problem, provided you have some experience with Clojure. As a result, Noir is a perfect tool for the problem we are going to solve.

Without further ado let's see how Noir works. The easiest way to set up a scaffolding of our future application is by using Leiningen. Leiningen is a Clojure build tool, very similar to Maven. In Maven you would do mvn archetype:generate, in Leiningen you run

$ lein new noir bookshelf

where bookshelf is the name of our application. This creates a directory called bookshelf where you can find some Noir template files, plus Leiningen project descriptor

/.gitignore
/project.clj
/README.md
/resources/public/css/
/resources/public/css/reset.css
/resources/public/img/
/resources/public/js/
/src/bookshelf/models/
/src/bookshelf/server.clj
/src/bookshelf/views/common.clj
/src/bookshelf/views/welcome.clj
/test/bookshelf/

You can ignore .gitignore, it's already configured properly. Before we make our initial checkin, it's a good practice to edit project.clj and README.md to replace FIXME's. We will edit README file again later when we finish the development to provide more information on how to use the application. Before we move to Noir, let's quickly review project.clj

(defproject bookshelf "0.1.0-SNAPSHOT"
  :description "Bookshelf site"
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [noir "1.3.0-beta3"]]
  :main bookshelf.server)

project.clj is a Clojure version of pom.xml (in fact, you can use pom.xml if you want, Clojure perfectly understands it). First two lines are obvious. Dependency entry specifies which JAR files we need to make our application work. In our case we need only two JARs. Leiningen will check Maven central repository as well as Clojars to download the required JARs with all transitive dependencies. The last line in the project descriptor says which namespace contains the main method. In our case it is bookshelf.server. You can find the source of this namespace in /src/bookshelf/server.clj file. Let's take a look at this file

(ns bookshelf.server
  (:require [noir.server :as server]))

(server/load-views-ns 'bookshelf.views)

(defn -main [& m]
  (let [mode (keyword (or (first m) :dev))
        port (Integer. (get (System/getenv) "PORT" "8080"))]
    (server/start port {:mode mode
                        :ns 'bookshelf})))

The third line specifies the prefixes of the namespaces that will be loaded by Noir server. By default Leiningen generates two files satisfying this criterion: bookshelf/views/common.clj and bookshelf/views/welcome.clj, but you can create more if your project becomes more complex. Since Noir scans namespaces by prefixes, you can even put your files in the nested directories under bookshelf/views, no changes in server.clj required. To start Noir server, run

$ lein run

This will start Jetty web server bound to localhost at port 8080. If you want to change the default port, say to 3030, run the following command

$ export PORT=3030; lein run

Besides port, you can specify few other parameters such as :mode, :jetty-options, etc. (you can see all available options in the server source). I'll show below how to specify production mode, for example, when we deploy the final application.

If you started the server with the default port, open your browser at http://localhost:8080. You should see the Noir's start page

This page by itself contains all you need to get started with Noir, so you can safely stop reading this blog, and just follow the instructions on that page. Those who continue reading this tutorial and wondering where that start page is coming from, please open /src/bookshelf/views/welcome.clj

(ns bookshelf.views.welcome
  (:require [bookshelf.views.common :as common]
            [noir.content.getting-started])
  (:use [noir.core :only [defpage]]))

(defpage "/welcome" []
  (common/layout
    [:p "Welcome to bookshelf"]))

Third line tells us that the source of the start page is in noir/content/getting-started file. If you are curious where this file is, look here. Search for (defpage "/" [] …) to see how the start page is defined. On your web-site you probably want the start page to be different, so you can remove [noir.content.getting-started] from :require section.

The next thing to notice on the snippet above is (defpage "/welcome" [] …) function. That's how you define URL mappings (or routes, in Noir lingo) of your web-site. (Internally, Noir uses Compojure library to handle the routing.) It is similar to @RequestMapping annotations in Spring-MVC, where you specify which method is called when a user hits the given URL. As you can see, we have only one mapping at the moment, /welcome. Since we are building bookshelf application, let's rename it to /books. Also, to be even more explicit, let's rename the whole file to books.clj. Don't forget to update the namespace. Your books.clj file should now look like this

(ns bookshelf.views.books
  (:require [bookshelf.views.common :as common])
  (:use [noir.core :only [defpage]]))

(defpage "/books" []
  (common/layout
    [:p "Welcome to bookshelf"]))

If you go to http://localhost:8080/books in your browser, you should see this screen

By looking at the source of this page, you find it a proper HTML with head and body elements. Those are generated by Hiccup library, which we'll discuss in a moment. One thing I want to mention about defpage is that you can get the same result if you change /books route definition as follows

(defpage "/books" []
  "<html>
     <head>
       <title>bookshelf</title>
     </head>
     <body>
       <p>Welcome to bookshelf</p>
     </body>
   </html>")

It is just a theoretical exercise, in reality nobody hard-codes the entire HTML inside the Clojure code.

Now let's take look at how the page content is generated. If you look at the routing definition in books.clj, you see that the body of defpage function is a call to layout function defined in /src/bookshelf/views/common.clj. Let's open this file

(ns bookshelf.views.common
  (:use [noir.core :only [defpartial]]
        [hiccup.page :only [include-css html5]]))

(defpartial layout [& content]
  (html5
    [:head
     [:title "bookshelf"]
     (include-css "/css/reset.css")]
    [:body
     [:div#wrapper
      content]]))

defpartial is just a wrapper on top of hiccup.core/html function. Hiccup is an XML/HTML rendering library in Clojure. The idea behind it is pretty simple: You build a tree using Clojure vectors, and Hiccup transforms it to a valid HTML. If you are familiar with Groovy MarkupBuilder, it's the same idea. For example, let's define a couple of trees: head and body

(def head
  [:head
   [:title "bookshelf"]])
(def body
  [:body
   [:div
    [:p "Welcome to bookshelf"]]])

Here is what you see in REPL when it evaluates different Hiccup HTML formats (I pretty formatted them for visibility purposes)

(hiccup.page/html5 head body)
;=> "<!DOCTYPE html>
<html>
  <head>
    <title>bookshelf</title>
  </head>
  <body>
    <div><p>Welcome to bookshelf</p></div>
  </body>
</html>"

(hiccup.page/html4 head body)
;=> "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">
<html>
  <head>
    <title>bookshelf</title>
  </head>
  <body>
    <div><p>Welcome to bookshelf</p></div>
  </body>
</html>"

(hiccup.page/xhtml head body)
;=> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">
<html xmlns=\"http://www.w3.org/1999/xhtml\">
  <head>
    <title>bookshelf</title>
  </head>
  <body>
    <div><p>Welcome to bookshelf</p></div>
  </body>
</html>"

Noir's default format is html5, the first example above. You can change it to any other format if needed.

The last piece of common.clj I want to mention is (include-css "/css/reset.css"). This is another function from Hiccup library. It generates <link href="/css/reset.css" rel="stylesheet" type="text/css"> element and inserts it in the head element of the page. If you recall the scaffolding we generated at the beginning, there is a directory called /resources/public where Noir keeps CSS files, JavaScript, and images required by your web-site. By default Noir creates reset.css in the corresponding subdirectory. Later we'll create other stylesheets and update common.clj appropriately.

Now, after we covered all the basics, we are ready to build our application.

Controller and View

Let's build our view and controller first. While doing that we'll figure out what data we need from the back-end. That's called top-down design.

We define controllers and views in books.clj file. For now I also include the model in this file. We'll move it to a proper namespace later when we are done with the front-end. Here is the new version of books.clj

(ns bookshelf.views.books
  (:require [bookshelf.views.common :as common])
  (:use [noir.core :only [defpage]]
        [hiccup.element :only [link-to]]))

(defn books []
  [{:author "Fogus M., Houser C."
    :title "The Joy Of Clojure"
    :year "2011"
    :format "pdf"
    :id 1}
   {:author "Fogus M., Houser C."
    :title "The Joy Of Clojure"
    :year "2011"
    :format "epub"
    :id 2}])

(defn- list-books []
  [:table
   [:thead
    [:tr
     [:th "Author"]
     [:th "Title"]
     [:th "Published"]
     [:th "Format"]]]
   (into [:tbody]
         (for [book (books)]
           [:tr
            [:td (:author book)]
            [:td (:title book)]
            [:td (:year book)]
            [:td (link-to (clojure.string/join "/" ["/books" (:id book) (:format book)])
                          (:format book))]]))])

(defpage "/books" []
  (common/layout
    (list-books)))

There are three functions in this file: books (model), list-books (view), and /books (controller/router). The controller is essentially the same as before and the model is simply a vector of maps. Each map contains the same keys as column names on the book table. View might look complicated, but there shouldn't be any surprise here either — it's again an ordinary tree. The only new element here is link-to function defined in hiccup.element namespace. We could really build it directly using [:a {:href …}] vector, but link-to is a standard way to do it in Hiccup.

If you refresh http://localhost:8080/books now (Leiningen/Jetty supports hot redeployment), you will see this screen

The important part of the view function is the format of the links. For the current book model the links are http://localhost:8080/books/1/pdf and http://localhost:8080/books/2/epub. You can verify it by hovering the mouse over them. Those links are actually the main goal of our application. By clicking the link I want to load (or download) the book into the browser and read it. To implement this feature add the following functions to books.clj

(defn get-file [id]
  nil)

(defn- ctype [format]
  (if (= "pdf" format) "application/pdf" "text/plain"))

(defpage "/books/:id/:format" {:keys [id format]}
  (content-type (ctype format) (java.io.ByteArrayInputStream. (get-file id))))

Ignore for a moment get-file function — it does nothing. Later we will move it to model namespace and implement it properly, but we need some placeholder now to compile the application. ctype is a helper function that maps the file format to HTTP content type. I have three ebook formats in my library: ePub, Mobi, and PDF. The first two are plain text formats from HTTP content type perspective. Only PDF requires special type.

The interesting part of the snippet is the controller definition. If you ever implemented REST in Spring MVC, you should see the familiar pattern here. Like Spring, Noir (via Compojure) supports path variables. If you click on http://localhost:8080/books/1/pdf link in the browser, Noir calls the corresponding controller and binds id variable to "1" and format variable to "pdf". When we implement get-file function, it should return the file with the given id as an array of bytes. Controller then wraps the array into a stream and Noir pushes it to HTTP response. content-type function is defined in noir.response namespace, so we need to add [noir.response :only [content-type]] to :use section at the top of books.clj.

That's it, in terms of functionality the controller and the view of our application are done.

Model

Now we need to extract the logic that creates a model from presentation tier to middle-tier. In Noir that's what models directory is for. In our case it's /src/bookshelf/models. Let's create a file called db.clj in that directory, and move there books and get-file functions from bookshelf.views.books namespace. We have to update books.clj to load the new namespace. It should now look like this

(ns bookshelf.views.books
  (:require [bookshelf.models.db :as db]
            [bookshelf.views.common :as common])
  (:use [noir.core :only [defpage]]
        [noir.response :only [content-type]]
        [hiccup.element :only [link-to]]))

(defn- list-books []
  [:table
   [:thead
    [:tr
     [:th "Author"]
     [:th "Title"]
     [:th "Published"]
     [:th "Format"]]]
   (into [:tbody]
         (for [book (db/books)]
           [:tr
            [:td (:author book)]
            [:td (:title book)]
            [:td (:year book)]
            [:td (link-to (clojure.string/join "/" ["/books" (:id book) (:format book)])
                          (:format book))]]))])

(defpage "/books" []
  (common/layout
    (list-books)))

(defn- ctype [format]
  (if (= "pdf" format) "application/pdf" "text/plain"))

(defpage "/books/:id/:format" {:keys [id format]}
  (content-type (ctype format) (java.io.ByteArrayInputStream. (db/get-file id))))

If you refresh your browser, nothing should change.

All right, now it's time to implement our model properly. Recall that our model should scan the directory it's running in for the files with the format Author-Title-Year.FileFormat, and convert each of those files to byte array. Here is how I implemented it

(ns bookshelf.models.db
  (:use [clojure.java.shell :only (sh)]))

(defn- list-files [dir]
  (clojure.string/split (:out (sh "ls" dir)) #"\n"))

(defn- parse [file]
  (when-let [match (re-matches #"([^-]+)-([^-]+)-(\d+)\.(\S+)" file)]
    (zipmap [:file :author :title :year :format] match)))

(defn- add-id [book]
  (assoc book :id (clojure.string/replace (:file book) #"[., ]" "")))

(defn books []
  (->> (list-files ".") (map parse) (remove nil?) (map add-id)))

(defn- file [id]
  (let [mapping (into {} (for [b (books)] [(:id b) (:file b)]))]
    (get mapping id)))

(defn get-file [id]
  (with-open [input  (java.io.FileInputStream. (file id))
              buffer (java.io.ByteArrayOutputStream.)]
    (clojure.java.io/copy input buffer)
    (.toByteArray buffer)))

Let's take a look what each of the functions does. Function list-files returns a vector of file names that reside in the given directory dir. To find all files in the directory I'm using clojure.java.shell/sh function which executes ls command. This works fine on Linux and Mac, but I'm not sure about Windows. Function parse checks if the given file name has the required format. If so, it returns a map {:file file, :author Author, :title Title, :year Year, :format FileFormat}, otherwise it returns nil. add-id function removes dots, commas, and spaces from the file name and add the result as a book ID to the book map. Function books is just a composition of those three functions, and it returns the result expected by the view.

Function file returns the file by given ID. The implementation above is not efficient, but my library is too small to notice any performance issues. Finally, get-file function finds the file by ID and returns it as a byte array. Those four lines is a pretty standard idiom which you can find in many Clojure source files.

Now we are ready to test our application. For testing purposes I'm going to copy a couple of e-books I recently received updates for to the project home directory. The content of this directory looks like this

.gitignore
README.md
Thomas D.-Programming Ruby 1.9-2010.epub
Thomas D.-Programming Ruby 1.9-2010.pdf
project.clj
resources
src
test

I refresh my browser and here I can see these two books

If I click on pdf, I can read the book in my browser

OK, the application is functional. The next step is to make it little bit prettier.

Styling

Since our application is written in Noir framework, let's make it look like Noir. First, I download Noir background image and save it to /resources/public/img directory. Second, I create a stylesheet /resources/public/css/noir.css which resembles Noir's original

body {
    background: #2a2b2b;
    color: #d1d9e1;
    background: url('/img/bg.png');
    padding: 60px 60px;
    font-family: 'Helvetica Neue',Helvetica,Verdana;
}
a {
    text-decoration: none;
    color: #d1d9e1;
}
a:hover {
    color: #6bffbd;
}
h1 {
    color: #6bffbd;
}

Then I update bookshelf.views.common namespace to include new CSS

(defpartial layout [& content]
  (html5
    [:head
     [:title "Bookshelf"]
     (include-css "/css/reset.css")
     (include-css "/css/noir.css")]
    [:body
     [:div#wrapper
      content]]))

Finally, I want to add a header to the page in bookshelf.views.books

(defpage "/books" []
  (common/layout
    [:h1 "Books"]
    (list-books)))

Refresh books web page on the browser to see the changes

The last thing left unstyled is the book table. I won't style it directly, because I want to add client-side sorting to it, and I happen to know that TableSorter JavaScript library provides its own style.

JavaScript

TableSorter is a jQuery plugin, so you need to download jQuery first. Grab the latest min and save it to /resources/public/js directory. Then, download tablesorter.zip that contains both JavaScript and stylesheet files. As before, JavaScript goes to /resources/public/js and stylesheets go to /resources/public/css directory. Here is the resources directory structure I have after everything is saved

resources/public/css/tablesorter/asc.gif
resources/public/css/tablesorter/bg.gif
resources/public/css/tablesorter/desc.gif
resources/public/css/tablesorter/style.css
resources/public/js/jquery-1.8.2.min.js
resources/public/js/jquery.tablesorter.js

If you are curious, here is my style.css. I changed the original tablesorter css a little bit to better fit Noir theme

table.tablesorter {
    margin:10px 0pt 15px;
    font-size: 10pt;
    width: 100%;
    text-align: left;
}
table.tablesorter thead tr th, table.tablesorter tfoot tr th {
    color: #000000;
    background-color: #b0b8c0;
    border: 1px solid #2a2b2b;
    font-size: 10pt;
    padding: 4px;
}
table.tablesorter thead tr .header {
    background-image: url(bg.gif);
    background-repeat: no-repeat;
    background-position: center right;
    cursor: pointer;
}
table.tablesorter tbody td {
    color: #b0b8c0;
    padding: 4px;
    background-image: url('/img/bg.png');
    vertical-align: top;
}
table.tablesorter tbody tr.odd td {
    background-color:#F0F0F6;
}
table.tablesorter thead tr .headerSortUp {
    background-image: url(asc.gif);
}
table.tablesorter thead tr .headerSortDown {
    background-image: url(desc.gif);
}
table.tablesorter thead tr .headerSortDown, table.tablesorter thead tr .headerSortUp {
    background-color: #6bffbd;
}

To enable tablesorter we have to update both view files. Few changes in bookshelf.views.common

(ns bookshelf.views.common
  (:use [noir.core :only [defpartial]]
        [hiccup.page :only [include-css include-js html5]]))

(defpartial layout [& content]
  (html5
    [:head
     [:title "Bookshelf"]
     (include-js "/js/jquery-1.8.2.min.js")
     (include-js "/js/jquery.tablesorter.js")
     (include-css "/css/reset.css")
     (include-css "/css/tablesorter/style.css")
     (include-css "/css/noir.css")]
    [:body
     [:div#wrapper
      content]]))

and few changes in bookshelf.views.books

(ns bookshelf.views.books
  (:require [bookshelf.models.db :as db]
            [bookshelf.views.common :as common])
  (:use [noir.core :only [defpage]]
        [noir.response :only [content-type]]
        [hiccup.element :only [link-to javascript-tag]]))

(defn- list-books []
  [:table.tablesorter {:id "bookTable"}
   [:thead
    [:tr
     [:th "Author"]
     [:th "Title"]
     [:th "Published"]
     [:th "Format"]]]
   (into [:tbody]
         (for [book (db/books)]
           [:tr
            [:td (:author book)]
            [:td (:title book)]
            [:td (:year book)]
            [:td (link-to (clojure.string/join "/" ["/books" (:id book) (:format book)])
                          (:format book))]]))])

(defpage "/books" []
  (common/layout
    [:h1 "Books"]
    (javascript-tag "$(document).ready(function() {$(\"#bookTable\").tablesorter();});")
    (list-books)))

(defn- ctype [format]
  (if (= "pdf" format) "application/pdf" "text/plain"))

(defpage "/books/:id/:format" {:keys [id format]}
  (content-type (ctype format) (java.io.ByteArrayInputStream. (db/get-file id))))

After we refresh the browser, we should see the final design and be able to sort the table

Ship it!

OK, we are ready to ship. But before we build a deployable artifact, we, as professional developers, should update documentation (README.md in our case) and finalize the version of the application (project.clj)

(defproject bookshelf "0.1.0"
  :description "Bookshelf site"
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [noir "1.3.0-beta3"]]
  :main bookshelf.server)

I want to run this application as a a standalone Java web application, without any dependency on Leiningen. Therefore I have to add :gen-class to server.clj

(ns bookshelf.server
  (:require [noir.server :as server])
  (:gen-class))

(server/load-views-ns 'bookshelf.views)

(defn -main [& m]
  (let [mode (keyword (or (first m) :dev))
        port (Integer. (get (System/getenv) "PORT" "8080"))]
    (server/start port {:mode mode
                        :ns 'bookshelf})))

Now package the application by running the following command

$ lein uberjar

As a result of this command, bookshelf-0.1.0-standalone.jar artifact is created in the target directory. I scp this file to my server, to the directory where my books are located, and start the app

$ export PORT=3030; nohup java -jar bookshelf-0.1.0-standalone.jar prod > nohup.out 2>&1 &

And that's basically it. We have created a simplest web application in Clojure, which might be useful by its own. But more importantly, you've learned how to build it. I hope you enjoyed reading this tutorial as I enjoyed writing it.

Recap

If you want to create web application in Clojure, try Noir. Noir is small and easy to pick up.

• use defpage macro to define URL routes
• use defpartial macro to build views
• use Leiningen to run local web server
• use REPL to experiment with business logic
• have fun with Clojure

Resources

1. Long in-depth Noir tutorial: http://yogthos.net/blog/22
2. Source code of this tutorial: https://github.com/ndpar/bookshelf
3. Noir: http://www.webnoir.org
4. Hiccup: https://github.com/weavejester/hiccup

P.S. After I wrote the draft of this post I found another tutorial on Clojure web development, which even has the same name for the application! If I knew about it before, I wouldn't probably write mine. But since it's already typed, let it be published.