Side Notes

Cygwin git-svn messed up

2011-11-23T13:09:00.010-05:00

I upgraded my Cygwin from version 1.5 to 1.7 (finally), and found that git-svn command was broken

$ git svn rebase
Can't locate SVN/Core.pm in @INC (@INC contains:
/usr/lib/perl5/site_perl/5.10
/usr/lib/perl5/site_perl/5.14/i686-cygwin
/usr/lib/perl5/site_perl/5.14
/usr/lib/perl5/vendor_perl/5.14/i686-cygwin
/usr/lib/perl5/vendor_perl/5.14
/usr/lib/perl5/5.14/i686-cygwin
/usr/lib/perl5/5.14
/usr/lib/perl5/vendor_perl/5.10
/usr/lib/perl5/site_perl/5.8 .) at /usr/lib/git-core/git-svn line 42.

Usually this error indicates that subversion-perl package is not installed, but that was not the case as we can see it from cygcheck output:

$ cygcheck -c subversion-perl
Cygwin Package Information
Package              Version        Status
subversion-perl      1.7.1-1        OK

As it turned out, the problem was in the package dependency management. From the error above we saw that perl was looking for SVN/Core.pm in /usr/lib/perl5/vendor_perl/5.14/i686-cygwin directory. But the latest subversion-perl installed it in the different place

$ cygcheck -l subversion-perl
...
/usr/lib/perl5/vendor_perl/5.10/i686-cygwin/SVN/Core.pm
...

The problem was solved by downgrading perl from 5.14.x-x to 5.10.x-x.

If you have the same problem, check which version of perl package you have installed

$ cygcheck -f /usr/bin/perl
perl-5.10.1-5

It must be the same as the version used by subversion-perl.

Ford Marbles

2011-11-05T14:19:00.008-04:00

I found these marvelous renderings of Ford circles on flickr. I can't help but share them here.

As Thomae's function, Ford circles is another visual representation of rational numbers. You can investigate them here with interactive Wolfram demo.

Modulo who?

2011-09-16T12:59:00.004-04:00

When programmer and mathematician are talking about modulus or modulo, there is often a confusion what this term means. For programmer modulo means an operator that finds the remainder of division of one number by another, e.g. 5 mod 2 = 1. For mathematician modulo is a congruence relation between two numbers: a and b are said to be congruent modulo n, written a ≡ b (mod n), if their difference a − b is an integer multiple of n.

These two definitions are not equivalent. The former is a special case of the latter: if b mod n = a then a ≡ b (mod n). The inverse is not true in general case. 5 mod 2 = 1, and 1 ≡ 5 (mod 2) because 1 - 5 = -4 is integer multiple of 2. Now 5 ≡ 1 (mod 2) because 5 - 1=4 is evenly divisible by 2, but 1 mod 2 = 1, not 5.

The biggest confusion happens when programmer and mathematician start arguing about Gauss' famous golden theorem where both definitions of modulus can be used.

Thomae's function

2011-08-06T09:04:00.005-04:00

Thomae's function (a.k.a. Riemann function) is defined on the interval (0, 1) as follows

Here is the graph of this function with some points highlighted as plus symbols for better view.

This function has interesting property: it's continuous at all irrational numbers. It's easy to see this if you notice that for any positive ε there is finite number of points above the line y = ε. That means for any irrational number x₀ you can always construct a δ-neighbourhood that doesn't contain any point from the area above the line y = ε.

To generate the data file with point coordinates I used Common Lisp program:

(defun rational-numbers (max-denominator)
  (let ((result (list)))
    (loop for q from 2 to max-denominator do
      (loop for p from 1 to (1- q) do
        (pushnew (/ p q) result)))
    result))
 
(defun thomae-rational-points (abscissae)
  (mapcar (lambda (x) (list x (/ 1 (denominator x)))) abscissae))
 
(defun thomae (max-denominator)
  (let ((points (thomae-rational-points (rational-numbers max-denominator))))
    (with-open-file (stream "thomae.dat" :direction :output)
      (loop for point in points do
        (format stream "~4$ ~4$~%" (first point) (second point))))))
 
(thomae 500)

To create the images I used gnuplot commands:

plot "thomae.dat" using 1:2 with dots
plot "thomae.dat" using 1:2 with points

and Photoshop.

Rod Johnson on Entrepreneurialism

2011-06-30T18:35:00.002-04:00

One thing I think that you really need to be careful of as well, particularly if you, like me, are a programmer, is don’t get carried away writing code. Typically in my experience anyone who is a good programmer is pretty passionate about it, love writing code, get addicted to the process of writing code, fell pretty good about their code basis. As soon as you get down that path you are not thinking straight anymore and now you are increasing your emotional investment, you are having lots of fun writing interesting code and you are no longer in a place mentally where you are going to be trying to find some reason that you shouldn’t write that code. That has been a big lesson for me that the quicker I get to coding, the longer it takes me to ask the kind of questions I should ask upfront.
...
It is really, really hard to decide not to do things. One of the biggest killers of companies is trying to do too much. If you try to take on too many things you will assuredly fail, even if every one of those things is a good thing to do. It is incredibly hard to realize that a particular thing is a good idea, but you are not going to do it.
...
I think the biggest way to decide frankly if you are trading off business priorities is do the boring stuff like, look at the total addressable market, go and talk to customers, figure out what they will pay for. You really need to be guided by what the revenue is likely to be, and make sure you don’t just do something just because it’s cool.

Math and Physics of Benderama

2011-06-27T21:07:00.010-04:00

The last episode of Futurama has interesting formula involved. The entire plot is based on the Professor's latest invention — Banach-Tarski Dupla-Shrinker — the machine that produces two copies of any object at a 60% scale. It was just a matter of time when Bender found a good usage of this machine: to replicate himself. Then two small copies of Bender replicated themselves making four smaller copies, and so forth. At some point the Professor horrified the crew that if they don't stop this unlimited growth, the total mass of all Benders will eventually be so big that the entire Earth will be consumed during the process of replication. As a proof he demonstrated this formula of the mass of all generations of Bender

This is a perfect toy for a science geek. The first obvious question it brings: is this formula mathematically correct? As it turns out, it is not. Considering the scale of 60%, the cubic dependency of volume on linear dimension, and the constant density of all copies, the formula should be the following

As you can see the total mass of infinite number of Benders actually converges to approximately 1.76 M₀. So from Math perspective there is nothing to worry about. But what if our assumption of constant density is invalid. Would it be a problem from Physics perspective? Let's see.

Knowing that every new copy has a size of 0.6 of the original it was made from, we have the following formula for the size of Bender in the n^th generation

This exponential function becomes very small pretty soon. In the 154^th generation it already reaches the Planck length, after which the further replication is physically impossible. If we calculate the total mass of 154 Bender's generations using the Professor's formula, we get H(154) × 238 kg ≈ 1,337.56 kg, which is nothing comparing to the Earth mass.

So we have to admit that from both Math and Physics perspective the Professor was wrong, and there was no real threat to the Earth.

Although the Professor's formula doesn't describe the replication process adequately, it's still a beautiful piece of Math because it's a formula of harmonic series. If you want to know why harmonic series is beautiful and which real processes it describes, read this nice article of John H. Webb.

And don't miss the next episode of Futurama this Thursday :-)

Functional Groovy switch statement

2011-06-08T21:30:00.002-04:00

In the previous post I showed how to replace chained if-else statements in Groovy with one concise switch. It was done for the special case of if-stement where every branch was evaluated using the same condition function. Today I want to make a generalization of that technique by allowing to use different conditionals.

Suppose your code looks like this:

if (param % 2 == 0) {
    'even'
} else if (param % 3 == 0) {
    'threeven'
} else if (0 < param) {
    'positive'
} else {
    'negative'
}

As soon as every condition operates on the same parameter, you can replace the entire chain with a switch. In this scenario param becomes a switch parameter and conditions become case parameters of Closure type. The only thing we need to do is to override Closure.isCase() method as I described in the previous post. The safest way to do it is to create a category class:

class CaseCategory {
    static boolean isCase(Closure casePredicate, Object switchParameter) {
        casePredicate.call switchParameter
    }
}

Now we can replace if-statement with the following switch:

use (CaseCategory) {
    switch (param) {
        case { it % 2 == 0 } : return 'even'
        case { it % 3 == 0 } : return 'threeven'
        case { 0 < it }      : return 'positive'
        default              : return 'negative'
    }
}

We can actually go further and extract in-line closures:

def even = {
    it % 2 == 0
}
def threeven = {
    it % 3 == 0
}
def positive = {
    0 < it
}

After which the code becomes even more readable:

use (CaseCategory) {
    switch (param) {
        case even     : return 'even'
        case threeven : return 'threeven'
        case positive : return 'positive'
        default       : return 'negative'
    }
}

Nothing new under the sun

2011-06-07T12:46:00.002-04:00

Every generation of software developers needs its own fad. For my generation it was Agile, for generation before it was OOP, and before that it was another big thing. Gerald Weinberg, one of the most influential people in our industry, blogged yesterday about this issue. With over 50 years of experience in software development he knows what he is talking about. Read his blog post — he has a very good point.

P.S. I'm wondering what will be the next big thing. Will it be Cloud?

Multimethods in Groovy

2011-06-05T20:52:00.003-04:00

Every time I switch from Groovy to Java I have to remind myself that some things that seem so natural and work as expected in Groovy, don't work in Java. One of such differences is method dispatching. Groovy supports multiple dispatch, while Java does not. Therefore the following code works differently in Groovy and Java:

public class A {
    public void foo(A a) { System.out.println("A/A"); }
    public void foo(B b) { System.out.println("A/B"); }
}
public class B extends A {
    public void foo(A a) { System.out.println("B/A"); }
    public void foo(B b) { System.out.println("B/B"); }
}
public class Main {
    public static void main(String[] args) {
        A a = new A();
        A b = new B();
        a.foo(a);
        b.foo(b);
    }  
}

$ java Main
A/A
B/A

$ groovy Main.groovy
A/A
B/B

Reversing Groovy switch statement

2011-06-01T14:15:00.002-04:00

Recently I've been working on a Groovy code that had many methods with long multibranch conditionals like this:

def parse(message, options) {
    if (options.contains('A')) {
        parseARule message
    } else if (options.contains(2)) {
        parseSmallDigitRule message
    ...
    } else if (options.contains(something)) {
        parseSomeRule message
    } else {
        parseSomeOtherRule message
    }
}

Although this code is working, it is hard to see which branch is called under which condition. It would be much better if we could replace this code with something like Lisp cond macro. The best candidate for such a task in Groovy would be a switch statement. If we could only refactor the code above to something like following, it would significantly improve readability:

def parse(message, options) {
    switch (options) {
        case 'A' : return parseARule(message)
        case 2   : return parseSmallDigitRule(message)
        ...
        case ... : return parseSomeRule(message)
        default  : return parseSomeOtherRule(message)
    }
}

Unfortunately, this code doesn't work out of the box in Groovy, but it works if we do some metaprogramming.

The way switch statement works in Groovy is a bit different than in Java. Instead of equals() it uses isCase() method to match case-value and switch-value. The default implementation of isCase() method falls back to equals() method, but some classes, including Collection, override this behaviour. That's why in Groovy you can do things like this:

switch (value) {
    case ['A','E','I','O','U'] : return 'vowel'
    case 0..9                  : return 'digit'
    case Date                  : return 'date'
    default                    : return 'something else'
}

For our purposes we need some sort of reverse switch, where collection is used as a switch-value, and String and Integer are used as a case-value. To do this we need to override default implementation of isCase() method on String and Integer classes. It's not possible in Java, but is very easy in Groovy. You can change method implementation globally by replacing it in corresponding meta class, or locally with the help of categories. Let's create a category that swaps object and subject of isCase() method:

class CaseCategory {
    static boolean isCase(String string, Collection col) {
        reverseCase(string, col)
    }
    static boolean isCase(Integer integer, Collection col) {
        reverseCase(integer, col)
    }
    // Add more overloaded methods here if needed

    private static boolean reverseCase(left, right) {
        right.isCase(left)
    }
}

Now we can use this category to achieve the goal we stated at the beginning of this post:

def parse(message, options) {
    use (CaseCategory) {
        switch (options) {
            case 'A' : return parseARule(message)
            case 2   : return parseSmallDigitRule(message)
            ...
            case ... : return parseSomeRule(message)
            default  : return parseSomeOtherRule(message)
        }
    }
}

If you are comfortable with global method replacement, you can amend String and Integer meta classes. In this case you don't need to wrap switch statement with use keyword.

Anyways, with or without category, the final code looks better than the original noisy if-else chain. And you have learned the technique of reversing switch statement.

Lazy lists in Groovy

2011-02-03T22:16:00.015-05:00

I like lazy evaluation, and it's one of the reasons I like Haskell language so much. Although from engineering perspective lazy evaluation is probably not the most needed feature, it's definitely very useful for solving some mathematical problems.

Most languages don't have lazy evaluation out of the box, but you can implement it using some other language features. This is an interesting task, and I use it as a code kata which I practice every time I learn a new strict language.

So, how to implement lazy lists in strict languages? Very simple, if the language has functional capabilities. Namely, you build lazy list recursively by wrapping strict list within a function. Here is, for example, the strict empty list in Groovy:

[]

If we wrap it with a closure, it becomes lazy empty list:

{-> [] }

If we need a list with one element, we prepend (or speaking Lisp terminology 'cons') an element to lazy empty list, and make the result lazy again:

{-> [ element, {-> [] } ] }

To add more elements we continue the same process until all elements are lazily consed. Here is, for example, a lazy list with three elements a, b and c:

{-> [a, {-> [b, {-> [ c, {-> [] } ] } ] } ] }

Now, when you have an idea how to build lazy lists, let's build them Groovy way. We start with creating a class:

class LazyList {
    private Closure list

    private LazyList(list) {
        this.list = list
    }
}

The variable list encapsulates the closure wrapper of the list. We just need to expose some methods that allow constructing lists using procedure described above:

    static LazyList nil() {
        new LazyList( {-> []} )
    }

    LazyList cons(head) {
        new LazyList( {-> [head, list]} )
    }

Now we can construct lists by consing elements to empty list:

def lazylist = LazyList.nil().cons(4).cons(3).cons(2).cons(1)

To access elements of the list we implement two standard functions, car and cdr, that return head and tail of the list respectively.

    def car() {
        def lst = list.call()
        lst ? lst[0] : null
    }

    def cdr() {
        def lst = list.call()
        lst ? new LazyList(lst.tail()[0]) : nil()
    }

Here is how you use these functions to get first and second elements of the list constructed above

assert lazylist.car() == 1
assert lazylist.cdr().car() == 2

In Lisp there are built-in functions for various car and cdr compositions. For example, the previous assertion would be equivalent to function cadr. Instead of implementing all possible permutations, let's use Groovy metaprogramming to achieve the same goal.

    def methodMissing(String name, args) {
        def matcher = name =~ /c([ad])([ad]+)r/
        if (matcher) {
            matcher[0][2].reverse().toList().inject(this) {
                del, index -> del."c${index}r"()
            }."c${matcher[0][1]}r"()
        } else {
            throw new MissingMethodException(name, this.class, args)
        }
    }

It might look complicated, but in reality it's pretty simple if you are familiar with Groovy regex and functional programming. It's easier to explain by example. If we pass "caddr" as a value of name parameter, the method will create a chain on method calls .cdr().cdr().car() which will be applied to delegate of the operation which is our LazyList object.

With this method in place we can call car/cdr functions with arbitrary depth.

assert lazylist.caddr() == 3

If you create nested lazy lists, you can access any element of any nested list with this dynamic method.

def lmn = LazyList.nil().cons('N').cons('M').cons('L')
def almnz = LazyList.nil().cons('Z').cons(lmn).cons('A')
assert almnz.cadadr() == 'M'

With so many cons methods it's hard to see the structure of the list. Let's implement lazy method on ArrayList class that converts strict list to lazy. Again, we will use metaprogramming and functional techniques.

ArrayList.metaClass.lazy = {
    -> delegate.reverse().inject(LazyList.nil()) {list, item -> list.cons(item)}
}

Now we can rewrite the previous example as follows

def lazyfied = ['A', ['L','M','N'].lazy(), 'Z'].lazy()
assert lazyfied.cadadr() == 'M'

What have we accomplished so far? We learned how to build lazy lists from scratch and from strict lists. We know how to add elements to lazy lists, and how to access them. The next step is to implement fold function. fold is the fundamental operation in functional languages, so our lazy lists must provide it.

    boolean isEmpty() {
        list.call() == []
    }

    def fold(n, acc, f) {
        n == 0 || isEmpty() ? acc : cdr().fold(n-1, f.call(acc, car()), f)
    }

    def foldAll(acc, f) {
        isEmpty() ? acc : cdr().foldAll(f.call(acc, car()), f)
    }

The only difference between this fold function and the standard one is the additional parameter n. We will need it later when we implement infinite lists. foldAll function to lazy lists is the same as standard fold to strict lists.

assert [1,2,3,4,5].lazy().foldAll(0){ acc, i -> acc + i } == 15
assert [1,2,3,4,5].lazy().fold(3, 1){ acc, i -> acc * i } == 6

First example calculates the sum of all elements of the list, second calculates the product of first three elements.

If you have fold functions you can easily implement take functions

    def take(n) {
        fold(n, []) {acc, item -> acc << item}
    }

    def takeAll() {
        foldAll([]) {acc, item -> acc << item}
    }
   
    def toList() {
        takeAll()
    }

take is an inverse operation to lazy

assert [1,2,3,4,5].lazy().takeAll() == [1,2,3,4,5]
assert [1,2,3,4,5].lazy().take(3) == [1,2,3]

Our next goal is map function on lazy lists. Ideally I want the implementation look like this

    def map(f) {
        isEmpty() ? nil() : cdr().map(f).cons(f.call(car()))
    }

For some reason it doesn't work lazy way in Groovy — it's still strictly evaluated. Therefore I have to implement it directly with closure syntax

    def map(f) {
        isEmpty() ? nil() : new LazyList( {-> [f.call(car()), cdr().map(f).list]} )
    }

Unlike fold, lazy map is identical to strict map

assert [1,2,3,4,5].lazy().map{ 2 * it }.take(3) == [2,4,6]

The following example shows one of the benefits of laziness

assert [1,2,3,0,6].lazy().map{ 6 / it }.take(3) == [6,3,2]

map didn't evaluate the entire list, hence there was no exception. If you evaluate expression for all elements, the exception will be thrown

try {
    [1,2,3,0,6].lazy().map{ 6 / it }.takeAll()
}
catch (Exception e) {
    assert e instanceof ArithmeticException
}

For strict lists this is a default behaviour of map function.

The last function I want to implement is filter

    def filter(p) {
        isEmpty() ? nil() :
            p.call(car()) ? new LazyList( {-> [car(), cdr().filter(p).list]} ) :
                cdr().filter(p)
    }

In the following example we find first two elements greater than 2

assert [1,2,3,4,5].lazy().filter{ 2 < it }.take(2) == [3,4]

With the help of car/cdr, fold, map and filter you can implement any other function on lazy lists yourself. Here is, for example, the implementation of zipWith function

    static def zipWith(alist, blist, f) {
        alist.isEmpty() || blist.isEmpty() ? nil() :
            new LazyList( {-> [
                f.call(alist.car(), blist.car()),
                zipWith(alist.cdr(), blist.cdr(), f).list
            ]} )
    }

Now, after we implemented all lazy functions we need, let's define infinite lists

    private static sequence(int n) {
        {-> [n, sequence(n+1)]}
    }

    static LazyList integers(int n) {
        new LazyList(sequence(n))
    }

    static LazyList naturals() {
        integers(1)
    }

Infinite lists, from my point of view, is the most useful application of lazy lists

def naturals = LazyList.naturals()
assert naturals.take(3) == [1,2,3]

def evens = naturals.map { 2 * it }
assert evens.take(3) == [2,4,6]

def odds = naturals.filter { it % 2 == 1 }
assert odds.take(3) == [1,3,5]

assert naturals.cadddddddddr() == 10

def nonnegatives = naturals.cons(0)
assert nonnegatives.cadr() == 1

assert LazyList.zipWith(evens, odds){ x, y -> x * y }.take(4) == [2,12,30,56]

At this point you have all basic functionality implemented, and you should be able to extend this model to whatever you need in regards to lazy (infinite) lists. Happy lazy programming!

Resources and links

• Source code for this blog

• Lazy list implementation in Erlang

• Lazy list implementation in Lisp

Counting modifications in Git repository

2011-01-29T21:51:00.003-05:00

Recently Michael Feathers wrote a blog about Open-Closed Principle, where he described simple technique that measures the closure of code. I created a Groovy script which implements this technique for Git repositories. If you run it from the root of your Git project, it produces a CSV file with the statistics of how many times files have been modified. Feel free to use this script to find hot spots in your Git repository.

Git + Maven

2011-01-24T20:13:00.004-05:00

When I first started working with Git in my Maven projects (three years ago), it was very awkward. Half of the release commands didn't work at all. Second half worked, but with ugly workaround via faked remote repository, which violated the entire Git philosophy.

Since then most of the issues have been resolved, including the following three which I mostly needed:

support for local Git repositories;
separation of git-commit and git-push commands in Maven release plugin;
critical bug fixes in Maven release and scm plugins.

I created a cheat sheet describing the way I typically set up and manage Git-Maven projects. Feel free to use it in your projects as well.

Joe Armstrong on optimization

2010-11-18T22:24:00.002-05:00

Make it work, then make it beautiful, then if you really, really have to, make it fast. 90 percent of the time, if you make it beautiful, it will already be fast. So really, just make it beautiful! (from the book)

I think that people first of all write a problem, they solve the problem and then they sort of optimize this code, work on it and the code becomes very efficient but unreadable. What I think they should doing is specifying it with a domain specific language or some higher thing and then writing a compiler and then changing the compiler because it’s not efficient. Because then they would have the benefits of a clear specification and a fast implementation. What they do is they don’t keep these things separated and the language doesn’t support them separating it like that. (from the interview)

Erlang explained: Selective receive

2010-11-10T19:49:00.006-05:00

If you worked with Erlang you've probably heard about selective receive. But do you actually know how it works? I want to post here an excerpt from Joe Armstrong's book Programming Erlang where he explains how it works exactly (Section 8.6, p.153):

receive
    Pattern1 [when Guard1] -> Expressions1;
    Pattern2 [when Guard2] -> Expressions2;
    ...
after
    Time -> ExpressionTimeout
end
When we enter a receive statement, we start a timer (but only if an after section is present in the expression).
Take the first message in the mailbox and try to match it against Pattern1, Pattern2, and so on. If the match succeeds, the message is removed from the mailbox, and the expressions following the pattern are evaluated.
If none of the patterns in the receive statement matches the first message in the mailbox, then the first message is removed from the mailbox and put into a "save queue." The second message in the mailbox is then tried. This procedure is repeated until a matching message is found or until all the messages in the mailbox have been examined.
If none of the messages in the mailbox matches, then the process is suspended and will be rescheduled for execution the next time a new message is put in the mailbox. Note that when a new message arrives, the messages in the save queue are not rematched; only the new message is matched.
As soon as a message has been matched, then all messages that have been put into the save queue are reentered into the mailbox in the order in which they arrived at the process. If a timer was set, it is cleared.
If the timer elapses when we are waiting for a message, then evaluate the expressions ExpressionsTimeout and put any saved messages back into the mailbox in the order in which they arrived at the process.

Did you notice the concept of "save queue"? That's what many people are not aware of. Let's play with various scenarios and see the mailbox and save queue in action.

The first scenario is simple, nothing to test there in regards to mailbox. The second one is also straightforward:

1> self() ! a.
a
2> process_info(self()).
 ...
 {message_queue_len,1},
 {messages,[a]},
 ...
3> receive a -> 1; b -> 2 end.
1
4> process_info(self()).
 ...
 {message_queue_len,0},
 {messages,[]},
 ...

You send a message to the shell, you see it in the process mailbox, then you receive it by matching, after which the queue is empty. Standard queue behaviour.

Now let's test scenario 3,5:

1> self() ! c, self() ! d, self() ! a.
a
2> process_info(self()).
 ...
 {message_queue_len,3},
 {messages,[c,d,a]},
 ...
3> receive a -> 1; b -> 2 end.
1
4> process_info(self()).
 ...
 {message_queue_len,2},
 {messages,[c,d]},
 ...

Again, no surprises. Actually, this example demonstrates what people think when they hear about selective receive. Unfortunately we don't see what happened internally between lines 3 and 4. We are going to investigate it now by testing scenario 3,4.

This time start the shell in distributed mode so that we can connect to it later from the remote shell.

(foo@bar)1> register(shell, self()).
true
(foo@bar)2> shell ! c, shell ! d.
d
(foo@bar)3> process_info(whereis(shell)).
 ...
 {current_function,{erl_eval,do_apply,5}},
 ...
 {message_queue_len,2},
 {messages,[c,d]},
 ...
(foo@bar)4> receive a -> 1; b -> 2 end.

At this moment the shell is suspended - we are exactly at step 4. Go to remote shell, and type the following:

(foo@bar)1> process_info(whereis(shell)).
 ...
 {current_function,{erl_eval,receive_clauses,6}},
 ...
 {message_queue_len,0},
 {messages,[]},
 ...

That's interesting: no messages in the mailbox. As Joe said, they are in the save queue. Now send a matching message:

(foo@bar)2> shell ! a.
a

Go back to initial shell, which should be resumed now, and check the mailbox again:

1
(foo@bar)5> process_info(whereis(shell)).
 ...
 {current_function,{erl_eval,do_apply,5}},
 ...
 {message_queue_len,2},
 {messages,[c,d]},
 ...

That's what we saw in the previous test, but now you know what happens behind the scenes: messages are moved from the mailbox to the save queue and then back to the mailbox after the matching message arrives.

Now you should understand better how selective receive works. Next time you explore your Erlang process, keep in mind the save queue and disappearing and reappearing messages.

Book review: Erlang and OTP in Action

2010-11-06T16:44:00.017-04:00

Title: Erlang and OTP in Action
Author: Martin Logan, Eric Merritt, and Richard Carlsson
Paperback: 432 pages
Publisher: Manning Publications; November 2010
Language: English
ISBN-10: 1933988789
ISBN-13: 978-1933988788
$38.99 (amazon.com)

Overview

Even though this book has Erlang in its title, it's only about 15% of the content dedicated to Erlang language itself — the biggest portion of the book is about OTP. Nowadays, when more and more developers get familiar with Erlang, they need a new book that can boost them to the next level of proficiency, where they can produce industry standard code leveraging all the power of Erlang platform. This book is supposed to fill this gap!

Part One — The OTP basics

Chapter 1 — The Erlang/OTP platform

This chapter gives an overview of the important concepts and features of Erlang/OTP: concurrency, fault-tolerance, distribution. It discusses four inter-process communication paradigms — shared memory, STM, futures, message passing — and shows how the latter makes the distribution trivial to implement in Erlang. You will see how the linked processes and supervision trees build the foundation of Erlang famous fault-tolerance, and how three aspects of Erlang runtime system — sophisticated scheduler, non-blocking IO, and per-process garbage collection — complete the picture.

Chapter 2 — Erlang language essentials

Take a deep breath — this long chapter is going to be an Erlang Crash Course. If you already worked with the language, most of it won't be new for you, but it's still worthy to read it because there are many small things that you are probably not aware of or don't use very often. For example,

• are you familiar with all available shell functions, and break menu options?
• do you know how to work with multiple shells in one window?
• how lists are implemented internally, and how to use ++ operator efficiently?
• what's the difference between arithmetic and exact equality operators?
• do you know that all operators are actually functions, and 1+2 is the same as erlang:'+'(1,2)?
• that assignment operator is a form of pattern matching?
• that you can use pattern matching instead of regex: "http://" ++ Rest = "http://www.erlang.org"?
• what's the difference between case- and if-expressions, and between pattern matching and guards?
• that besides list comprehensions there are also bitstring comprehensions: << <<X:3>> || X <- [1,2,3,4,5,6,7] >>?
• which steps Erlang preprocessor performs?
• what's the difference between linked and monitored processes?
• what's the relationship between messages and signals?

There is also nice introduction to algorithms in this chapter with excellent examples of how to use tail-recursion and accumulators to improve performance.

Some important topics are covered briefly, like selective receive mechanism, for example. But at the end of the chapter authors give a list of useful Erlang resources, including books and web sites, so you should be able to find there the answers to all your language related questions.

Chapter 3 — Writing a TCP based RPC service

This chapter is about OTP behaviours. It describes what behaviour is, what are the benefits of it comparing to pure Erlang implementation, and which parts the behaviour consists of. As a 'Hello, World' example authors implemented TCP server!

Over the course of the chapter you will learn how to model client-server communication using gen_server behaviour, and how to implement active socket connection using get_tcp module.

It also shows the industry conventions and best practices of how to implement and layout behaviour module. You can use this chapter as a reference every time you need to implement a behaviour.

Code snippet: TCP server.

Chapter 4 — OTP applications and supervision

An OTP application is what ties your modules into a single unit. A supervisor is what makes your application fault-tolerant. From this chapter you will learn how to implement both behaviours properly: how to layout application directory, how to structure application descriptor, how to write child specifications and restart strategies, and how to generate application documentation. As before, all examples are accompanied by standard conventions and best practices.

Sample code: application directory layout.

Chapter 5 — Using the main graphical introspection tools

This chapter demonstrates how to use some of the Erlang graphical tools: appmon, webtool, pman, debugger, and table viewer. It's good to know that those tools exist, so when you encounter a problem in your code, you would be able to find the root cause and resolve it quickly.

Part Two — Building a production system

In the second part of the book you are going to apply all the knowledge you obtained in the first part to build a real world production system: distributed cache.

Chapter 6 — Implementing a caching system

How do you implement a cache? I guess there are many ways to do it, but I would never come up with the idea the authors of the book came up with. They use a separate process to store each value, and they map each key to its corresponding process. How cool is that?! This way of thinking is possible only in Erlang.

During the implementation of process management you will learn a new strategy when the supervisor creates multiple child-processes in runtime based on preconfigured template. It's different from what you saw in Chapter 4 where single child process was created on the application startup. One interesting twist here is an inversion of control — the worker process will call the supervisor to start it.

The rest of the chapter is dedicated to ETS tables (which are used here to store the mapping). You will see how to create tables and how to perform CRUD operations.

Chapter 7 — Logging and event handling

Logging is very important part of any system. In OTP there are two logging utilities: error_logger and SASL. error_logger is similar to log4x libraries in other languages. It provides basic functions (info, warning and error) that you can call from your code to print messages to the standard output. SASL is more sophisticated. It's an OTP application that logs life cycle events, including crash reports, from other applications. Both methods are thoroughly described in the first part of this chapter.

The second part explains how to implement custom event handler. An OTP event handler is just another behaviour that models observer pattern. You can use it for example to implement your own log appender which you can plug in to the error_logger.

The final section of the chapter provides a step-by-step guide of how to build a custom event stream, and how to integrate it with the cache application. This technique was totally new to me. I never worked with event handlers on such advanced level.

Code snippets: event manager and event handler.

Chapter 8 — Distributed Erlang/OTP

Distribution is one of the famous features of Erlang. It's very easy to build distributed applications, and more important, it's such a fun to play with Erlang clusters.

This chapter will guide you through all the methods and techniques you need to know to make your application distributed. You will learn how to start Erlang nodes in different modes, how to combine them into the clusters, how to define topology and isolate clusters from each other, and how to send messages between nodes in the same cluster.

One of the cool things I learned from this chapter is a remote shell. It's very similar to SSH but more powerful. Unlike SSH, Erlang remote shell is not a session — it's a real shell of the remote node where you can start any application including graphical tools!

The second half of the chapter discusses the problem of resource discovery: What's the best way to add a new node to the cluster and synchronize its state with existing nodes? The authors come up with a simple and elegant algorithm. You will use this algorithm in the next chapter to build distributed cache.

Code snippet: resource discovery algorithm.

Chapter 9 — Adding distribution to the cache with Mnesia

When you design a distributed system you have to make a choice which inter-node communication strategy you are going to use: synchronous or asynchronous. Chapter 9 starts with the comparison of these two approaches, their advantages and drawbacks.

The next step towards the distributed cache is obvious: making the cache storage distributed. As you remember from the chapter 6 the storage was implemented as ETS table. The easiest way to make it distributed is to replace it with Mnesia database. Why and how? You will find it in the next section of this chapter.

You will learn what Mnesia is, how to configure it properly, and how to manipulate the data. At the end you will meet beauty and the beast of read operations - query list comprehensions and match specifications. Equipped with all these knowledge you will easily replace ETS table with Mnesia, and make your cache distributed.

The most amazing part of the last section is an algorithm of dynamic table replication. You will definitely appreciate it after you learn it — it's the heart of true scalability.

Code snippet: working with Mnesia.

Chapter 10 — Packaging, services, deployment

At this moment you should be able to write non-trivial OTP applications. It's time to think now how to make your application easy to install and start. So far you have started it manually from the shell, and if your app had many dependencies, it was a tedious process. This chapter describes how to automate it.

In OTP a deployment unit is called release. In this chapter you will learn how to build it properly, i.e. how to create release metadata and configuration, resolve dependencies, and generate boot scripts. You will see different ways to start your application: locally in shell, as a daemon or in embedded mode.

After you release the application, you might want to share it with other people. That's what the next section is about. It shows you how to make a package — standard or customized, universal or OS-dependent — and how to install it on a different machine.

Part Three — Integrating and refining

In the previous part you built a distributed cache in OTP, and you can use it now from any Erlang application. This is already a big achievement, and you must be proud of it, but you can make it even bigger if you expose this wonderful functionality to other platforms. Erlang is known for its robustness and scalability, and it would be very beneficial for non-Erlang clients as well to utilize these qualities.

Chapter 11 — Text and REST (Communication via TCP and HTTP)

The first non-Erlang interface you are going to implement is TCP. If you remember, you already did it in Chapter 3 when you implemented TCP server. That server though had one significant limitation: it handled only one connection. The new implementation in this chapter is more efficient: it supports multiple concurrent connections.

The next interface is HTTP. Although it sounds similar to the previous one, the way you will implement it is totally different. You won't use standard gen_server behaviour. Instead, you will implement a custom behaviour which you are going to define yourself. This is a very advanced topic, and if you want to build extensible systems in Erlang, you need to understand all the details of how to do it. Fortunately, this section provides thorough instructions.

Over the course of this chapter you will also learn bunch of other useful things besides server behaviours. You will see how HTTP protocol works and how to design RESTful services on top of it, how to use TCP sockets more effectively, and how to increase stability of your system with well-designed OTP supervisors. You will have lots of fun doing binary pattern matching.

Code snippets: TCP interface, HTTP server behaviour, REST interface.

Chapter 12 — Drivers (Communication with C programs)

This chapter is tough — you have to be a C programmer to understand all the details. If you are not involved into C programming, it's still worthy to read it, just to understand the concept, although it might be hard to get through the entire text.

There are two types of drivers in Erlang: port drivers and linked-in drivers. The chapter starts with an overview of them both. It explains their benefits and drawbacks, how you should design the driver, and where you should handle the driver's state, global vs. instance variables.

The rest of the chapter is a tutorial of how to implement drivers. It describes three components that comprise driver implementation: C side, Erlang side and the protocol between them. It shows the differences in each component for both types of drivers, and it gives you recommendations of how you should approach the problem required communication with C code.

Chapter 13 — Jinterface (Communication with Java programs)

Unlike the C driver implementation, connecting together Erlang and Java is pretty simple: you instantiate OtpNode class in the JVM thread, and it becomes available as Erlang node to any running Erlang application. You can start sending messages between Java and Erlang, and all Erlang terms will be properly converted to Java classes, and vice versa. All the magic is done in Jinterface library, which is a part of OTP distribution, and what you need to know to start using it is perfectly explained in this chapter.

After you learn how to work with Jinterface, you will apply this knowledge to building the bridge between the cache you implemented in the previous chapters and HBase. HBase is one of the modern NoSQL databases. If you didn't work with it before, don't worry — the authors will show you how to get started with it, and how to implement HBase connector using Java API. Having this API in place, all you need to do is to link it with the Erlang cache using the technique described above.

By the end of the chapter (and in fact end of the book) you will have a distributed cache written in Erlang backed by NoSQL database via Erlang-Java bridge. I don't know about you, but I was actually very impressed after I finished the coding and saw the entire solution working on my machine. It's really amazing that you can build pretty sophisticate piece of software with such a small amount of code.

Code snippets: message receiver and message responder.

Chapter 14 — Optimization and performance

As we all know, premature optimization is the root of all evil. In other words, don't spend time optimizing your solution before you actually measured the performance. That implies you must know what and how to measure. In this chapter authors describe the approach you should take when you prepare performance test, as well as the basic tools available in Erlang for performance testing: cprof and fprof.

The second part of the chapter explains the Erlang programming language caveats. You will see
• how primitive data types stored in memory, and which data structures you should use to fulfil performance requirements;
• how to use some built-in functions and operators properly;
• how to call function in different ways, and how performant those calls are;
• how compiler optimizes pattern matching and tail recursion;
• whether to use OTP behaviours or plain Erlang processes.

That concludes the main content of the book.

There are also two appendices in this book. The first one describes how to install Erlang on the OS of your choice. The second explains what referential transparency is and why lists in Erlang are implemented as they are.

Conclusion

If you are an intermediate Erlang developer, go and buy this book! It will teach you how to build robust production systems following proven design principles and standard conventions. It will make your code easy to read and maintain. You will learn lots of new things and it will be a big step towards Erlang mastery.

Happy OTPing!

SpringOne2GX 2010

2010-10-31T14:12:00.018-04:00

Last week I attended SpringOne2GX conference in Chicago, the main event in Spring/Groovy/Grails community. Here I want to post my brief review of this conference.

First impressions

The hotel (Westin Lombard) was nice and clean. Internet: there were 2 wireless networks and one cable - everything was free and worked pretty well, signal was good in almost all rooms. The conference reception was well organized - every participant received bunch of souvenirs and special edition of NFJS magazine. I saw hundreds of smiling and happy people of different ages and different outfits. Most of them with Macs. Most of them know each other. The food was fantastic, especially dinner with wine and beer.

Day 1

The first day was mostly introduction and orientation. There was only one talk on the schedule.

Rod Johnson - Keynote (video)

I thought Spring was initially created 7 years ago but the oldest class in the source tree is dated by January 17, 2001, so Spring is actually almost 10 years old. Because of the anniversary the main theme of the presentation was: Where Spring goes in the next decade.

Since the core framework is well crafted already, the focus will be on the integration and making Spring portfolio as a platform for applications. There are three key values in Spring - portability, productivity and innovation - and the platform will be built along those dimensions.

Portability

In the past SpringSource made a good job by providing a framework that make Java applications easily portable across different application servers. The goal for the next decade is to expand the same portability to the cloud - Google AppEngine, vFabric, vmforce, etc.

Productivity

As we all know the ultimate reason of the Spring existence is to make the life of application developer easier, our work is more productive. The framework hides the low-level boilerplate, and provides well defined abstractions. In the next year there will be several features added to the Spring portfolio. Rod mentioned some of them:
- Seamless GWT integration
- Database reverse engineering with roundtripping support in Spring Roo 1.1. You will be able to generate the domain object tree based on your database schema, and it will be updated every time you change the database.
- Spring Payment Services project with Visa integration.

Another aspect of productivity is a tool suite, and here Spring gives you STS. Rod invited Christian Dupuis on the stage, where he demoed how to developed Grails applications in STS. If you are a Grails developer you should definitely take a look at the latest version of STS - it will increase your productivity significantly.

Innovation

There will be several new projects released in the Spring portfolio soon:
- Spring Social - application abstraction for social networks.
- Spring Mobile - platform for multi-device applications.
- Spring-AMQP - API for integration with RabbitMQ.
- Spring Data - API to work with NoSQL databases, in particular Neo4J support in Spring Roo.

Keith Donald demoed GreenHouse project and corresponding iPhone app. This is a reference implementation of Spring Mobile and Spring Social, and this app was really really useful during the conference when I needed to check the schedule and find the room.

At the end of the presentation Rod introduced, and Mik Kersten demoed, the next big thing - Code2Cloud. It's basically a tool that allows you to keep and manage your entire development environment in the cloud: the running app, the source code, the issue tracker, and the build server. Everything is in the cloud and configured by mouse click. It looks cool, and it definitely will be a buzz word in the next year, but I'm not sure if many people will use it. We'll see.

Day 2

I'm going to write only about technical sessions I attended.

Jürgen Höller - What's new in Spring Framework 3.1? (video)

That was one of the best talks of this conference: technical, right to the point, with well-wrtten slides, and personal charm of the presenter. Despite the number 3.1 in the title, Jürgen actually covered three versions of Spring framework: 3.0, 3.1, and 3.2. I'm going to briefly mention the interesting features, and if you want more details you can check the excellent on-line documentation.

Spring 3.0

- Custom annotations. You can create your own annotation by combining multiple existing annotations in one group. Spring automatically detects your annotation during the application context startup, and no special configuration is required. This is a very handy feature, especially when you copy-paste the same annotation group over and over again.

- Configuration classes and annotated factory methods. If you annotate a method with @Bean annotation Spring framework will make the output of the method a Spring bean. There are some other annotations supported, e.g. @Lazy.

- Standardized annotations. Spring now supports JSR-330 @Inject, JSR-250 @ManagedBean, and EJB 3.x @TransactionAnnotation.

- EL++. Expression language can be used now in bean definitions inside appcontext XML, and also in component annotations. Very powerful feature.

- REST support. Spring provides RestTemplate for client code, @PathVariable annotation, and special view resolvers on the server side. It's very interesting topic - check the documentation for details.

- Declarative model validation. You can specify data constraints right in your code by using annotations - very similar to what you have in GORM.

- Improved scheduling. New namespace, and @Scheduled and @Async annotations makes your appcontext smaller and more readable.

If you follow Spring releases, you probably use some or most of these features already. Now let's see what Spring 3.1 brings to us.

Spring 3.1

- Environment profiles for beans. Similar to Maven profiles but works in runtime. The idea here is to create a single deployment unit for all environments and enable certain Spring beans for specific environment. I can't wait to try this feature in our enterprise project.

- Cache abstraction. After 5 years of hibernation this feature is finally implemented. Spring provides an API to work with distributed cache, in particular in cloud environments. There will be adapters for most popular cache implementations, such as EhCache, GemFire, Coherence.

- Conversation management, or how Jürgen calls it HttpSession++. It's basically an extension of HttpSession shared across multiple browsers and window tabs. Looks very interesting.

- Enhanced Groovy support.

- c: namespace, which is a shortcut for <constructor-arg>, analogous to p: namespace for properties. Small feature that makes your appcontext consistent and more readable.

Spring 3.2

Java SE 7 support, JDBC 4.1, support for fork-join framework, general focus on concurrent programming.

Jeff Brown - GORM inside and out

This talk was also good. I worked a bit with GORM before, and had an idea how it's implemented, but it was useful to hear more details from one of the developers.

Jeff started with the background of GORM, the complexity of Hibernate and JPA, and how GORM solves this problem following convention-over-configuration and sensible defaults strategy. He showed how to model the domain objects, what happens behind the scene when you link objects together, how to specify uni- and bi-directional relationships, and how to change default collection implementation in case of one-to-many relationship.

During the presentation he was switching back and forth between sides and terminal, so it was easy to follow and understand the evolution of the sample application. He explained how to introduce various constraints into the model and how Grails would validate them. One of the interesting features I didn't know about was how to test internationalized error messages. You don't need to change your locale for that, simply add lang=your_language parameter to the URL, and Grails will switch to that language for all subsequent requests. Pretty handy.

He concluded the talk by showing how dynamic finders are implemented in GORM using Groovy metaprogramming feature. Interesting part here is that you can implement similar things in your Groovy code using the same technique, basically having custom mini-GORM in your Groovy project.

Venkat Subramaniam - Improving your Groovy code quality

The title of this presentation was little bit misleading for me. I expected Venkat to show some Groovy specific mistakes and how to avoid them. Instead, he was talking about the errors that in most cases are equally applied to any programming language. He mentioned various code smells and explained how to fix them. If you are interested, you can download the slides from Venkat's web site.

He also gave an advise how to maintain the high code quality:
- Have a respectable colleague review your code.
- Use code analysis tools like CodeNarc and Sonar Groovy plugin.

One of the topics he covered was the usage of the 'return' keyword in Groovy. That was interesting. Compare the following two functions and guess what they return:

def func1() {
    try {
        5
    } finally {
        return 22
    }
}

def func2() {
    try {
        5
    } finally {
        22
    }
}

Paul King, Guillaume Laforge - Groovy.DSLs (from: beginner, to: expert) (video)

This would be very nice presentation if the speakers didn't try to cover too much. This talk could be easily split into two: one is an overview of Groovy language and another one is DSL. Unfortunately they spent lot of time on theoretical DSL part and Groovy overview, so the practical DSL part was too short from my perspective. The good thing though is that I have slides now, so I can dig deeper into this subject at my spare time.

In the second part of the talk Paul and Guillaume explained which features of Groovy language make it so simple to create DSLs. Here are some of them:
- Static imports and import aliases.
- Simplified collection syntax.
- Small or no language noise.
- Aggregating multiple method calls using 'with' construct.
- Closures.
- Operator overloading.
- Metaprogramming.

In the last part speakers talked about different patterns and techniques of DSL implementations. They provided a comprehensive list of books you might want to read if you are interested in building DSLs.

Adrian Colyer - Technical keynote (video)

Adrian's talk was mostly a reiteration of Rod's keynote from the previous day with some technical details. He mentioned Spring Payment and Spring Data projects, bean profiles and cache support in the Spring core. He showed Spring portability in action by providing links to Spring applications deployed on Google AppEngine and vmForce.

Another interesting part was 20 minutes dedicated to RabbitMQ and Spring-AMQP. He even mentioned Spring-Erlang project which is supposed to be a convenient abstraction on top of standard Jinterface library.

As a continuation of innovation theme Graeme Rocher demoed GORM support for NoSQL databases. That was cool. He simply uninstalled Hibernate plugin and installed Redis plugin, without touching data model. Everything worked perfect. Right now Spring works with Redis and GemFire, but soon they are going to add support for CouchDB, Cassandra, Riak, Neo4j, and MongoDB. Another interesting thing Graeme showed was grails-console. It's a pretty nice tool, you should check it out. It allows you to interact with the Grails data storage using GORM features. Very handy.

Another co-presenter was Keith Donald who demoed Spring Social and Spring Mobile. He explained how OAuth works, and how interoperability with social networks was implemented in GreenHouse.

The keynote was concluded by Jon Travis who demoed SpringInsight.

Day 3

Venkat Subramaniam - Functional programming in Groovy

That was an excellent talk and nice start of the new conference day. Venkat explained main concepts and values of functional programming, and illustrated the theory with comprehensible examples.

He compared imperative and functional style of programming by showing how to implement for-loop using inject() function in Groovy. I think it was one of the best explanations of functional folding I've ever heard. He also demonstrated map and filter operations using collect() and findAll() methods.

He clarified the difference between function value and closure, and between iterative procedure and iterative process. He gave an example on how to pass closure as a parameter to simulate function object in Groovy. He also showed how to replace tail-recursion, which Groovy doesn't support, with inject() method call.

The presentation was concluded with an example of how to use functional techniques to build DSLs in Groovy.

Matthias Radestock, Mark Pollack, Mark Fisher - RabbitMQ and Spring-AMQP (video)

If you read my blog, you know that RabbitMQ is one of my latest interests. I decided to go to this talk just to see how the creators would present their projects. It turned out to be a nice introduction to RabbitMQ and Spring-AMQP. They explained main concepts of AMQP and how it is different from JMS. Here I want to give you some ideas which were not obvious for me when I started working with RabbitMQ.

- Messaging is all about decoupling, and AMQP is much more flexible than JMS in terms of publisher-consumer decomposition.
- All resources are dynamically created and destroyed by clients - the static pre-configuration is optional.
- Exchanges are stateless, they don't keep messages, they only copy and dispatch them. Queues hold the messages and deliver one message to a single client. They neither do routing nor message copying.
- Queue never receives the same message twice.
- If the message doesn't match routing key it's dropped.
- Because of the open protocol, you can use all available TCP tools to monitor your message traffic.

Besides AMQP implementation RabbitMQ also provides some other useful features like custom exchanges, exchange-to-exchange routing, different protocol adaptors, etc. Spring, as usual, gives you a consistent API on top of the RabbitMQ client which hides all low-level boilerplate and makes your application code more readable.

Good presentation, great guys.

Craig Walls - Developing social-ready web applications (video)

This presentation was about integrating your Java code with different social networks. There are three types of such integration: widgets, embedded code, REST API. Craig briefly explained first two, and then dived into REST.

All popular social networks provide REST API which allows you to communicate with them. For simple operations, like search, you can just use standard Spring RestTemplate class to retrieve the data. Try for example the following URLs:
- http://api.twitter.com/1/friends/ids.xml?screen_name=ndpar
- http://search.twitter.com/search.json?q=s2gx
- https://graph.facebook.com/ndpar

This basic approach fails though if you try to post a new message, because you have to be authorized for update operations. That's where OAuth comes in. The idea behind OAuth is pretty simple: instead of sharing your user-password with different clients, it uses generated tokens. This model is more flexible because if you want to revoke the permission from particular client you don't need to change your password and notify rest of the clients - you just remove that client's token from the list of authorized clients and that's it. The only problem with OAuth and social networks is that they support different versions of OAuth. This problem is solved by Spring Social project.

Spring Social offers consistent template-based API across different social providers. It basically gives you an OAuth aware RestTemplate, so you can do something like this:

TwitterTemplate twitter = new TwitterTemplate(API_KEY, API_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET);
twitter.updateStatus("Hello #s2gx !");
twitter.retweet(26887414177L);

If you are in a social network business, definitely take a look at Spring Social.

Mark Pollack, Chris Richardson - Using Spring with non-relational databases (video)

Relational databases are great, right? They've been with us for ages. Everybody knows how to work with them, how to build SQL statements. Every language provides ODBC library. There are bunch of frameworks that make developer's life easier. So why so sudden buzz around NoSQL?

Mark and Chris started their talk highlighting some problems that exist in relational database world:
- Object-relational impedance mismatch. Complicated mapping of rich domain model to relational schema. Relational schema rigidity.
- Extremely difficult/impossible to scale write operations.
- Suboptimal performance in some cases.

All these issues are addressed in NoSQL databases. Although keep in mind that it's not coming for free - you have to trade off ACID semantics, transactions and some other features of RDBMS. But if scalability is more important for you than consistency then NoSQL is your way to go.

There are tons of NoSQL databases available for you, but they all can be split into 4 categories based on their data model:
- Key-Value: Amazon Dynamo, Redis, Riak, Voldemort.
- Column: Google Bigtable, HBase, Cassandra.
- Document: CouchDB, MongoDB.
- Graph: Neo4j, Sones, InfiniteGraph.

Mark and Chris talked about each type, what their typical use cases are, and how their APIs look like. They showed examples for Redis, Cassandra, MongoDB, CouchDB and Neo4j. Then they introduced Spring Data project which, as everything from SpringSource, simplifies the application development and eliminates low-level code. Right now they support most of the popular NoSQL databases, and they plan to add more in the future.

The project is in active development phase, and the new contributors are welcome. So if it sounds interesting for you, go and check it out.

Day 4

Hans Dockter - Gradle - a better way to build

I never played with Gradle, so I was very curious to see how it looks like. According to Hans, who is the creator of this tool, Gradle is a general purpose build system with Groovy DSL interface. It's written in Java and provides build-in support for Java, Groovy, Scala, web and OSGi projects. It's a build language, so you can extend it for your own purposes if needed.

If you compare it with Ant, Gradle is definitely much better because it's more compact and flexible. It offers dependency resolution with integration with Maven and Ivy repositories. It also has some advanced features like incremental builds for custom tasks and parallel testing.

The only problem I had with this presentation was that Hans kept comparing Gradle with Maven. In my opinion they are not comparable. They have different philosophy if you want. All Maven 'constraints' are imposed by design, so it makes no sense to blame Maven for them. I think Ant-Gradle comparison is more appropriate and that's what Hans should have emphasized.

Other than that the session was pretty informative, and I have a better picture of Gradle now.

Brian Sletten - Groovy + The Semantic Web

I had no idea what Semantic Web was. I saw this term first time on the conference schedule, so I decided to go to this talk just to educate myself. I cannot even briefly describe all the discoveries I made during this presentation because I still feel little bit overwhelmed. I just want to provide some links from Brian's slides that can guide you if you want to learn this concept.

- Semantic Web - article from wikipedia.
- Formal W3C specs: RDF, RDFa, SKOS, SPARQL, OWL.
- SPARQL demo.
- RDFa distiller and parser. Try to feed Brian's test page URL (http://bosatsu.net/nfjs/test.html) to the distiller and see what it returns.
- OG - open graph protocol.
- Jena - Java API to work with Semantic Web.
- Java-RDFa parser.
- Pellet - Java API for OWL.

Conclusion

Whew! This happens to be longer review than I planned initially. If you are still with me you deserve my applause!

There were much more presentations at this conference but because of the tight schedule I had to sacrifice 80% of them. My overall impression from this conference is very positive. If you are a Spring/Groovy/Grails developer I encourage you to go to this event next year. The biggest benefit of it: You start seeing the Spring as a universe, not as a bunch of separate projects. You cannot get this feeling from the documentation, even if it's perfect as the Spring one.

Working with RabbitMQ in Spring applications

2010-08-02T18:53:00.005-04:00

Recently SpringSource released Spring AMQP 1.0.0.M1. Now, if you are a Spring shop working with RabbitMQ, you don't need to write low level code to connect to RabbitMQ server anymore. Instead, you can use well-known Spring abstractions (message templates and containers) to produce/consume AMQP messages, the same approach you would use for JMS. Here is my previous example re-implemented using Spring AMQP.

Very simple application classes (sender and receiver)

import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.beans.factory.annotation.Autowired;

public class MessageSender {

    @Autowired
    private AmqpTemplate template;

    public void send(String text) {
        template.convertAndSend(text);
    }
}

import org.springframework.amqp.core.Message;
import org.springframework.amqp.core.MessageListener;

public class MessageHandler implements MessageListener {

    @Override
    public void onMessage(Message message) {
        System.out.println("Received message: " + message);
    }
}

and pretty standard application context

<context:annotation-config />

<bean id="rabbitConnectionFactory" class="org.springframework.amqp.rabbit.connection.SingleConnectionFactory"
    p:username="guest" p:password="guest" p:virtualHost="/" p:port="5672">
    <constructor-arg value="lab.ndpar.com" />
</bean>

<bean id="rabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate"
    p:connectionFactory-ref="rabbitConnectionFactory"
    p:routingKey="myRoutingKey"
    p:exchange="myExchange" />

<bean id="messageSender" class="com.ndpar.spring.rabbitmq.MessageSender" />


<bean class="org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer"
    p:connectionFactory-ref="rabbitConnectionFactory"
    p:queueName="myQueue"
    p:messageListener-ref="messageListener" />

<bean id="messageListener" class="com.ndpar.spring.rabbitmq.MessageHandler" />

That's it, simple and clean.

Resources

• Spring AMQP official page

• Source code for this blog

Integrating RabbitMQ with ejabberd

2010-03-31T19:06:00.012-04:00

Last few days I've been trying to make RabbitMQ and ejabberd work smoothly together by means of mod_rabbitmq gateway. The official mod_rabbitmq document is pretty clear but the installation chapter is rather short. Plus, it presumes that ejabberd is installed from the source tree, which might not be the case. Here I want to give you more detailed instructions on the installation/configuration process in case mod_rabbitmq doesn't work for you out of the box.

My environment is Ubuntu 9.10 with rabbitmq-server and ejabberd packages installed via apt-get. Both RabbitMQ and ejabberd are up and running. Now I want them to talk to each other and route messages properly.

Compiling mod_rabbitmq

If you have the same environment as mine you can just download the binary and the header files, and copy them to the corresponging ejabberd folders (see last two lines in the bash snippet below). Alternatively you can compile mod_rabbitmq.beam file yourself:

$ git clone git://git.process-one.net/ejabberd/mainline.git ejabberd
$ cd ejabberd
$ git checkout -b 2.1.x origin/2.1.x
$ cd src
$ wget http://hg.rabbitmq.com/rabbitmq-xmpp/raw-file/73c129561101/src/mod_rabbitmq.erl
$ wget http://hg.rabbitmq.com/rabbitmq-xmpp/raw-file/73c129561101/src/rabbit.hrl
$ ./configure --disable-tls
$ make
$ sudo cp mod_rabbitmq.beam /usr/lib/ejabberd/ebin/
$ sudo cp rabbit.hrl /usr/lib/ejabberd/include/

Configuring mod_rabbitmq

You need to know the short name of the machine you are running RabbitMQ on. Use hostname -s command for this. Open /etc/ejabberd/ejabberd.cfg file for edit, find modules section, and add mod_rabbitmq stanza to the list

{modules,
[
  {mod_adhoc,    []},
  ...
  {mod_rabbitmq, [{rabbitmq_node, rabbit@yourhostname}]},
  ...
  {mod_version,  []}
]}.

Replace yourhostname with your machine short name. In my case it was ubuntu.

Setting up cookie

To make RabbitMQ and ejabberd work together, they have to run in the same Erlang cluster. That means they have to use the same cookie file. By default RabbitMQ is installed under rabbitmq user with /var/lib/rabbitmq home directory, and ejabberd under ejabberd user with /var/lib/ejabberd home directory. If you compare their cookies

$ sudo cat /var/lib/rabbitmq/.erlang.cookie
$ sudo cat /var/lib/ejabberd/.erlang.cookie

they will most likely be different. That's why if you restarted ejabberd now you would see exception in RabbitMQ log: "Connection attempt from disallowed node ejabberd@ubuntu". To fix it just copy one cookie file to another

$ sudo /etc/init.d/ejabberd stop
$ sudo mv /var/lib/ejabberd/.erlang.cookie /var/lib/ejabberd/.erlang.cookie.orig
$ sudo cp /var/lib/rabbitmq/.erlang.cookie /var/lib/ejabberd/.erlang.cookie
$ sudo chown ejabberd:ejabberd /var/lib/ejabberd/.erlang.cookie
$ sudo /etc/init.d/ejabberd start

The installation part is now done, and you are good to go.

Adding rabbit buddy to your roster

The rabbit's JID comprises two parts: exchange name and routing domain. To find the latter one, look at the /var/log/ejabberd/ejabberd.log file. Searching for "Routing" you should get something like this

=INFO REPORT==== 2010-03-30 21:35:22 ===
{contacted_rabbitmq,rabbit@ubuntu}

=INFO REPORT==== 2010-03-30 21:35:22 ===
I(<0.314.0>:mod_rabbitmq:90) : Routing: "rabbitmq.jabber.ndpar.com"

This is the buddy's domain. For the name you can use any exchange name available in the RabbitMQ server. Run sudo rabbitmqctl list_exchanges command and pick up the name from the list. I use amq.fanout exchange which exists in every RabbitMQ server. So I go to my IM client (Adium) and add this user to the buddies list

amq.fanout@rabbitmq.jabber.ndpar.com

Rabbit's greetings

To publish a message to RabbitMQ I use the same Groovy script as in the previous post. I just amended the exchange name and routing key

channel.basicPublish 'amq.fanout', '', null, 'Hello, world!'.bytes

Run the script and voilà, you've got mail

Troubleshooting

Here are some hints for you if something goes wrong.

• While working with mod_rabbitmq keep an eye on the log files of both RabbitMQ and ejabberd:

$ tail -f /var/log/ejabberd/ejabberd.log
$ tail -f /var/log/rabbitmq/rabbit.log

• Check which exchanges, queues and bindings the RabbitMQ server has:

$ sudo rabbitmqctl list_exchanges
$ sudo rabbitmqctl list_queues
$ sudo rabbitmqctl list_bindings

• If you screw up something there, you can roll back to the default values:

$ sudo rabbitmqctl stop_app
$ sudo rabbitmqctl reset
$ sudo rabbitmqctl start_app

• Check ejabberd web admin, it has lots of information there

http://yourdomainname:5280/admin

• If your IM client is Adium, check its folder periodically — it tends to collect some garbage there:

~/Library/Application Support/Adium 2.0/Users/Default/libpurple

Resources

• Tony Garnock-Jones' presentation slides about RabbitMQ and its extensions

Get started with RabbitMQ

2010-03-14T14:54:00.009-04:00

RabbitMQ is an open-source implementation of AMQP. If you don't know what AMQP is, I encourage you to check it out on the official web site, or alternatively read articles listed on the reference page. Here I want to mention only the reasons why it drew my attention as an Erlang enthusiast and Java developer working in financial industry:

AMQP is a replacement for TIBCO Randezvous;
in terms of functionality it's a superset of JMS;
it's written in Erlang, which means fault-tolerance, reliability and high performance.

In this blog post I just want to show how to install RabbitMQ on Ubuntu box, and verify that it works with simple Groovy client.

Installing RabbitMQ server

As everything with Ubuntu, this step is pretty trivial:

$ sudo apt-get install rabbitmq-server

The only requirement for this package is Erlang distribution. If you already have Erlang installed on your system, the installation of rabbitmq-server is a quick procedure. The following directories will be created during the installation:

/usr/lib/rabbitmq/bin	executables added to the path
/usr/lib/erlang/lib/rabbitmq_server-1.x.x	compiled modules
/var/lib/rabbitmq/mnesia	persistent storage for messages
/var/log/rabbitmq	log files (e.g. startup_log, rabbit.log)

After installation is finished the RabbitMQ server is started and listens to incoming requests on port 5672. You can check /var/log/rabbitmq/startup_log file to see if everything was ok.

Groovy clients

I followed official Java client API to build two scripts: consumer.groovy

import com.rabbitmq.client.*

@Grab(group='com.rabbitmq', module='amqp-client', version='1.7.2')
params = new ConnectionParameters(
    username: 'guest',
    password: 'guest',
    virtualHost: '/',
    requestedHeartbeat: 0
)
factory = new ConnectionFactory(params)
conn = factory.newConnection('lab.ndpar.com', 5672)
channel = conn.createChannel()

exchangeName = 'myExchange'; queueName = 'myQueue'

channel.exchangeDeclare exchangeName, 'direct'
channel.queueDeclare queueName
channel.queueBind queueName, exchangeName, 'myRoutingKey'

def consumer = new QueueingConsumer(channel)
channel.basicConsume queueName, false, consumer

while (true) {
    delivery = consumer.nextDelivery()
    println "Received message: ${new String(delivery.body)}"
    channel.basicAck delivery.envelope.deliveryTag, false
}
channel.close()
conn.close()

and publisher.groovy

import com.rabbitmq.client.*

@Grab(group='com.rabbitmq', module='amqp-client', version='1.7.2')
params = new ConnectionParameters(
    username: 'guest',
    password: 'guest',
    virtualHost: '/',
    requestedHeartbeat: 0
)
factory = new ConnectionFactory(params)
conn = factory.newConnection('lab.ndpar.com', 5672)
channel = conn.createChannel()

channel.basicPublish 'myExchange', 'myRoutingKey', null, "Hello, world!".bytes

channel.close()
conn.close()

Now start consumer in one terminal window

$ groovy consumer.groovy

and run publisher in another:

$ groovy publisher.groovy

On the consumer window you should see Received message: Hello, world! text, which means RabbitMQ works correctly.

Monitoring logs

You can check RabbitMQ logs by doing tail -f /var/log/rabbitmq/rabbit.log For example, starting the consumer results the following log entries:

=INFO REPORT==== 14-Mar-2010::11:20:53 ===
accepted TCP connection on 0.0.0.0:5672 from 192.168.2.10:62424

=INFO REPORT==== 14-Mar-2010::11:20:53 ===
starting TCP connection <0.24154.1> from 192.168.2.10:62424

Running the publisher:

=INFO REPORT==== 14-Mar-2010::11:22:08 ===
accepted TCP connection on 0.0.0.0:5672 from 192.168.2.10:62432

=INFO REPORT==== 14-Mar-2010::11:22:08 ===
starting TCP connection <0.24232.1> from 192.168.2.10:62432

=INFO REPORT==== 14-Mar-2010::11:22:08 ===
closing TCP connection <0.24232.1> from 192.168.2.10:62432

Now if we terminate the consumer by ^C there will be a warning

=WARNING REPORT==== 14-Mar-2010::11:25:03 ===
exception on TCP connection <0.24154.1> from 192.168.2.10:62424
connection_closed_abruptly

=INFO REPORT==== 14-Mar-2010::11:25:03 ===
closing TCP connection <0.24154.1> from 192.168.2.10:62424

but the connection is closed properly by the server.

That's it for now. Stay tuned for the future updates on my RabbitMQ experience.

Links

• Rapid application prototyping with Groovy DSL

Installing ejabberd on Ubuntu

2010-02-23T21:40:00.008-05:00

Recently I've installed ejabberd server on Ubuntu box. Thanks to this nice document, the process was pretty straightforward. My experience was little bit different from the author's one, so I want to show here exact steps I did to make it work, maybe it will be helpful for you too.

The first step is to install the required package. You can use Synaptic Package Manager or just command line:

sudo apt-get install ejabberd

During the installation a new user, ejabberd, will be created in the system. This is the user the server will be running on. When installation is finished ejabberd server is started. To configure the server you need to stop it

sudo /etc/init.d/ejabberd stop

Next step is to configure administrator and hosts. Open /etc/ejabberd/ejabberd.cfg file for edit and make the following change

%% Admin user
{acl, admin, {user, "andrey", "jabber.ndpar.com"}}.
%% Hostname
{hosts, ["localhost", "ubuntu", "jabber.ndpar.com"]}.

For admin you need to specify the user name and domain name that you want to use as a Jabber ID. By default it's localhost and it's functional but it's better to change it to something meaningful. The list of hostnames is tricky. In theory you can provide there just localhost but in practice it didn't work for me. After digging into some Erlang exceptions I got while registering admin account (see next step) I came to conclusion that in the list of hostnames there must be a short hostname of the box. You can get it by running hostname -s command (in my case it was "ubuntu"). In addition you can provide other hostnames you like, but the short one is mandatory.

When you are done with editing, start the server

sudo /etc/init.d/ejabberd start

Now it's time to register the admin user we configured on the previous step. Run the following command replacing password placeholders with the actual password and providing user name and domain name from ejabberd.cfg file

sudo ejabberdctl register andrey jabber.ndpar.com xxxxxx

That's it! You have now working XMPP server with one registered user. To verify that everything is ok, in your browser go to the admin page of the server (http://jabber.ndpar.com:5280/admin) and check the statistics. You'll be asked to type your JID and password, so use the information you entered on the previous step

As a note, I didn't create my own SSL certificate because for isolated intranet the default one is quite enough. If you are not comfortable with that feel free to create a new certificate following the steps from the original article.

Now you are ready to add newly created account to your Jabber client. In Adium, for example, go to File -> Add Acount -> Jabber and provide server hostname/IP, JID and password.

Click OK button, accept security certificate permanently and go online.

Now, to really enjoy IM you need more users on your server. The best part here is that you can create new users just from your Jabber client. You can actually do many things from the client, and you don't need to ssh to the remote server and run command for that. Just go to File -> your ejabberd account, and chose whatever you need from the menu

Pretty cool, eh — client and admin tool in one place.

Multithreaded XmlSlurper

2010-02-10T19:33:00.009-05:00

Groovy XmlSlurper is a nice tool to parse XML documents, mostly because of the elegant GPath dot-notation. But how efficient is XmlSlurper when it comes to parsing of thousands of XMLs per second? Let's do some simple test

class XmlParserTest {

    static int iterations = 1000

    def xml = """
       <root>
         <node1 aName='aValue'>
           <node1.1 aName='aValue'>1.1</node1.1>
           <node1.2 aName='aValue'>1.2</node1.2>
           <node1.3 aName='aValue'>1.3</node1.3>
         </node1>
         <node2 aName='aValue'>
           <node2.1 aName='aValue'>2.1</node2.1>
           <node2.2 aName='aValue'>2.2</node2.2>
           <node2.3 aName='aValue'>2.3</node2.3>
         </node2>
         <nodeN aName='aValue'>
           <nodeN.1 aName='aValue'>N.1</nodeN.1>
           <nodeN.2 aName='aValue'>N.2</nodeN.2>
           <nodeN.3 aName='aValue'>N.3</nodeN.3>
         </nodeN>
       </root>
    """

    def parseSequential() {
        iterations.times {
            def root = new XmlSlurper().parseText(xml)
            assert 'aValue' == root.node1.@aName.toString()
        }
    }

    @Test void testSequentialXmlParsing() {
        long start = System.currentTimeMillis()
        parseSequential()
        long stop = System.currentTimeMillis()
        println "${iterations} XML documents parsed sequentially in ${stop-start} ms"
    }
}

I ran this test on my 4-core machine and I got

1000 XML documents parsed sequentially in 984 ms

Not really good (0.984 ms per document) but we didn't expect much from single threaded application. Let's parallelize this process

class XmlParserTest {
    ...
    static int threadCount = 5
    ...
    @Test void testParallelXmlParsing() {
        def threads = []
        long start = System.currentTimeMillis()
        threadCount.times {
            threads << Thread.start { parseSequential() }
        }
        threads.each { it.join() }
        long stop = System.currentTimeMillis()
        println "${threadCount * iterations} XML documents parsed parallelly by ${threadCount} threads in ${stop - start} ms"
    }
}

And the result is

5000 XML documents parsed parallelly by 5 threads in 1750 ms

This is definitely better (0.35 ms per document) but doesn't look like parallel processing — the test time shouldn't increase in true parallelism.

The problem here is the default constructor of XmlSlurper. It does too much: first, it initializes XML parser factory loading bunch of classes; second, it creates new XML parser, which is quite expensive operation. Now imaging this happens thousand times per second.

Luckily, XmlSlurper has another constructor, with XML parser parameter, so we can create the parser up-front and pass it to the slurper. Unfortunately, we cannot reuse one parser instance between several slurpers because XML parser is not thread-safe — you have to finish parsing one document before you can use the same parser to parse another.

The solution here is to use preconfigured pool of parsers. Let's create one based on Apache commons-pool library.

public class XmlParserPoolableObjectFactory implements PoolableObjectFactory {
    private SAXParserFactory parserFactory;

    public XmlParserPoolableObjectFactory() {
        parserFactory = SAXParserFactory.newInstance();
    }
    public Object makeObject() throws Exception {
        return parserFactory.newSAXParser();
    }
    public boolean validateObject(Object obj) {
        return true;
    }
    // Other methods left empty
}

public class XmlParserPool {
    private final GenericObjectPool pool;

    public XmlParserPool(int maxActive) {
        pool = new GenericObjectPool(new XmlParserPoolableObjectFactory(), maxActive,
                GenericObjectPool.WHEN_EXHAUSTED_BLOCK, 0);
    }
    public Object borrowObject() throws Exception {
        return pool.borrowObject();
    }
    public void returnObject(Object obj) throws Exception {
        pool.returnObject(obj);
    }
}

Now we can change our test

class XmlParserTest {
    static XmlParserPool parserPool = new XmlParserPool(1000)
    ...
    def parseSequential() {
        iterations.times {
            def parser = parserPool.borrowObject()
            def root = new XmlSlurper(parser).parseText(xml)
            parserPool.returnObject(parser)
            assert 'aValue' == root.node1.@aName.toString()
        }
    }
}

and run it again

1000 XML documents parsed sequentially in 203 ms
5000 XML documents parsed parallelly by 5 threads in 172 ms

That's much better (0.034 ms per document), and most importantly multi-threading really works now.

Resources

• Source code for this blog

• Article "Improve performance in your XML applications"

• GPath vs XPath

• commons-pool home page

Distributed cache in Erlang

2010-01-23T20:21:00.011-05:00

Implementing distributed cache in Erlang is relatively simple task because concurrency, distribution and failover mechanisms are built in the language. In fact, it's so simple that this task is a part of Erlang tutorial. Here I want to show the complete solution which is only 100 lines of code.

I'm going to implement the cache as a typical Erlang server application, that means set of three modules: server, supervisor and application. As underlined storage I'm using Mnesia database which is a part of standard Erlang distribution. It doesn't probably give you the best performance, but it does provide automatic replication. The cache is deployed on three nodes, each node on a separate machine.

Clients will connect to in-memory slave nodes, the master node is dedicated to persistence.

Configure Erlang cluster

Create file .erlang.cookie containing one line with random text. Copy this file to every machine in a cluster to home directory of the user who will start Erlang VM. Make sure this file has unix permissions 600.

Check /etc/hosts on every box to verify that every machine knows others by name.

Set up Mnesia database

Open terminals on all machines and enter Erlang prompt

ubuntu$ erl -sname master
Erlang R13B01 (erts-5.7.2) [source] [rq:1] [async-threads:0] [kernel-poll:false]
Eshell V5.7.2  (abort with ^G)

macBook$ erl -sname slave1
Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
Eshell V5.7.3  (abort with ^G)

iMac$ erl -sname slave2
Erlang R13B03 (erts-5.7.4) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
Eshell V5.7.4  (abort with ^G)

From one machine ping other two

(slave1@macBook)1> net_adm:ping(master@ubuntu).
pong
(slave1@macBook)2> net_adm:ping(slave2@iMac).
pong

Create database configuration

(slave1@macBook)3> mnesia:create_schema([slave1@macBook, slave2@iMac, master@ubuntu]).
ok

Start database on all nodes

(master@ubuntu)1> application:start(mnesia).
ok
(slave1@macBook)4> application:start(mnesia).
ok
(slave2@iMac)1> application:start(mnesia).
ok

Create cache table

(slave1@macBook)5> rd(mycache, {key, value}).
mycache
(slave1@macBook)6> mnesia:create_table(mycache, [{attributes, record_info(fields, mycache)},
{disc_only_copies, [master@ubuntu]}, {ram_copies, [slave1@macBook, slave2@iMac]}]).
{atomic,ok}

Stop database and quit Erlang VM

(slave1@macBook)7> application:stop(mnesia).
ok
(slave2@iMac)2> application:stop(mnesia).
ok
(master@ubuntu)2> application:stop(mnesia).
ok

Implement Erlang application

The main module of this application is mycache.erl

-module(mycache).
-export([start/0, stop/0]).
-export([put/2, get/1, remove/1]).
-export([init/1, terminate/2, handle_call/3, handle_cast/2]).
-behaviour(gen_server).
-include("mycache.hrl").

% Start/stop functions

start() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

stop() ->
    gen_server:cast(?MODULE, stop).

% Functional interface

put(Key, Value) ->
    gen_server:call(?MODULE, {put, Key, Value}).

get(Key) ->
    gen_server:call(?MODULE, {get, Key}).

remove(Key) ->
    gen_server:call(?MODULE, {remove, Key}).

% Callback functions

init(_) ->
    application:start(mnesia),
    mnesia:wait_for_tables([mycache], infinity),
    {ok, []}.

terminate(_Reason, _State) ->
    application:stop(mnesia).

handle_cast(stop, State) ->
    {stop, normal, State}.

handle_call({put, Key, Value}, _From, State) ->
    Rec = #mycache{key = Key, value = Value},
    F = fun() ->
            case mnesia:read(mycache, Key) of
                [] ->
                    mnesia:write(Rec),
                    null;
                [#mycache{value = OldValue}] ->
                    mnesia:write(Rec),
                    OldValue
            end
        end,
    {atomic, Result} = mnesia:transaction(F),
    {reply, Result, State};

handle_call({get, Key}, _From, State) ->
    case mnesia:dirty_read({mycache, Key}) of
        [#mycache{value = Value}] -> {reply, Value, []};
        _ -> {reply, null, State}
    end;

handle_call({remove, Key}, _From, State) ->
    F = fun() ->
            case mnesia:read(mycache, Key) of
                [] -> null;
                [#mycache{value = Value}] ->
                    mnesia:delete({mycache, Key}),
                    Value
            end
        end,
    {atomic, Result} = mnesia:transaction(F),
    {reply, Result, State}.

It implements Erlang generic server behaviour and provides three client functions – put, get, remove – with the same signature as similar methods in java.util.Map interface.

Next file is a supervisor for the cache, mycache_sup.erl

-module(mycache_sup).
-export([start/0]).
-export([init/1]).
-behaviour(supervisor).

start() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init(_) ->
    MycacheWorker = {mycache, {mycache, start, []}, permanent, 30000, worker, [mycache, mnesia]},
    {ok, {{one_for_all, 5, 3600}, [MycacheWorker]}}.

It's going to monitor the main cache process and restart it in case of crash.

Next file, mycache_app.erl, provides methods to start and stop our cache gracefully within Erlang VM

-module(mycache_app).
-export([start/2, stop/1]).
-behaviour(application).

start(_Type, _StartArgs) ->
    mycache_sup:start().

stop(_State) ->
    ok.

Create application descriptor, mycache.app

{application, mycache,
 [{description, "Distributed cache"},
  {vsn, "1.0"},
  {modules, [mycache, mycache_sup, mycache_app]},
  {registered, [mycache, mycache_sup]},
  {applications, [kernel, stdlib]},
  {env, []},
  {mod, {mycache_app, []}}]}.

The last module is optional, it provides a quick way to load our application on VM startup

-module(mycache_boot).
-export([start/0]).

start() ->
    application:start(mycache).

That's it. Compile all these modules and copy binaries to all machines in the cluster. Place the binaries in the same folder you created Mnesia configuration.

Run Erlang application

Start Erlang VMs and load the application

ubuntu$ erl -sname master -s mycache_boot
Erlang R13B01 (erts-5.7.2) [source] [rq:1] [async-threads:0] [kernel-poll:false]

macBook$ erl -sname slave1 -s mycache_boot
Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

iMac$ erl -sname slave2 -s mycache_boot
Erlang R13B03 (erts-5.7.4) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

The cache is ready. You can start using it

(slave1@macBook)1> mycache:put("mykey", "myvalue").
null
(slave2@iMac)1> mycache:get("mykey").
"myvalue"
(master@ubuntu)1> mycache:put("mykey", "newvalue").
"myvalue"
(slave1@macBook)2> mycache:remove("mykey").
"newvalue"
(master@ubuntu)2> mycache:get("mykey").
null

It works! So, what do we actually achieve here with about 100 lines of Erlang code and bit of scripting?

• Distribution I run the app on three physical boxes, and it's transparent for the clients.
• Scaleability To add a new node to the cluster is just a matter of Mnesia re-configuration and copying of binary files to the new box.
• Concurrency Write and remove operations are transactional, and because of concurrent nature of Erlang itself our data is consistent and can be accessed by thousands of client processes.
• Fault tolerance Try to kill mycache process inside Erlang VM; it will be restarted automatically by supervisor and data will be replicated from other nodes to the new process.
• Persistence is optional and provided by Mnesia module.

All these benefits are given for free by Erlang/OTP, and it's not the end.

Call Erlang cache from Java

There are several ways of integrating Erlang applications with other languages. For Java the most convenient one is JInterface library. Here is the implementation of java.util.Map interface that communicates with the cache application we've just developed

import com.ericsson.otp.erlang.*;

public class ErlStringMap implements Map<String, String> {

    private final OtpSelf self;
    private final OtpPeer other;
    private final String cacheModule;

    public ErlStringMap(String client, String cookie, String serverNode, String cacheModule) {
        try {
            self = new OtpSelf(client, cookie);
            other = new OtpPeer(serverNode);
            this.cacheModule = cacheModule;
        } catch (Exception e) {
            throw new RuntimeException(e.getMessage(), e);
        }
    }

    public String put(String key, String value) {
        return remoteCall("put", key, value);
    }

    public String get(Object key) {
        return remoteCall("get", (String) key);
    }

    public String remove(Object key) {
        return remoteCall("remove", (String) key);
    }

    private String remoteCall(String method, String... args) {
        try {
            OtpConnection connection = self.connect(other);
            connection.sendRPC(cacheModule, method, stringsToErlangStrings(args));
            OtpErlangObject received = connection.receiveRPC();
            connection.close();
            return parse(received);
        } catch (Exception e) {
            throw new RuntimeException(e.getMessage(), e);
        }
    }

    private OtpErlangObject[] stringsToErlangStrings(String[] strings) {
        OtpErlangObject[] result = new OtpErlangObject[strings.length];
        for (int i = 0; i < strings.length; i++) result[i] = new OtpErlangString(strings[i]);
        return result;
    }

    private String parse(OtpErlangObject otpObj) {
        if (otpObj instanceof OtpErlangAtom) {
            OtpErlangAtom atom = (OtpErlangAtom) otpObj;
            if (atom.atomValue().equals("null")) return null;
            else throw new IllegalArgumentException("Only atom null is supported");

        } else if (otpObj instanceof OtpErlangString) {
            OtpErlangString str = (OtpErlangString) otpObj;
            return str.stringValue();
        }
        throw new IllegalArgumentException("Unexpected type " + otpObj.getClass().getName());
    }

    // Other methods are omitted
}

Now from the Java application we can use our distributed cache same way we are using HashMap

String cookie = FileUtils.readFileToString(new File("/Users/andrey/.erlang.cookie"));
Map<String, String> map = new ErlStringMap("client1", cookie, "slave1@macBook", "mycache");
map.put("foo", "bar")

Performance?

Let's deploy Erlang and Java nodes following this topology

Here is the speed of the cache operations I get in the Java client:

write 30.385 ms
read 1.23 ms
delete 21.665 ms

If we remove network, i.e. move all VMs, Java and Erlang, to the same box, we'll get the following performance:

write 2.091 ms
read 1.35 ms
delete 2.057 ms

And if we also disable persistence, the numbers will be

write 1.75 ms
read 1.38 ms
delete 1.75 ms

As you can see, performance is not the best, but keep in mind that the purpose of this post is not to build production ready cache application, but show the power of Erlang/OTP in building distributed fault-tolerant systems. As an exercise, try to implement the same functionality using JDK only.

Resources

• Source code used in the blog.

• Upcoming book where authors seem to implement similar application.

Ralph Johnson on learning to program

2009-12-04T22:30:00.002-05:00

In general, there are 2 types of books. There is theory books and there is practical books. You can learn theory from a book. You cannot learn anything practical from a book. You learn practical things by practicing. Practical books are great, you read them, they tell you some ideas, but they tell you ideas of things to practice and you have to go off and practice it. It's only after you practice that you actually know. Then you'll say "Oh, now I know what that guy meant when he said that!"

That's how you learn. You only learn that stuff by doing. It's like any program - you can't actually learn to program by reading a book. Of course you read the book, but you have to get to the system and you write little programs and you write bigger and bigger programs and after a while you know how to program in the language, but you've got to do it to learn. You are not really going to learn by just reading the book.

When I was young I was a sort of person who listened to older people. Some young people don't, but I was a young person who did, but even so, they would tell me a lot of things. I would remember it, I wasn't like I disbelieved them, but it just a little bit hard to believe and I wasn't sure. Now, I'm over 50 and a lot of stuff makes an awful lot more sense to me and I say "Those old guys were right when telling me things", but you can't until you live it, until you go through it yourself, you really can't know it for sure.

When you get a pattern book, you need to be bold. You need to just try the stuff out and you need to expect you are going to make mistakes, too, so don't try it out when someone's life is on the line. You try it out in the privacy of your own home, where you are not going to damage anybody too much. I think that's actually a problem with industry - companies don't let people just experiment and play around. Everything is "This is production code, we're going live next week and here is a book, read that and figure out how to put it in there." It doesn't matter what the book is on - whether it's going to be a database, or parallel programming, you have to do it 3 or 4 times until you really get comfortable with it and people are always putting this code out there that they just barely figured out how to make it run. That's not smart. Managers should give people more time to get it right before they push it out to the world. Some do, but probably yours doesn't, if yours is like most of them.

watch the interview

Finding first broken build with git-bisect

2009-11-26T20:32:00.005-05:00

With git it's so easy to find which commit broke the build. Suppose you know that commit with tag release-1.0 was good, i.e. all tests passed. The latest commit, however, is failing.

So the build was broken somewhere in between release-1.0 and master HEAD. To find the exact one you just need to run two git commands. First

$ git bisect start HEAD release-1.0

Here you specify which commit you know is broken, and which one you remember was successful. After running this command you can see that git marked two commits with good and bad labels, and put the search starting point at the revision in the middle.

The second command is git bisect run cmd, where cmd is the script used as a criterion of successful build. If you use Maven then your command would be:

$ git bisect run mvn clean test

After you hit Enter button in the terminal, git starts running your criterion script using binary search algorithm. It might take time to finish this task, which depends on the number of revisions and the execution time of the script. Eventually it stops and shows you which commit caused the build failure.

7933e4658ea852754120fbc8fec34b2b85932e48 is first bad commit
commit 7933e4658ea852754120fbc8fec34b2b85932e48
Author: Andrey Paramonov 
Date:   Wed Nov 25 21:07:30 2009 -0500

    Changed method implementation

:040000 040000 c0f3f9ef13d7daa4671205b9518c168a9ac10fe3 5a1cf3d6fb28d6f815c172319630b5d55ce4dc10 M     src
bisect run success

If you look at the visual tool, you will spot the first bad commit by the label bisect/bad.

Resources

• git-bisect manual page

State Machines in Erlang

2009-11-12T22:43:00.003-05:00

There is some sort of confusion in the object-oriented community about functional languages. How is it possible to implement stateful application if the language has no concept of state? It turns out that it's actually quite possible, although the solution is completely deferent from what we see in the OO realm. In Erlang, for example, state can be implemented by using process messaging and tail recursion. This approach is so elegant that after you've learned it, the OO way of doing this looks unnatural. The code below is the Erlang implementation of Uncle Bob's FSM example. Look at it. Isn't that code clean and expressive? It looks almost like DSL but it's actually regular Erlang syntax.

-module(turnstile).
-export([start/0]).
-export([coin/0, pass/0]).
-export([init/0]).

start() -> register(turnstile, spawn(?MODULE, init, [])).

% Initial state

init() -> locked().

% Events

coin() -> turnstile ! coin.
pass() -> turnstile ! pass.

% States and transitions

locked() ->
    receive
        pass ->
            alarm(),
            locked();
        coin ->
            unlock(),
            unlocked()
    end.

unlocked() ->
    receive
        pass ->
            lock(),
            locked();
        coin ->
            thankyou(),
            unlocked()
    end.

% Actions

alarm() -> io:format("You shall not pass!~n").
unlock() -> io:format("Unlocking...~n").
lock() -> io:format("Locking...~n").
thankyou() -> io:format("Thank you for donation~n").

The idea behind this code is simple. Every state is implemented as a function that does two things: it listens for messages sent by other processes; when message is received the appropriate action is taken and one of the state-functions called recursively. Simple, right? And thread-safe!

Parsing files using Groovy regex

2009-10-22T19:36:00.005-04:00

In my previous post I mentioned several ways of defining regular expressions in Groovy. Here I want to show how we can use Groovy regex to find/replace data in the files.

Parsing properties file (simplified)¹

Data: each line in the file has the same structure; the entire line can be matched by single regex. Problem: transform each line to the object. Solution: construct regex with capturing parentheses, apply it to each line, extract captured data. Demonstrates: File.eachLine method, matrix syntax of Matcher object.

def properties = [:]
new File('path/to/some.properties').eachLine { line ->
    if ((matcher = line =~ /^([^#=].*?)=(.+)$/)) {
        properties[matcher[0][1]] = matcher[0][2]
    }
}
println properties

Parsing csv files (simplified)²

Data: each line in the file has the same structure; the line consists of the blocks separated by some character sequence. Problem: transform each line to the list of objects. Solution: construct regex with capturing parentheses, parse each line with the regex in a loop extracting captured data. Demonstrates: ~// Pattern defenition, Matcher.group method, \G regex meta-sequence.

def regex = ~/\G(?:^|,)(?:"([^"]*+)"|([^",]*+))/
new File('path/to/file.csv').eachLine { line ->
    def fields = []
    def matcher = regex.matcher(line)
    while (matcher.find()) {
        fields << (matcher.group(1) ?: matcher.group(2))
    }
    println fields
}

Finding snapshot dependencies in the pom (simplified)³

Data: file contains blocks with known boundaries (possibly crossing multiple lines). Problem: extract the blocks satisfying some criteria. Solution: read the entire file into the string, construct regex with capturing parentheses, apply the regex to the string in a loop. Demonstrates: File.text property, list syntaxt of Matcher object, named capture, global \x regex modifier, local \s regex modifier.

def pom = new File('path/to/pom.xml').text
def matcher = pom =~ '''(?x)
    <dependency>                          \\s*
      <groupId>([^<]+)</groupId>          \\s*
      <artifactId>([^<]+)</artifactId>    \\s*
      <version>(.+?-SNAPSHOT)</version>   (?s:.*?)
    </dependency>
'''
matcher.each { matched, groupId, artifactId, version ->
    println "$groupId:$artifactId:$version"
}

Finding stacktraces in the log

Data: file contains entries each of which starts with the same pattern and can span multiple lines. Typical example is log4j log files:

2009-10-16 15:32:12,157 DEBUG [com.ndpar.web.RequestProcessor] Loading user
2009-10-16 15:32:13,258 ERROR [com.ndpar.web.UserController] id to load is required for loading
java.lang.IllegalArgumentException: id to load is required for loading
     at org.hibernate.event.LoadEvent.(LoadEvent.java:74)
     at org.hibernate.event.LoadEvent.(LoadEvent.java:56)
     at org.hibernate.impl.SessionImpl.get(SessionImpl.java:839)
     at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
     at org.springframework.orm.hibernate3.HibernateTemplate$1.doInHibernate(HibernateTemplate.java:531)
     at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
     at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
     at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:525)
     at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:519)
     at com.ndpar.dao.UserManager.getUser(UserManager.java:90)
     ... 62 more
2009-10-16 15:32:14,659 DEBUG [com.ndpar.jms.MessageListener] Received message:
    ... multi-line message ...
2009-10-16 15:32:15,169 INFO  [com.ndpar.dao.UserManager] User: ...

Problem: find entries satisfying some criteria. Solution: read the entire file into the string⁴, construct regex with capturing parentheses and lookahead, split the string into entries, loop through the result and apply criteria to each entry. Demonstrates: regex interpolation, combined global regex modifiers \s and \m.

def log = new File('path/to/your.log').text
def logLineStart = /^\d{4}-\d{2}-\d{2}/
def splitter = log =~ """(?xms)
    (    ${logLineStart}   .*?)
    (?=  ${logLineStart} | \\Z)
"""
splitter.each { matched, entry ->
    if (entry =~ /(?m)^(?:\t| {8})at/) println entry
}

Replacing text in the file

Use Groovy one-liner to perform the replacement. Here is the Tim's example in Groovy:

$ groovy -p -i -e '(line =~ /1\.6/).replaceAll("2.0-alpha-1-SNAPSHOT")' `find . -name pom.xml`

Resources

• Groovy regexes
• Groovy one-liners
• Using String.replaceAll method

Footnotes

1. This example is for demonstration purposes only. In real program you would just use Properties.load method.
2. The regex is simplified. If you want the real one, take a look at Jeffrey Friedl's example.
3. Again, in reality you would find snapshots using mvn dependency:resolve | grep SNAPSHOT command.
4. This approach won't work for big files. Take a look at this script for practical solution.

GParallelizer Performance

2009-10-14T22:44:00.018-04:00

GParallelizer is a Groovy wrapper for new Java concurrency library. It allows you to perform list and map operations using parallel threads, which in theory leverages the full power of multi-processor computations. Here I want to check if it's true in reality. I run the following tests on my dual-core MacBook

import static org.gparallelizer.Parallelizer.*
import org.gparallelizer.ParallelEnhancer
import org.junit.Before
import org.junit.Test

class GParsTest {

    def list = []

    @Before void setUp() {
        1000000.times {
            list << (float) Math.random()
        }
    }

    @Test void sequential() {
        def start = System.currentTimeMillis()
        list.findAll { it < 0.4 }
        def duration = System.currentTimeMillis() - start

        println "Sequential: ${duration}ms"
    }

    @Test void parallel_with_enhancer() {
        ParallelEnhancer.enhanceInstance list

        def start = System.currentTimeMillis()
        list.findAllAsync { it < 0.4 }
        def duration = System.currentTimeMillis() - start

        println "Parallel with enhancer: ${duration}ms"
    }

    @Test void parallel_with_parallelizer_2() {
        parallelWithParallelizer 2
    }

    @Test void parallel_with_parallelizer_3() {
        parallelWithParallelizer 3
    }

    @Test void parallel_with_parallelizer_5() {
        parallelWithParallelizer 5
    }

    @Test void parallel_with_parallelizer_10() {
        parallelWithParallelizer 10
    }

    def parallelWithParallelizer(threads) {
        def start = System.currentTimeMillis()
        withParallelizer(threads) {
            list.findAllAsync { it < 0.4 }
        }
        def duration = System.currentTimeMillis() - start

        println "Parallel with parallelizer (${threads}): ${duration}ms"
    }
}

And here is the output

Sequential: 774ms
Parallel with enhancer: 9311ms
Parallel with parallelizer (2): 1785ms
Parallel with parallelizer (3): 769ms
Parallel with parallelizer (5): 500ms
Parallel with parallelizer (10): 722ms

Something strange happened with mixed-in ParallelEnhancer, but with Parallelizer performance improved indeed. With optimal thread pool size parallel processing is 35% faster than sequential.

Conclusion: Use GPars methods if you need to process big amount of data. Try different config parameters to find the best solution for your particular problem.

Resources

• Brian Goetz on new concurrency library
• Vaclav Pech on GPars

Converting XML to POGO

2009-10-06T19:34:00.011-04:00

Suppose we want to convert XML to Groovy bean:

class MyBean {
    String strField
    float floatField
    int intField
    boolean boolField
}

def message = "<xml stringAttr='String Value' boolAttr='true' />"

def xml = new XmlSlurper().parseText(message)

def bean = new MyBean(
    strField: xml.@stringAttr,
    boolField: xml.@boolAttr
)

Everything looks good, even assertions succeed

assert 'String Value' == bean.strField
assert bean.boolField

Now let's try false value:

message = "<xml stringAttr='String Value' boolAttr='false' />"
assert !bean.boolField

Oops, the assertion failed. Why? Because xml.@boolAttr cast to boolean always returns true. The correct implementation must be like this:

message = "<xml stringAttr='String Value' floatAttr='3.14' intAttr='9' boolAttr='false' />"

xml = new XmlSlurper().parseText(message)

bean = new MyBean(
    strField: xml.@stringAttr.toString(),
    floatField: xml.@floatAttr.toFloat(),
    intField: xml.@intAttr.toInteger(),
    boolField: xml.@boolAttr.toBoolean()
)
assert 'String Value' == bean.strField
assert 3.14F == bean.floatField
assert 9 == bean.intField
assert !bean.boolField

Now everything works properly. The moral of this blog post: Create more unit tests (assertions), especially when you work with dynamic language.

Resources

• Converting String to Boolean

Building regular expressions in Groovy

2009-09-25T21:43:00.009-04:00

Because of compact syntax regular expressions in Groovy are more readable than in Java. Here is how Jeffrey Friedl's example would look like in Groovy:

def subDomain  = '(?i:[a-z0-9]|[a-z0-9][-a-z0-9]*[a-z0-9])' // simple regex in single quotes
def topDomains = """
       (?x-i : com        \\b     # you can put whitespaces and comments
             | edu        \\b     # inside regex in eXtended mode
             | biz        \\b
             | in(?:t|fo) \\b     # but you have to escape
             | mil        \\b     # backslashes in multiline strings
             | net        \\b
             | org        \\b
             | [a-z][a-z] \\b
       )"""

def hostname = /(?:${subDomain}\.)${topDomains}/  // variable substitution in slashy strings

def NOT_IN   = /;\"'<>()\[\]{}\s\x7F-\xFF/  // backslash is not escaped in slashy strings
def NOT_END  = /!.,?/
def ANYWHERE = /[^${NOT_IN}${NOT_END}]/
def EMBEDDED = /[$NOT_END]/                 // you can ommit {} around var name

def urlPath  = "/$ANYWHERE*($EMBEDDED+$ANYWHERE+)*"

def url =
    """(?x:
             \\b

             # match the hostname part
             (
               (?: ftp | http s? ): // [-\\w]+(\\.\\w[-\\w]*)+
             |
               $hostname
             )

             # allow optional port
             (?: :\\d+ )?

             # rest of url is optional, and begins with /
             (?: $urlPath )?
       )"""

assert 'http://www.google.com/search?rls=en&q=regex&ie=UTF-8&oe=UTF-8' ==~ url

As you can see, there are several options, and for every subexpression you can choose the one that's more expressive.

Resources

• Martin Fowler on composed regexes
• Pragmatic Dave on regexes in Ruby
• Feature request to make regexes even groovier
• Mastering Regular Expressions — best regex book
• Groovy Pattern and Matcher classes

Fill parameters in LCDS Assembler methods

2009-09-22T20:24:00.015-04:00

Last few days we spent debugging some nasty bug in the code that uses LiveCycle managed collections. We were adding/removing items to/from collections on the server side. We saw that server sent a message to the client, client did receive the message, but then it ignored it and didn't update the collection (data grid). After digging into the client side logs we found the reason of such a misbehaviour.

If you have two destinations that share the same channel

<service id="data-service" class="flex.data.DataService">
   <destination id="MyFirstDestination">
       <channels>
           <channel ref="my-rtmp-channel"/>
       </channels>
       <properties>
           <source>example.MyFirstAssembler</source>
       </properties>
   </destination>
   <destination id="MySecondDestination">
       <channels>
           <channel ref="my-rtmp-channel"/>
       </channels>
       <properties>
           <source>example.MySecondAssembler</source>
       </properties>
   </destination>
</service>

LCDS uses fillParameters as a key in the managed collections cache. That means fillParameters must be immutable.

public class MyFirstAssembler extends flex.data.assemblers.AbstractAssembler {

  @Override
  public int refreshFill(List fillParameters, Object newItem, boolean isCreate, Object oldItem, List changes) {
    // Never change fillParameters!
  }

  @Override
  public Collection fill(List fillParameters) {
    // Never change fillParameters!
  }
}

Adobe documentation says nothing about this, so keep this rule in mind when working with LCDS managed collections.

Resources

• Flex log viewer

Measuring LiveCycle Performance: Message Size

2009-08-20T19:15:00.007-04:00

The method of measuring performance provided by LCDS works only in situations when producer and consumer of messages are both on the Flex side. For Data Services that means you can obtain some metrics only for initial collection fill:

Original message size(B): 499
Response message size(B): 17687
Total time (s): -1250809384.8
Network Roundtrip time (s): -1250809384.868
Server processing time (s): 0.068
Server adapter time (s): 0.014
Server non-adapter time (s): 0.054

If you want to know message size and response time for messages pushed from Java server to Flex client, this method doesn't help* in the current version of LCDS (2.6.1). Adobe promised to add this feature in the future release but for now you have to use other methods. Here is what I use to measure message size.

1. JMX. By default LCDS exposes some useful metrics through JMX:

2. Flex log. If you enable log in the services-config.xml, you will see something like this in the output console for every data push:

Thread[1563082333@qtp0-0,5,main] registering write interest for Connection '1752654181'.
Thread[my-rtmp-SocketServer-Reactor1,5,main] unregistering write interest for Connection '1752654181'.
Thread[my-rtmp-SocketServer-Reactor1Writer,5,main] Connection '1752654181' starting a write.
Thread[my-rtmp-SocketServer-Reactor1Writer,5,main] chunk output stream writing message; ack state: 3
...
Thread[my-rtmp-SocketServer-Reactor1Writer,5,main] Connection '1752654181' finished a write. 233 bytes were written.

3. If you don't have access to the server, you can use any network protocol analyzer (WireShark is really good) on the client side to monitor size of packets received from the server.

* Actually, there is one undocumented feature that can be used with the described method to measure size of "create" messages, but Adobe does not recommend to use it.

Resources

• Part 1: Measuring LiveCycle Performance: Errors

Git on Cygwin

2009-08-13T20:07:00.003-04:00

Here are the steps I perform to use Git on Cygwin:

• Install necessary packages: git, gitk, subversion, subversion-perl. That's because I use Subversion via Git bridge.

• Amend C:\cygwin\lib\perl5\vendor_perl\5.xx\Git.pm file using this patch. That's to fix small bug causing "Permission denied: Can't open '/cygdrive/c/DOCUME~1/myself/LOCALS~1/Temp/report.tmp'" error.

• Configure white spaces using this command to fix trailing spaces warnings.

Double in ActionScript, Java, and MS SQL

2009-07-23T22:14:00.006-04:00

ActionScript 3

• There are three numeric data types in AS3: int, uint, and Number.
• They are not primitives because they can be instantiated using constructors.
• They are not "real" objects because they cannot be null, and they have default values:

myNumber:Number;
myNumber.toString(); // No NPE thrown

• Default value for type Number is NaN (not zero).

Java

• BlazeDS converts AS3 Number type to Java Double.
• NaN is idempotent of conversion:

NaN (Java) -> NaN (AS3) -> NaN (Java)

• null is not! Keep it in mind when you work with BlazeDS:

null (Java) -> 0 (AS3) -> 0.0 (Java)

If Java NaN doesn't have special meaning in your application, you can use it as a "replacement" for null in Java-Flex communication.

MS SQL

• Doesn't support NaN value for numeric columns.
• All NaN values should be replaced by null before saving entity in the database, otherwise you will get exception:

com.microsoft.sqlserver.jdbc.SQLServerException: The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Parameter 24 (""): The supplied value is not a valid instance of data type real. Check the source data for invalid values. An example of an invalid value is data of numeric type with scale greater than precision.

In my current project I'm using all three languages, and I have to convert NaN to null back and forth for every object:

NaN (AS3) <-> NaN (Java) <-> null (Java) <-> null (MS SQL)

So I created small utility class that replaces all JavaBean properties of particular type from one value to another:

ExtendedPropertyUtils.replacePropertyValue(myBean, Double.NaN, null);
ExtendedPropertyUtils.replacePropertyValue(myBean, null, Double.NaN);

Feel free to use it if you have the same problem.

Resources

• Feature request to Adobe to introduce nullable Number type.
• Other solutions for similar issues in BlazeDS.

Tom DeMarco on Software Engineering

2009-07-20T20:36:00.003-04:00

Software development is inherently different from a natural science such as physics, and its metrics are accordingly much less precise in capturing the things they set out to describe.
...
Strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value.
To my mind, the question that’s much more important than how to control a software project is, why on earth are we doing so many projects that deliver such marginal value?
...
So, how do you manage a project without controlling it? Well, you manage the people and control the time and money. You say to your team leads, for example, “I have a finish date in mind, and I’m not even going to share it with you. When I come in one day and tell you the project will end in one week, you have to be ready to package up and deliver what you’ve got as the final product. Your job is to go about the project incrementally, adding pieces to the whole in the order of their relative value, and doing integration and documentation and acceptance testing incrementally as you go.”
...
For the past 40 years we’ve tortured ourselves over our inability to finish a software project on time and on budget. But this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

read the whole article

Michael Feathers on Functional Programming

2009-07-17T21:07:00.005-04:00

One of the things that seems like a rather pessimistic observation, but I think it's true to a degree, that the number of programmers who are able to or willing to think in a mathematically sophisticated way about code is relatively small, in comparison to the total population of programmers. I think that even though functional programming is becoming more popular, it is a bit of uphill battle for the industry and it may become just a very strong good niche tool for the people who are able to use it very well. I'm glad to see it's being brought up in prominence now, but I'm wondering if we'll ever see a day when everybody is doing work in functional programming. On the other hand, we got to the point where closures are becoming part of practically every programming language. It only took 30 years, so there is hope, I guess.

watch the interview

Spring integration tests with mocked collaborators

2009-07-16T22:29:00.006-04:00

Sometimes when you write integration tests that load Spring application context, you want to mock some collaborators just to verify that they've been called with correct data. This is a quite legitimate approach, but you have to keep in mind that Spring caches application contexts during the test execution. That means, after you replace a bean with a mock, all subsequent tests that share the same application context will use mocked instance instead of real one. That might cause tests to fail in very peculiar way. For example, test succeeds in Eclipse but fails in command line; you create new test and the old one starts failing, etc.

To fix the problem, you need to reload application context after every test that uses mocks. The best way to do this is to use @DirtiesContext annotation. In Spring 2.5 this was a method level annotation, but starting with Spring 3.0RC1 you can use it on the class level (thanks Spring!). So the rule of thumb is:

If you mock a bean in the Spring integration test, annotate the test class with @DirtiesContext

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations={"classpath:appclicationContext.xml"})
@DirtiesContext
public class IntegrationTest {

    @Autowired
    private Application application;

    private Collaborator mockCollaborator;

    @Before
    public void setUp() {
        mockCollaborator = mock(Collaborator.class);
        application.setCollaborator(mockCollaborator);
    }

    @Test
    public void collaborator_is_called_once() {
        ...
        verify(mockCollaborator, times(1)).methodCall(...);
    }
}

Resources

• Source files for this post
• Spring annotations documentation
• Class level @DirtiesContext annotation
• How to build Spring 3

Iteration Length

2009-07-07T19:12:00.003-04:00

Iteration length in agile software development is the time between two consecutive customer feedbacks.

You can set up iteration length to 2 weeks in the JIRA, but if you meet your customer twice a year then your iteration length is 6 months. You can do iteration demo every 2 weeks, but if nobody except your manager sees the demo then your iteration length is not 2 weeks. You can even release some artifacts and change the version of your application every 2 weeks, but if your customer never tried to use your application then your iteration length is definitely not 2 weeks.

In general, without customer feedback there is no iteration. It's a cascade.

Tony Hoare: 50 Years with Legacy Code

2009-07-06T20:11:00.006-04:00

Q: With your time in the computer science field, I'm guessing that you've seen some common trends, things that have remained consistent throughout time since 1960. What are those trends that have remained consistent and how do you think it will continue to the future?

A: I'm sorry to say it's the mistakes that remained consistent, not the virtues. The problems that afflicted us in the 1960s were the difficulty of predicting performance of large systems, the difficulty of discovering requirements, the difficulty of implementing code that was coherent across large-scale module boundaries. All of these things are still with us. I suppose I could say that even in 1960 living with legacy code was there. Dykstra once said that every day when he comes into work he thanks providence that he is not charged with the responsibility of living with legacy code - that's certainly still with us.

watch the interview

Collaborating Using git-svn

2009-07-03T20:22:00.008-04:00

I like Git, and I'm using it everywhere. It gives you so much power that once you taste it, you won't want to come back to traditional source control systems. One of the Git benefits is collaboration friendliness. Git encourages collaboration, ideas exchange, and code review. If your team is using Git then you know how easy it is to share your code with your co-workers. But sometimes you are the only person in the team who uses Git, and everybody else is on Subversion. Don't worry, you still can share your ideas by means of git-svn tool, and here I want to show you how. The process is not as simple as native Git (via pull/push or patch/apply) but it's better than nothing.

Suppose you have an idea and you are eager to try it. You don't want to create a branch in Subversion because you don't know if your idea will work out, and committing all your crazy stuff in Subversion can easily pollute it. So you create a local Git branch and start working.

$ git checkout -b topic/great-idea

You code, test, git-add, git-commit, code, test, etc. At some point you see that your idea was great indeed and it's time to show the amazing results to your teammates. Now you need to "push" your Git branch to Subversion. To do this you have to create Subversion brunch first

$ svn copy http://svn.repo.path/project-name/trunk \
http://svn.repo.path/project-name/branches/great-idea \
-m "Created branch for my cool stuff"

Next step is to add this Subversion branch as a remote branch to Git configuration. Open .git/config file with a text editor. You should see something like this

[core]
       repositoryformatversion = 0
       filemode = true
       bare = false
       logallrefupdates = true
[svn-remote "svn"]
       url = http://svn.repo.path
       fetch = project-name/trunk:refs/remotes/trunk

If you built your Git repository by cloning Subversion repository (which you most likely did), you will have one or more svn-remote sections in this configuration file. You need to add another one for new Subversion branch.

[core]
       repositoryformatversion = 0
       filemode = true
       bare = false
       logallrefupdates = true
[svn-remote "svn"]
       url = http://svn.repo.path
       fetch = project-name/trunk:refs/remotes/trunk
[svn-remote "svn-great-idea"]
       url = http://svn.repo.path
       fetch = project-name/branches/great-idea:refs/remotes/great-idea

Next step is to fetch Subversion branch, but first you need to know revision number when this branch was created. Run this command

$ svn log --stop-on-copy http://svn.repo.path/project-name/branches/great-idea

You should get something like this:

------------------------------------------------------------------------
r2165 | andy | 2009-07-03 14:36:59 -0400 (Fri, 03 Jul 2009) | 1 line
 
Created branch for my cool stuff
------------------------------------------------------------------------

The first number of the output is what we are looking for:

$ git svn fetch svn-great-idea -r2165

Now you have the Subversion branch in you Git. If you run git branch -a command you will see "great-idea" branch in the list.

 master
* topic/great-idea
 great-idea
 trunk

You shouldn't work on remote branch, so let's create local one:

$ git branch svn-branch/great-idea remotes/great-idea

I put it in svn-branch namespace just to make it visually clear that this branch is in synch with Subversion. The next set of commands is a standard way to bring your work from one local branch to another. In our case: from initial topic/great-idea to svn-branch/great-idea

$ git rebase svn-branch/great-idea
$ git checkout svn-branch/great-idea
$ git rebase topic/great-idea

That's it. Now you are ready to commit your code to Subversion:

$ git svn dcommit

Done. As an option, you can delete initial branch because you have a Subversion backed copy of it:

$ git branch -D topic/great-idea

Resources

• Ian Boston's post explaining how to add Subversion branches to Git.

Measuring LiveCycle Performance: Errors

2009-06-30T20:51:00.015-04:00

There are several ways to measure LiveCycle performance. One of them is to call appropriate method on MessagePerformanceUtils class. This approach is pretty straightforward but sometimes you might get an error:

Destination '...' either does not exist or the destination has no channels defined (and the application does not define any default channels.)

That means your are using statically configured channels and you don't package services configuration into the SWF file. To fix it, in the Flex Builder add config files folder to Flex source path and specify 'services' compiler argument:

Although it solves the problem, this approach is not suitable for real project as you don't want to compile SWF file with hard coded services configuration. Instead of that you would create dynamic channels on the client side, and configure them using IoC framework (i.e. Parsley, Prana or your own). And if you do that you will most likely get the following error:

Error: Message is missing MPI headers. Verify that all participants have it enabled

The reason of that is: you configured MPI headers only on the server side, but not on the Flex side. To fix it, you need to set recordMessageTimes and recordMessageSizes properties of Channel class to true. The problem is that those properties are read-only, so you cannot assign them to any value directly. But here is a trick: you can use applySettings() method:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" creationComplete="init()">
    <mx:Script>
        <![CDATA[
            import mx.messaging.Channel;
            import mx.messaging.ChannelSet;
            import mx.messaging.channels.RTMPChannel;
            import mx.messaging.events.MessageEvent;
            import mx.messaging.messages.MessagePerformanceUtils;

            private function init():void {
                ds.channelSet = createChannelSet();
            }

            private function createChannelSet():ChannelSet {
                var channels:Array = new Array();
                channels.push(createRtmpChannel());

                var result:ChannelSet = new ChannelSet();
                result.channels = channels;
                return result;
            }

            private function createRtmpChannel():Channel {
                var result:Channel = ... // get it from IoC
                result.applySettings(customSettings());
                return result;
            }

            private function customSettings():XML {
                return <channel-definition>
                    <properties>
                        <record-message-times>true</record-message-times>
                        <record-message-sizes>true</record-message-sizes>
                    </properties>
                </channel-definition>;
            }

            private function messageHandler(event:MessageEvent):void {
                var performanceUtils:MessagePerformanceUtils = new MessagePerformanceUtils(event.message);
                statistics.text = performanceUtils.prettyPrint();
            }
        ]]>
    </mx:Script>

    <mx:DataService id="ds" destination="MyDestination" result="messageHandler(event)" />
    <mx:ArrayCollection id="domainObjects" />
    <mx:TextArea id="statistics" />
</mx:Application>

Resources

• Check out example sources from GitHub.

Let's get started

2006-06-21T16:09:00.002-04:00

This blog becomes a successor to my jroller.