Exporting Solr documents

Recently I had to copy some documents from one Solr server to another. I expected Solr already had an interface that allowed me to extract documents in the same format they were inserted. In that case I would pipe an output of one curl command to another, and consider the job done. As it turned out, the format of Solr input document is different than the output format. Here is how input document looks like:

<add>
    <doc>
        <field name="id">12345</field>
        <field name="articlestate">published</field>
        <field name="articletype">news</field>
        <field name="body">Lorem ipsum dolor...</field>
        <field name="referenceid">175820</field>
        <field name="referenceid">163786</field>
        <field name="created">2011-02-15T14:57:54.766Z</field>
    </doc>
</add>

Notice the flat structure of this document: all element names are the same regardless of the filed type, and arrays (referenceid) are not grouped. Now compare it to the output format. Here is what you get when you execute a query against a Solr server:

<response>
    <lst name="responseHeader">
        <int name="status">0</int>
        <int name="QTime">1</int>
        <lst name="params">
            <str name="q">id:12345</str>
        </lst>
    </lst>
    <result name="response" numFound="1" start="0">
        <doc>
            <str name="id">12345</str>
            <str name="articlestate">published</str>
            <str name="articletype">news</str>
            <str name="body">Lorem ipsum dolor...</str>
            <arr name="referenceid">
                <str>175820</str>
                <str>163786</str>
            </arr>
            <date name="created">2011-02-15T14:57:54.766Z</date>
        </doc>
    </result>
</response>

Even if we ignore the response header, the structure of the response/result/doc is not the same as of input document: the element names reflect the types, the arrays are grouped. If you try to add this document to a Solr server, you will get an error "unexpected XML tag", obviously. I googled for couple hours on how to convert an output document to an input, and, to my surprise, didn't find any solution. (If you happen to know the solution, please leave a reference in the comments.)

Anyways, I implemented my own converter in Groovy, which solved the problem. I post it here in case somebody needs it.

Note: You can also use this script to re-index Solr.

Functional Groovy switch statement

In the previous post I showed how to replace chained if-else statements in Groovy with one concise switch. It was done for the special case of if-stement where every branch was evaluated using the same condition function. Today I want to make a generalization of that technique by allowing to use different conditionals.

Suppose your code looks like this:

if (param % 2 == 0) {
    'even'
} else if (param % 3 == 0) {
    'threeven'
} else if (0 < param) {
    'positive'
} else {
    'negative'
}

As soon as every condition operates on the same parameter, you can replace the entire chain with a switch. In this scenario param becomes a switch parameter and conditions become case parameters of Closure type. The only thing we need to do is to override Closure.isCase() method as I described in the previous post. The safest way to do it is to create a category class:

class CaseCategory {
    static boolean isCase(Closure casePredicate, Object switchParameter) {
        casePredicate.call switchParameter
    }
}

Now we can replace if-statement with the following switch:

use (CaseCategory) {
    switch (param) {
        case { it % 2 == 0 } : return 'even'
        case { it % 3 == 0 } : return 'threeven'
        case { 0 < it }      : return 'positive'
        default              : return 'negative'
    }
}

We can actually go further and extract in-line closures:

def even = {
    it % 2 == 0
}
def threeven = {
    it % 3 == 0
}
def positive = {
    0 < it
}

After which the code becomes even more readable:

use (CaseCategory) {
    switch (param) {
        case even     : return 'even'
        case threeven : return 'threeven'
        case positive : return 'positive'
        default       : return 'negative'
    }
}

Multimethods in Groovy

Every time I switch from Groovy to Java I have to remind myself that some things that seem so natural and work as expected in Groovy, don't work in Java. One of such differences is method dispatching. Groovy supports multiple dispatch, while Java does not. Therefore the following code works differently in Groovy and Java:

public class A {
    public void foo(A a) { System.out.println("A/A"); }
    public void foo(B b) { System.out.println("A/B"); }
}
public class B extends A {
    public void foo(A a) { System.out.println("B/A"); }
    public void foo(B b) { System.out.println("B/B"); }
}
public class Main {
    public static void main(String[] args) {
        A a = new A();
        A b = new B();
        a.foo(a);
        b.foo(b);
    }  
}

$ java Main
A/A
B/A

$ groovy Main.groovy
A/A
B/B

Reversing Groovy switch statement

Recently I've been working on a Groovy code that had many methods with long multibranch conditionals like this:

def parse(message, options) {
    if (options.contains('A')) {
        parseARule message
    } else if (options.contains(2)) {
        parseSmallDigitRule message
    ...
    } else if (options.contains(something)) {
        parseSomeRule message
    } else {
        parseSomeOtherRule message
    }
}

Although this code is working, it is hard to see which branch is called under which condition. It would be much better if we could replace this code with something like Lisp cond macro. The best candidate for such a task in Groovy would be a switch statement. If we could only refactor the code above to something like following, it would significantly improve readability:

def parse(message, options) {
    switch (options) {
        case 'A' : return parseARule(message)
        case 2   : return parseSmallDigitRule(message)
        ...
        case ... : return parseSomeRule(message)
        default  : return parseSomeOtherRule(message)
    }
}

Unfortunately, this code doesn't work out of the box in Groovy, but it works if we do some metaprogramming.

The way switch statement works in Groovy is a bit different than in Java. Instead of equals() it uses isCase() method to match case-value and switch-value. The default implementation of isCase() method falls back to equals() method, but some classes, including Collection, override this behaviour. That's why in Groovy you can do things like this:

switch (value) {
    case ['A','E','I','O','U'] : return 'vowel'
    case 0..9                  : return 'digit'
    case Date                  : return 'date'
    default                    : return 'something else'
}

For our purposes we need some sort of reverse switch, where collection is used as a switch-value, and String and Integer are used as a case-value. To do this we need to override default implementation of isCase() method on String and Integer classes. It's not possible in Java, but is very easy in Groovy. You can change method implementation globally by replacing it in corresponding meta class, or locally with the help of categories. Let's create a category that swaps object and subject of isCase() method:

class CaseCategory {
    static boolean isCase(String string, Collection col) {
        reverseCase(string, col)
    }
    static boolean isCase(Integer integer, Collection col) {
        reverseCase(integer, col)
    }
    // Add more overloaded methods here if needed

    private static boolean reverseCase(left, right) {
        right.isCase(left)
    }
}

Now we can use this category to achieve the goal we stated at the beginning of this post:

def parse(message, options) {
    use (CaseCategory) {
        switch (options) {
            case 'A' : return parseARule(message)
            case 2   : return parseSmallDigitRule(message)
            ...
            case ... : return parseSomeRule(message)
            default  : return parseSomeOtherRule(message)
        }
    }
}

If you are comfortable with global method replacement, you can amend String and Integer meta classes. In this case you don't need to wrap switch statement with use keyword.

Anyways, with or without category, the final code looks better than the original noisy if-else chain. And you have learned the technique of reversing switch statement.

Lazy lists in Groovy

I like lazy evaluation, and it's one of the reasons I like Haskell language so much. Although from engineering perspective lazy evaluation is probably not the most needed feature, it's definitely very useful for solving some mathematical problems.

Most languages don't have lazy evaluation out of the box, but you can implement it using some other language features. This is an interesting task, and I use it as a code kata which I practice every time I learn a new strict language.

So, how to implement lazy lists in strict languages? Very simple, if the language has functional capabilities. Namely, you build lazy list recursively by wrapping strict list within a function. Here is, for example, the strict empty list in Groovy:

[]

If we wrap it with a closure, it becomes lazy empty list:

{-> [] }

If we need a list with one element, we prepend (or speaking Lisp terminology 'cons') an element to lazy empty list, and make the result lazy again:

{-> [ element, {-> [] } ] }

To add more elements we continue the same process until all elements are lazily consed. Here is, for example, a lazy list with three elements a, b and c:

{-> [a, {-> [b, {-> [ c, {-> [] } ] } ] } ] }

Now, when you have an idea how to build lazy lists, let's build them Groovy way. We start with creating a class:

class LazyList {
    private Closure list

    private LazyList(list) {
        this.list = list
    }
}

The variable list encapsulates the closure wrapper of the list. We just need to expose some methods that allow constructing lists using procedure described above:

    static LazyList nil() {
        new LazyList( {-> []} )
    }

    LazyList cons(head) {
        new LazyList( {-> [head, list]} )
    }

Now we can construct lists by consing elements to empty list:

def lazylist = LazyList.nil().cons(4).cons(3).cons(2).cons(1)

To access elements of the list we implement two standard functions, car and cdr, that return head and tail of the list respectively.

    def car() {
        def lst = list.call()
        lst ? lst[0] : null
    }

    def cdr() {
        def lst = list.call()
        lst ? new LazyList(lst.tail()[0]) : nil()
    }

Here is how you use these functions to get first and second elements of the list constructed above

assert lazylist.car() == 1
assert lazylist.cdr().car() == 2

In Lisp there are built-in functions for various car and cdr compositions. For example, the previous assertion would be equivalent to function cadr. Instead of implementing all possible permutations, let's use Groovy metaprogramming to achieve the same goal.

    def methodMissing(String name, args) {
        def matcher = name =~ /c([ad])([ad]+)r/
        if (matcher) {
            matcher[0][2].reverse().toList().inject(this) {
                del, index -> del."c${index}r"()
            }."c${matcher[0][1]}r"()
        } else {
            throw new MissingMethodException(name, this.class, args)
        }
    }

It might look complicated, but in reality it's pretty simple if you are familiar with Groovy regex and functional programming. It's easier to explain by example. If we pass "caddr" as a value of name parameter, the method will create a chain on method calls .cdr().cdr().car() which will be applied to delegate of the operation which is our LazyList object.

With this method in place we can call car/cdr functions with arbitrary depth.

assert lazylist.caddr() == 3

If you create nested lazy lists, you can access any element of any nested list with this dynamic method.

def lmn = LazyList.nil().cons('N').cons('M').cons('L')
def almnz = LazyList.nil().cons('Z').cons(lmn).cons('A')
assert almnz.cadadr() == 'M'

With so many cons methods it's hard to see the structure of the list. Let's implement lazy method on ArrayList class that converts strict list to lazy. Again, we will use metaprogramming and functional techniques.

ArrayList.metaClass.lazy = {
    -> delegate.reverse().inject(LazyList.nil()) {list, item -> list.cons(item)}
}

Now we can rewrite the previous example as follows

def lazyfied = ['A', ['L','M','N'].lazy(), 'Z'].lazy()
assert lazyfied.cadadr() == 'M'

What have we accomplished so far? We learned how to build lazy lists from scratch and from strict lists. We know how to add elements to lazy lists, and how to access them. The next step is to implement fold function. fold is the fundamental operation in functional languages, so our lazy lists must provide it.

    boolean isEmpty() {
        list.call() == []
    }

    def fold(n, acc, f) {
        n == 0 || isEmpty() ? acc : cdr().fold(n-1, f.call(acc, car()), f)
    }

    def foldAll(acc, f) {
        isEmpty() ? acc : cdr().foldAll(f.call(acc, car()), f)
    }

The only difference between this fold function and the standard one is the additional parameter n. We will need it later when we implement infinite lists. foldAll function to lazy lists is the same as standard fold to strict lists.

assert [1,2,3,4,5].lazy().foldAll(0){ acc, i -> acc + i } == 15
assert [1,2,3,4,5].lazy().fold(3, 1){ acc, i -> acc * i } == 6

First example calculates the sum of all elements of the list, second calculates the product of first three elements.

If you have fold functions you can easily implement take functions

    def take(n) {
        fold(n, []) {acc, item -> acc << item}
    }

    def takeAll() {
        foldAll([]) {acc, item -> acc << item}
    }
   
    def toList() {
        takeAll()
    }

take is an inverse operation to lazy

assert [1,2,3,4,5].lazy().takeAll() == [1,2,3,4,5]
assert [1,2,3,4,5].lazy().take(3) == [1,2,3]

Our next goal is map function on lazy lists. Ideally I want the implementation look like this

    def map(f) {
        isEmpty() ? nil() : cdr().map(f).cons(f.call(car()))
    }

For some reason it doesn't work lazy way in Groovy — it's still strictly evaluated. Therefore I have to implement it directly with closure syntax

    def map(f) {
        isEmpty() ? nil() : new LazyList( {-> [f.call(car()), cdr().map(f).list]} )
    }

Unlike fold, lazy map is identical to strict map

assert [1,2,3,4,5].lazy().map{ 2 * it }.take(3) == [2,4,6]

The following example shows one of the benefits of laziness

assert [1,2,3,0,6].lazy().map{ 6 / it }.take(3) == [6,3,2]

map didn't evaluate the entire list, hence there was no exception. If you evaluate expression for all elements, the exception will be thrown

try {
    [1,2,3,0,6].lazy().map{ 6 / it }.takeAll()
}
catch (Exception e) {
    assert e instanceof ArithmeticException
}

For strict lists this is a default behaviour of map function.

The last function I want to implement is filter

    def filter(p) {
        isEmpty() ? nil() :
            p.call(car()) ? new LazyList( {-> [car(), cdr().filter(p).list]} ) :
                cdr().filter(p)
    }

In the following example we find first two elements greater than 2

assert [1,2,3,4,5].lazy().filter{ 2 < it }.take(2) == [3,4]

With the help of car/cdr, fold, map and filter you can implement any other function on lazy lists yourself. Here is, for example, the implementation of zipWith function

    static def zipWith(alist, blist, f) {
        alist.isEmpty() || blist.isEmpty() ? nil() :
            new LazyList( {-> [
                f.call(alist.car(), blist.car()),
                zipWith(alist.cdr(), blist.cdr(), f).list
            ]} )
    }

Now, after we implemented all lazy functions we need, let's define infinite lists

    private static sequence(int n) {
        {-> [n, sequence(n+1)]}
    }

    static LazyList integers(int n) {
        new LazyList(sequence(n))
    }

    static LazyList naturals() {
        integers(1)
    }

Infinite lists, from my point of view, is the most useful application of lazy lists

def naturals = LazyList.naturals()
assert naturals.take(3) == [1,2,3]

def evens = naturals.map { 2 * it }
assert evens.take(3) == [2,4,6]

def odds = naturals.filter { it % 2 == 1 }
assert odds.take(3) == [1,3,5]

assert naturals.cadddddddddr() == 10

def nonnegatives = naturals.cons(0)
assert nonnegatives.cadr() == 1

assert LazyList.zipWith(evens, odds){ x, y -> x * y }.take(4) == [2,12,30,56]

At this point you have all basic functionality implemented, and you should be able to extend this model to whatever you need in regards to lazy (infinite) lists. Happy lazy programming!

Resources and links

• Source code for this blog

• Lazy list implementation in Erlang

• Lazy list implementation in Lisp

Counting modifications in Git repository

Recently Michael Feathers wrote a blog about Open-Closed Principle, where he described simple technique that measures the closure of code. I created a Groovy script which implements this technique for Git repositories. If you run it from the root of your Git project, it produces a CSV file with the statistics of how many times files have been modified. Feel free to use this script to find hot spots in your Git repository.

SpringOne2GX 2010

Last week I attended SpringOne2GX conference in Chicago, the main event in Spring/Groovy/Grails community. Here I want to post my brief review of this conference.

First impressions

The hotel (Westin Lombard) was nice and clean. Internet: there were 2 wireless networks and one cable - everything was free and worked pretty well, signal was good in almost all rooms. The conference reception was well organized - every participant received bunch of souvenirs and special edition of NFJS magazine. I saw hundreds of smiling and happy people of different ages and different outfits. Most of them with Macs. Most of them know each other. The food was fantastic, especially dinner with wine and beer.

Day 1

The first day was mostly introduction and orientation. There was only one talk on the schedule.

Rod Johnson - Keynote (video)

I thought Spring was initially created 7 years ago but the oldest class in the source tree is dated by January 17, 2001, so Spring is actually almost 10 years old. Because of the anniversary the main theme of the presentation was: Where Spring goes in the next decade.

Since the core framework is well crafted already, the focus will be on the integration and making Spring portfolio as a platform for applications. There are three key values in Spring - portability, productivity and innovation - and the platform will be built along those dimensions.

Portability

In the past SpringSource made a good job by providing a framework that make Java applications easily portable across different application servers. The goal for the next decade is to expand the same portability to the cloud - Google AppEngine, vFabric, vmforce, etc.

Productivity

As we all know the ultimate reason of the Spring existence is to make the life of application developer easier, our work is more productive. The framework hides the low-level boilerplate, and provides well defined abstractions. In the next year there will be several features added to the Spring portfolio. Rod mentioned some of them:
- Seamless GWT integration
- Database reverse engineering with roundtripping support in Spring Roo 1.1. You will be able to generate the domain object tree based on your database schema, and it will be updated every time you change the database.
- Spring Payment Services project with Visa integration.

Another aspect of productivity is a tool suite, and here Spring gives you STS. Rod invited Christian Dupuis on the stage, where he demoed how to developed Grails applications in STS. If you are a Grails developer you should definitely take a look at the latest version of STS - it will increase your productivity significantly.

Innovation

There will be several new projects released in the Spring portfolio soon:
- Spring Social - application abstraction for social networks.
- Spring Mobile - platform for multi-device applications.
- Spring-AMQP - API for integration with RabbitMQ.
- Spring Data - API to work with NoSQL databases, in particular Neo4J support in Spring Roo.

Keith Donald demoed GreenHouse project and corresponding iPhone app. This is a reference implementation of Spring Mobile and Spring Social, and this app was really really useful during the conference when I needed to check the schedule and find the room.

At the end of the presentation Rod introduced, and Mik Kersten demoed, the next big thing - Code2Cloud. It's basically a tool that allows you to keep and manage your entire development environment in the cloud: the running app, the source code, the issue tracker, and the build server. Everything is in the cloud and configured by mouse click. It looks cool, and it definitely will be a buzz word in the next year, but I'm not sure if many people will use it. We'll see.

Day 2

I'm going to write only about technical sessions I attended.

Jürgen Höller - What's new in Spring Framework 3.1? (video)

That was one of the best talks of this conference: technical, right to the point, with well-wrtten slides, and personal charm of the presenter. Despite the number 3.1 in the title, Jürgen actually covered three versions of Spring framework: 3.0, 3.1, and 3.2. I'm going to briefly mention the interesting features, and if you want more details you can check the excellent on-line documentation.

Spring 3.0

- Custom annotations. You can create your own annotation by combining multiple existing annotations in one group. Spring automatically detects your annotation during the application context startup, and no special configuration is required. This is a very handy feature, especially when you copy-paste the same annotation group over and over again.

- Configuration classes and annotated factory methods. If you annotate a method with @Bean annotation Spring framework will make the output of the method a Spring bean. There are some other annotations supported, e.g. @Lazy.

- Standardized annotations. Spring now supports JSR-330 @Inject, JSR-250 @ManagedBean, and EJB 3.x @TransactionAnnotation.

- EL++. Expression language can be used now in bean definitions inside appcontext XML, and also in component annotations. Very powerful feature.

- REST support. Spring provides RestTemplate for client code, @PathVariable annotation, and special view resolvers on the server side. It's very interesting topic - check the documentation for details.

- Declarative model validation. You can specify data constraints right in your code by using annotations - very similar to what you have in GORM.

- Improved scheduling. New namespace, and @Scheduled and @Async annotations makes your appcontext smaller and more readable.

If you follow Spring releases, you probably use some or most of these features already. Now let's see what Spring 3.1 brings to us.

Spring 3.1

- Environment profiles for beans. Similar to Maven profiles but works in runtime. The idea here is to create a single deployment unit for all environments and enable certain Spring beans for specific environment. I can't wait to try this feature in our enterprise project.

- Cache abstraction. After 5 years of hibernation this feature is finally implemented. Spring provides an API to work with distributed cache, in particular in cloud environments. There will be adapters for most popular cache implementations, such as EhCache, GemFire, Coherence.

- Conversation management, or how Jürgen calls it HttpSession++. It's basically an extension of HttpSession shared across multiple browsers and window tabs. Looks very interesting.

- Enhanced Groovy support.

- c: namespace, which is a shortcut for <constructor-arg>, analogous to p: namespace for properties. Small feature that makes your appcontext consistent and more readable.

Spring 3.2

Java SE 7 support, JDBC 4.1, support for fork-join framework, general focus on concurrent programming.

Jeff Brown - GORM inside and out

This talk was also good. I worked a bit with GORM before, and had an idea how it's implemented, but it was useful to hear more details from one of the developers.

Jeff started with the background of GORM, the complexity of Hibernate and JPA, and how GORM solves this problem following convention-over-configuration and sensible defaults strategy. He showed how to model the domain objects, what happens behind the scene when you link objects together, how to specify uni- and bi-directional relationships, and how to change default collection implementation in case of one-to-many relationship.

During the presentation he was switching back and forth between sides and terminal, so it was easy to follow and understand the evolution of the sample application. He explained how to introduce various constraints into the model and how Grails would validate them. One of the interesting features I didn't know about was how to test internationalized error messages. You don't need to change your locale for that, simply add lang=your_language parameter to the URL, and Grails will switch to that language for all subsequent requests. Pretty handy.

He concluded the talk by showing how dynamic finders are implemented in GORM using Groovy metaprogramming feature. Interesting part here is that you can implement similar things in your Groovy code using the same technique, basically having custom mini-GORM in your Groovy project.

Venkat Subramaniam - Improving your Groovy code quality

The title of this presentation was little bit misleading for me. I expected Venkat to show some Groovy specific mistakes and how to avoid them. Instead, he was talking about the errors that in most cases are equally applied to any programming language. He mentioned various code smells and explained how to fix them. If you are interested, you can download the slides from Venkat's web site.

He also gave an advise how to maintain the high code quality:
- Have a respectable colleague review your code.
- Use code analysis tools like CodeNarc and Sonar Groovy plugin.

One of the topics he covered was the usage of the 'return' keyword in Groovy. That was interesting. Compare the following two functions and guess what they return:

def func1() {
    try {
        5
    } finally {
        return 22
    }
}

def func2() {
    try {
        5
    } finally {
        22
    }
}

Paul King, Guillaume Laforge - Groovy.DSLs (from: beginner, to: expert) (video)

This would be very nice presentation if the speakers didn't try to cover too much. This talk could be easily split into two: one is an overview of Groovy language and another one is DSL. Unfortunately they spent lot of time on theoretical DSL part and Groovy overview, so the practical DSL part was too short from my perspective. The good thing though is that I have slides now, so I can dig deeper into this subject at my spare time.

In the second part of the talk Paul and Guillaume explained which features of Groovy language make it so simple to create DSLs. Here are some of them:
- Static imports and import aliases.
- Simplified collection syntax.
- Small or no language noise.
- Aggregating multiple method calls using 'with' construct.
- Closures.
- Operator overloading.
- Metaprogramming.

In the last part speakers talked about different patterns and techniques of DSL implementations. They provided a comprehensive list of books you might want to read if you are interested in building DSLs.

Adrian Colyer - Technical keynote (video)

Adrian's talk was mostly a reiteration of Rod's keynote from the previous day with some technical details. He mentioned Spring Payment and Spring Data projects, bean profiles and cache support in the Spring core. He showed Spring portability in action by providing links to Spring applications deployed on Google AppEngine and vmForce.

Another interesting part was 20 minutes dedicated to RabbitMQ and Spring-AMQP. He even mentioned Spring-Erlang project which is supposed to be a convenient abstraction on top of standard Jinterface library.

As a continuation of innovation theme Graeme Rocher demoed GORM support for NoSQL databases. That was cool. He simply uninstalled Hibernate plugin and installed Redis plugin, without touching data model. Everything worked perfect. Right now Spring works with Redis and GemFire, but soon they are going to add support for CouchDB, Cassandra, Riak, Neo4j, and MongoDB. Another interesting thing Graeme showed was grails-console. It's a pretty nice tool, you should check it out. It allows you to interact with the Grails data storage using GORM features. Very handy.

Another co-presenter was Keith Donald who demoed Spring Social and Spring Mobile. He explained how OAuth works, and how interoperability with social networks was implemented in GreenHouse.

The keynote was concluded by Jon Travis who demoed SpringInsight.

Day 3

Venkat Subramaniam - Functional programming in Groovy

That was an excellent talk and nice start of the new conference day. Venkat explained main concepts and values of functional programming, and illustrated the theory with comprehensible examples.

He compared imperative and functional style of programming by showing how to implement for-loop using inject() function in Groovy. I think it was one of the best explanations of functional folding I've ever heard. He also demonstrated map and filter operations using collect() and findAll() methods.

He clarified the difference between function value and closure, and between iterative procedure and iterative process. He gave an example on how to pass closure as a parameter to simulate function object in Groovy. He also showed how to replace tail-recursion, which Groovy doesn't support, with inject() method call.

The presentation was concluded with an example of how to use functional techniques to build DSLs in Groovy.

Matthias Radestock, Mark Pollack, Mark Fisher - RabbitMQ and Spring-AMQP (video)

If you read my blog, you know that RabbitMQ is one of my latest interests. I decided to go to this talk just to see how the creators would present their projects. It turned out to be a nice introduction to RabbitMQ and Spring-AMQP. They explained main concepts of AMQP and how it is different from JMS. Here I want to give you some ideas which were not obvious for me when I started working with RabbitMQ.

- Messaging is all about decoupling, and AMQP is much more flexible than JMS in terms of publisher-consumer decomposition.
- All resources are dynamically created and destroyed by clients - the static pre-configuration is optional.
- Exchanges are stateless, they don't keep messages, they only copy and dispatch them. Queues hold the messages and deliver one message to a single client. They neither do routing nor message copying.
- Queue never receives the same message twice.
- If the message doesn't match routing key it's dropped.
- Because of the open protocol, you can use all available TCP tools to monitor your message traffic.

Besides AMQP implementation RabbitMQ also provides some other useful features like custom exchanges, exchange-to-exchange routing, different protocol adaptors, etc. Spring, as usual, gives you a consistent API on top of the RabbitMQ client which hides all low-level boilerplate and makes your application code more readable.

Good presentation, great guys.

Craig Walls - Developing social-ready web applications (video)

This presentation was about integrating your Java code with different social networks. There are three types of such integration: widgets, embedded code, REST API. Craig briefly explained first two, and then dived into REST.

All popular social networks provide REST API which allows you to communicate with them. For simple operations, like search, you can just use standard Spring RestTemplate class to retrieve the data. Try for example the following URLs:
- http://api.twitter.com/1/friends/ids.xml?screen_name=ndpar
- http://search.twitter.com/search.json?q=s2gx
- https://graph.facebook.com/ndpar

This basic approach fails though if you try to post a new message, because you have to be authorized for update operations. That's where OAuth comes in. The idea behind OAuth is pretty simple: instead of sharing your user-password with different clients, it uses generated tokens. This model is more flexible because if you want to revoke the permission from particular client you don't need to change your password and notify rest of the clients - you just remove that client's token from the list of authorized clients and that's it. The only problem with OAuth and social networks is that they support different versions of OAuth. This problem is solved by Spring Social project.

Spring Social offers consistent template-based API across different social providers. It basically gives you an OAuth aware RestTemplate, so you can do something like this:

TwitterTemplate twitter = new TwitterTemplate(API_KEY, API_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET);
twitter.updateStatus("Hello #s2gx !");
twitter.retweet(26887414177L);

If you are in a social network business, definitely take a look at Spring Social.

Mark Pollack, Chris Richardson - Using Spring with non-relational databases (video)

Relational databases are great, right? They've been with us for ages. Everybody knows how to work with them, how to build SQL statements. Every language provides ODBC library. There are bunch of frameworks that make developer's life easier. So why so sudden buzz around NoSQL?

Mark and Chris started their talk highlighting some problems that exist in relational database world:
- Object-relational impedance mismatch. Complicated mapping of rich domain model to relational schema. Relational schema rigidity.
- Extremely difficult/impossible to scale write operations.
- Suboptimal performance in some cases.

All these issues are addressed in NoSQL databases. Although keep in mind that it's not coming for free - you have to trade off ACID semantics, transactions and some other features of RDBMS. But if scalability is more important for you than consistency then NoSQL is your way to go.

There are tons of NoSQL databases available for you, but they all can be split into 4 categories based on their data model:
- Key-Value: Amazon Dynamo, Redis, Riak, Voldemort.
- Column: Google Bigtable, HBase, Cassandra.
- Document: CouchDB, MongoDB.
- Graph: Neo4j, Sones, InfiniteGraph.

Mark and Chris talked about each type, what their typical use cases are, and how their APIs look like. They showed examples for Redis, Cassandra, MongoDB, CouchDB and Neo4j. Then they introduced Spring Data project which, as everything from SpringSource, simplifies the application development and eliminates low-level code. Right now they support most of the popular NoSQL databases, and they plan to add more in the future.

The project is in active development phase, and the new contributors are welcome. So if it sounds interesting for you, go and check it out.

Day 4

Hans Dockter - Gradle - a better way to build

I never played with Gradle, so I was very curious to see how it looks like. According to Hans, who is the creator of this tool, Gradle is a general purpose build system with Groovy DSL interface. It's written in Java and provides build-in support for Java, Groovy, Scala, web and OSGi projects. It's a build language, so you can extend it for your own purposes if needed.

If you compare it with Ant, Gradle is definitely much better because it's more compact and flexible. It offers dependency resolution with integration with Maven and Ivy repositories. It also has some advanced features like incremental builds for custom tasks and parallel testing.

The only problem I had with this presentation was that Hans kept comparing Gradle with Maven. In my opinion they are not comparable. They have different philosophy if you want. All Maven 'constraints' are imposed by design, so it makes no sense to blame Maven for them. I think Ant-Gradle comparison is more appropriate and that's what Hans should have emphasized.

Other than that the session was pretty informative, and I have a better picture of Gradle now.

Brian Sletten - Groovy + The Semantic Web

I had no idea what Semantic Web was. I saw this term first time on the conference schedule, so I decided to go to this talk just to educate myself. I cannot even briefly describe all the discoveries I made during this presentation because I still feel little bit overwhelmed. I just want to provide some links from Brian's slides that can guide you if you want to learn this concept.

- Semantic Web - article from wikipedia.
- Formal W3C specs: RDF, RDFa, SKOS, SPARQL, OWL.
- SPARQL demo.
- RDFa distiller and parser. Try to feed Brian's test page URL (http://bosatsu.net/nfjs/test.html) to the distiller and see what it returns.
- OG - open graph protocol.
- Jena - Java API to work with Semantic Web.
- Java-RDFa parser.
- Pellet - Java API for OWL.

Conclusion

Whew! This happens to be longer review than I planned initially. If you are still with me you deserve my applause!

There were much more presentations at this conference but because of the tight schedule I had to sacrifice 80% of them. My overall impression from this conference is very positive. If you are a Spring/Groovy/Grails developer I encourage you to go to this event next year. The biggest benefit of it: You start seeing the Spring as a universe, not as a bunch of separate projects. You cannot get this feeling from the documentation, even if it's perfect as the Spring one.

Get started with RabbitMQ

RabbitMQ is an open-source implementation of AMQP. If you don't know what AMQP is, I encourage you to check it out on the official web site, or alternatively read articles listed on the reference page. Here I want to mention only the reasons why it drew my attention as an Erlang enthusiast and Java developer working in financial industry:

• AMQP is a replacement for TIBCO Randezvous;
  
• in terms of functionality it's a superset of JMS;
  
• it's written in Erlang, which means fault-tolerance, reliability and high performance.

In this blog post I just want to show how to install RabbitMQ on Ubuntu box, and verify that it works with simple Groovy client.

Installing RabbitMQ server

As everything with Ubuntu, this step is pretty trivial:

$ sudo apt-get install rabbitmq-server

The only requirement for this package is Erlang distribution. If you already have Erlang installed on your system, the installation of rabbitmq-server is a quick procedure. The following directories will be created during the installation:

/usr/lib/rabbitmq/bin	executables added to the path
/usr/lib/erlang/lib/rabbitmq_server-1.x.x	compiled modules
/var/lib/rabbitmq/mnesia	persistent storage for messages
/var/log/rabbitmq	log files (e.g. startup_log, rabbit.log)

After installation is finished the RabbitMQ server is started and listens to incoming requests on port 5672. You can check /var/log/rabbitmq/startup_log file to see if everything was ok.

Groovy clients

I followed official Java client API to build two scripts: consumer.groovy

import com.rabbitmq.client.*

@Grab(group='com.rabbitmq', module='amqp-client', version='1.7.2')
params = new ConnectionParameters(
    username: 'guest',
    password: 'guest',
    virtualHost: '/',
    requestedHeartbeat: 0
)
factory = new ConnectionFactory(params)
conn = factory.newConnection('lab.ndpar.com', 5672)
channel = conn.createChannel()

exchangeName = 'myExchange'; queueName = 'myQueue'

channel.exchangeDeclare exchangeName, 'direct'
channel.queueDeclare queueName
channel.queueBind queueName, exchangeName, 'myRoutingKey'

def consumer = new QueueingConsumer(channel)
channel.basicConsume queueName, false, consumer

while (true) {
    delivery = consumer.nextDelivery()
    println "Received message: ${new String(delivery.body)}"
    channel.basicAck delivery.envelope.deliveryTag, false
}
channel.close()
conn.close()

and publisher.groovy

import com.rabbitmq.client.*

@Grab(group='com.rabbitmq', module='amqp-client', version='1.7.2')
params = new ConnectionParameters(
    username: 'guest',
    password: 'guest',
    virtualHost: '/',
    requestedHeartbeat: 0
)
factory = new ConnectionFactory(params)
conn = factory.newConnection('lab.ndpar.com', 5672)
channel = conn.createChannel()

channel.basicPublish 'myExchange', 'myRoutingKey', null, "Hello, world!".bytes

channel.close()
conn.close()

Now start consumer in one terminal window

$ groovy consumer.groovy

and run publisher in another:

$ groovy publisher.groovy

On the consumer window you should see Received message: Hello, world! text, which means RabbitMQ works correctly.

Monitoring logs

You can check RabbitMQ logs by doing tail -f /var/log/rabbitmq/rabbit.log For example, starting the consumer results the following log entries:

=INFO REPORT==== 14-Mar-2010::11:20:53 ===
accepted TCP connection on 0.0.0.0:5672 from 192.168.2.10:62424

=INFO REPORT==== 14-Mar-2010::11:20:53 ===
starting TCP connection <0.24154.1> from 192.168.2.10:62424

Running the publisher:

=INFO REPORT==== 14-Mar-2010::11:22:08 ===
accepted TCP connection on 0.0.0.0:5672 from 192.168.2.10:62432

=INFO REPORT==== 14-Mar-2010::11:22:08 ===
starting TCP connection <0.24232.1> from 192.168.2.10:62432

=INFO REPORT==== 14-Mar-2010::11:22:08 ===
closing TCP connection <0.24232.1> from 192.168.2.10:62432

Now if we terminate the consumer by ^C there will be a warning

=WARNING REPORT==== 14-Mar-2010::11:25:03 ===
exception on TCP connection <0.24154.1> from 192.168.2.10:62424
connection_closed_abruptly

=INFO REPORT==== 14-Mar-2010::11:25:03 ===
closing TCP connection <0.24154.1> from 192.168.2.10:62424

but the connection is closed properly by the server.

That's it for now. Stay tuned for the future updates on my RabbitMQ experience.

Links

• Rapid application prototyping with Groovy DSL

Multithreaded XmlSlurper

Groovy XmlSlurper is a nice tool to parse XML documents, mostly because of the elegant GPath dot-notation. But how efficient is XmlSlurper when it comes to parsing of thousands of XMLs per second? Let's do some simple test

class XmlParserTest {

    static int iterations = 1000

    def xml = """
       <root>
         <node1 aName='aValue'>
           <node1.1 aName='aValue'>1.1</node1.1>
           <node1.2 aName='aValue'>1.2</node1.2>
           <node1.3 aName='aValue'>1.3</node1.3>
         </node1>
         <node2 aName='aValue'>
           <node2.1 aName='aValue'>2.1</node2.1>
           <node2.2 aName='aValue'>2.2</node2.2>
           <node2.3 aName='aValue'>2.3</node2.3>
         </node2>
         <nodeN aName='aValue'>
           <nodeN.1 aName='aValue'>N.1</nodeN.1>
           <nodeN.2 aName='aValue'>N.2</nodeN.2>
           <nodeN.3 aName='aValue'>N.3</nodeN.3>
         </nodeN>
       </root>
    """

    def parseSequential() {
        iterations.times {
            def root = new XmlSlurper().parseText(xml)
            assert 'aValue' == root.node1.@aName.toString()
        }
    }

    @Test void testSequentialXmlParsing() {
        long start = System.currentTimeMillis()
        parseSequential()
        long stop = System.currentTimeMillis()
        println "${iterations} XML documents parsed sequentially in ${stop-start} ms"
    }
}

I ran this test on my 4-core machine and I got

1000 XML documents parsed sequentially in 984 ms

Not really good (0.984 ms per document) but we didn't expect much from single threaded application. Let's parallelize this process

class XmlParserTest {
    ...
    static int threadCount = 5
    ...
    @Test void testParallelXmlParsing() {
        def threads = []
        long start = System.currentTimeMillis()
        threadCount.times {
            threads << Thread.start { parseSequential() }
        }
        threads.each { it.join() }
        long stop = System.currentTimeMillis()
        println "${threadCount * iterations} XML documents parsed parallelly by ${threadCount} threads in ${stop - start} ms"
    }
}

And the result is

5000 XML documents parsed parallelly by 5 threads in 1750 ms

This is definitely better (0.35 ms per document) but doesn't look like parallel processing — the test time shouldn't increase in true parallelism.

The problem here is the default constructor of XmlSlurper. It does too much: first, it initializes XML parser factory loading bunch of classes; second, it creates new XML parser, which is quite expensive operation. Now imaging this happens thousand times per second.

Luckily, XmlSlurper has another constructor, with XML parser parameter, so we can create the parser up-front and pass it to the slurper. Unfortunately, we cannot reuse one parser instance between several slurpers because XML parser is not thread-safe — you have to finish parsing one document before you can use the same parser to parse another.

The solution here is to use preconfigured pool of parsers. Let's create one based on Apache commons-pool library.

public class XmlParserPoolableObjectFactory implements PoolableObjectFactory {
    private SAXParserFactory parserFactory;

    public XmlParserPoolableObjectFactory() {
        parserFactory = SAXParserFactory.newInstance();
    }
    public Object makeObject() throws Exception {
        return parserFactory.newSAXParser();
    }
    public boolean validateObject(Object obj) {
        return true;
    }
    // Other methods left empty
}

public class XmlParserPool {
    private final GenericObjectPool pool;

    public XmlParserPool(int maxActive) {
        pool = new GenericObjectPool(new XmlParserPoolableObjectFactory(), maxActive,
                GenericObjectPool.WHEN_EXHAUSTED_BLOCK, 0);
    }
    public Object borrowObject() throws Exception {
        return pool.borrowObject();
    }
    public void returnObject(Object obj) throws Exception {
        pool.returnObject(obj);
    }
}

Now we can change our test

class XmlParserTest {
    static XmlParserPool parserPool = new XmlParserPool(1000)
    ...
    def parseSequential() {
        iterations.times {
            def parser = parserPool.borrowObject()
            def root = new XmlSlurper(parser).parseText(xml)
            parserPool.returnObject(parser)
            assert 'aValue' == root.node1.@aName.toString()
        }
    }
}

and run it again

1000 XML documents parsed sequentially in 203 ms
5000 XML documents parsed parallelly by 5 threads in 172 ms

That's much better (0.034 ms per document), and most importantly multi-threading really works now.

Resources

• Source code for this blog

• Article "Improve performance in your XML applications"

• GPath vs XPath

• commons-pool home page

Parsing files using Groovy regex

In my previous post I mentioned several ways of defining regular expressions in Groovy. Here I want to show how we can use Groovy regex to find/replace data in the files.

Parsing properties file (simplified)¹

Data: each line in the file has the same structure; the entire line can be matched by single regex. Problem: transform each line to the object. Solution: construct regex with capturing parentheses, apply it to each line, extract captured data. Demonstrates: File.eachLine method, matrix syntax of Matcher object.

def properties = [:]
new File('path/to/some.properties').eachLine { line ->
    if ((matcher = line =~ /^([^#=].*?)=(.+)$/)) {
        properties[matcher[0][1]] = matcher[0][2]
    }
}
println properties

Parsing csv files (simplified)²

Data: each line in the file has the same structure; the line consists of the blocks separated by some character sequence. Problem: transform each line to the list of objects. Solution: construct regex with capturing parentheses, parse each line with the regex in a loop extracting captured data. Demonstrates: ~// Pattern defenition, Matcher.group method, \G regex meta-sequence.

def regex = ~/\G(?:^|,)(?:"([^"]*+)"|([^",]*+))/
new File('path/to/file.csv').eachLine { line ->
    def fields = []
    def matcher = regex.matcher(line)
    while (matcher.find()) {
        fields << (matcher.group(1) ?: matcher.group(2))
    }
    println fields
}

Finding snapshot dependencies in the pom (simplified)³

Data: file contains blocks with known boundaries (possibly crossing multiple lines). Problem: extract the blocks satisfying some criteria. Solution: read the entire file into the string, construct regex with capturing parentheses, apply the regex to the string in a loop. Demonstrates: File.text property, list syntaxt of Matcher object, named capture, global \x regex modifier, local \s regex modifier.

def pom = new File('path/to/pom.xml').text
def matcher = pom =~ '''(?x)
    <dependency>                          \\s*
      <groupId>([^<]+)</groupId>          \\s*
      <artifactId>([^<]+)</artifactId>    \\s*
      <version>(.+?-SNAPSHOT)</version>   (?s:.*?)
    </dependency>
'''
matcher.each { matched, groupId, artifactId, version ->
    println "$groupId:$artifactId:$version"
}

Finding stacktraces in the log

Data: file contains entries each of which starts with the same pattern and can span multiple lines. Typical example is log4j log files:

2009-10-16 15:32:12,157 DEBUG [com.ndpar.web.RequestProcessor] Loading user
2009-10-16 15:32:13,258 ERROR [com.ndpar.web.UserController] id to load is required for loading
java.lang.IllegalArgumentException: id to load is required for loading
     at org.hibernate.event.LoadEvent.(LoadEvent.java:74)
     at org.hibernate.event.LoadEvent.(LoadEvent.java:56)
     at org.hibernate.impl.SessionImpl.get(SessionImpl.java:839)
     at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
     at org.springframework.orm.hibernate3.HibernateTemplate$1.doInHibernate(HibernateTemplate.java:531)
     at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:419)
     at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
     at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:525)
     at org.springframework.orm.hibernate3.HibernateTemplate.get(HibernateTemplate.java:519)
     at com.ndpar.dao.UserManager.getUser(UserManager.java:90)
     ... 62 more
2009-10-16 15:32:14,659 DEBUG [com.ndpar.jms.MessageListener] Received message:
    ... multi-line message ...
2009-10-16 15:32:15,169 INFO  [com.ndpar.dao.UserManager] User: ...

Problem: find entries satisfying some criteria. Solution: read the entire file into the string⁴, construct regex with capturing parentheses and lookahead, split the string into entries, loop through the result and apply criteria to each entry. Demonstrates: regex interpolation, combined global regex modifiers \s and \m.

def log = new File('path/to/your.log').text
def logLineStart = /^\d{4}-\d{2}-\d{2}/
def splitter = log =~ """(?xms)
    (    ${logLineStart}   .*?)
    (?=  ${logLineStart} | \\Z)
"""
splitter.each { matched, entry ->
    if (entry =~ /(?m)^(?:\t| {8})at/) println entry
}

Replacing text in the file

Use Groovy one-liner to perform the replacement. Here is the Tim's example in Groovy:

$ groovy -p -i -e '(line =~ /1\.6/).replaceAll("2.0-alpha-1-SNAPSHOT")' `find . -name pom.xml`

Resources

• Groovy regexes
• Groovy one-liners
• Using String.replaceAll method

Footnotes

1. This example is for demonstration purposes only. In real program you would just use Properties.load method.
2. The regex is simplified. If you want the real one, take a look at Jeffrey Friedl's example.
3. Again, in reality you would find snapshots using mvn dependency:resolve | grep SNAPSHOT command.
4. This approach won't work for big files. Take a look at this script for practical solution.

GParallelizer Performance

GParallelizer is a Groovy wrapper for new Java concurrency library. It allows you to perform list and map operations using parallel threads, which in theory leverages the full power of multi-processor computations. Here I want to check if it's true in reality. I run the following tests on my dual-core MacBook

import static org.gparallelizer.Parallelizer.*
import org.gparallelizer.ParallelEnhancer
import org.junit.Before
import org.junit.Test

class GParsTest {

    def list = []

    @Before void setUp() {
        1000000.times {
            list << (float) Math.random()
        }
    }

    @Test void sequential() {
        def start = System.currentTimeMillis()
        list.findAll { it < 0.4 }
        def duration = System.currentTimeMillis() - start

        println "Sequential: ${duration}ms"
    }

    @Test void parallel_with_enhancer() {
        ParallelEnhancer.enhanceInstance list

        def start = System.currentTimeMillis()
        list.findAllAsync { it < 0.4 }
        def duration = System.currentTimeMillis() - start

        println "Parallel with enhancer: ${duration}ms"
    }

    @Test void parallel_with_parallelizer_2() {
        parallelWithParallelizer 2
    }

    @Test void parallel_with_parallelizer_3() {
        parallelWithParallelizer 3
    }

    @Test void parallel_with_parallelizer_5() {
        parallelWithParallelizer 5
    }

    @Test void parallel_with_parallelizer_10() {
        parallelWithParallelizer 10
    }

    def parallelWithParallelizer(threads) {
        def start = System.currentTimeMillis()
        withParallelizer(threads) {
            list.findAllAsync { it < 0.4 }
        }
        def duration = System.currentTimeMillis() - start

        println "Parallel with parallelizer (${threads}): ${duration}ms"
    }
}

And here is the output

Sequential: 774ms
Parallel with enhancer: 9311ms
Parallel with parallelizer (2): 1785ms
Parallel with parallelizer (3): 769ms
Parallel with parallelizer (5): 500ms
Parallel with parallelizer (10): 722ms

Something strange happened with mixed-in ParallelEnhancer, but with Parallelizer performance improved indeed. With optimal thread pool size parallel processing is 35% faster than sequential.

Conclusion: Use GPars methods if you need to process big amount of data. Try different config parameters to find the best solution for your particular problem.

Resources

• Brian Goetz on new concurrency library
• Vaclav Pech on GPars

Converting XML to POGO

Suppose we want to convert XML to Groovy bean:

class MyBean {
    String strField
    float floatField
    int intField
    boolean boolField
}

def message = "<xml stringAttr='String Value' boolAttr='true' />"

def xml = new XmlSlurper().parseText(message)

def bean = new MyBean(
    strField: xml.@stringAttr,
    boolField: xml.@boolAttr
)

Everything looks good, even assertions succeed

assert 'String Value' == bean.strField
assert bean.boolField

Now let's try false value:

message = "<xml stringAttr='String Value' boolAttr='false' />"
assert !bean.boolField

Oops, the assertion failed. Why? Because xml.@boolAttr cast to boolean always returns true. The correct implementation must be like this:

message = "<xml stringAttr='String Value' floatAttr='3.14' intAttr='9' boolAttr='false' />"

xml = new XmlSlurper().parseText(message)

bean = new MyBean(
    strField: xml.@stringAttr.toString(),
    floatField: xml.@floatAttr.toFloat(),
    intField: xml.@intAttr.toInteger(),
    boolField: xml.@boolAttr.toBoolean()
)
assert 'String Value' == bean.strField
assert 3.14F == bean.floatField
assert 9 == bean.intField
assert !bean.boolField

Now everything works properly. The moral of this blog post: Create more unit tests (assertions), especially when you work with dynamic language.

Resources

• Converting String to Boolean

Building regular expressions in Groovy

Because of compact syntax regular expressions in Groovy are more readable than in Java. Here is how Jeffrey Friedl's example would look like in Groovy:

def subDomain  = '(?i:[a-z0-9]|[a-z0-9][-a-z0-9]*[a-z0-9])' // simple regex in single quotes
def topDomains = """
       (?x-i : com        \\b     # you can put whitespaces and comments
             | edu        \\b     # inside regex in eXtended mode
             | biz        \\b
             | in(?:t|fo) \\b     # but you have to escape
             | mil        \\b     # backslashes in multiline strings
             | net        \\b
             | org        \\b
             | [a-z][a-z] \\b
       )"""

def hostname = /(?:${subDomain}\.)${topDomains}/  // variable substitution in slashy strings

def NOT_IN   = /;\"'<>()\[\]{}\s\x7F-\xFF/  // backslash is not escaped in slashy strings
def NOT_END  = /!.,?/
def ANYWHERE = /[^${NOT_IN}${NOT_END}]/
def EMBEDDED = /[$NOT_END]/                 // you can ommit {} around var name

def urlPath  = "/$ANYWHERE*($EMBEDDED+$ANYWHERE+)*"

def url =
    """(?x:
             \\b

             # match the hostname part
             (
               (?: ftp | http s? ): // [-\\w]+(\\.\\w[-\\w]*)+
             |
               $hostname
             )

             # allow optional port
             (?: :\\d+ )?

             # rest of url is optional, and begins with /
             (?: $urlPath )?
       )"""

assert 'http://www.google.com/search?rls=en&q=regex&ie=UTF-8&oe=UTF-8' ==~ url

As you can see, there are several options, and for every subexpression you can choose the one that's more expressive.

Resources

• Martin Fowler on composed regexes
• Pragmatic Dave on regexes in Ruby
• Feature request to make regexes even groovier
• Mastering Regular Expressions — best regex book
• Groovy Pattern and Matcher classes