Thursday, 13 August 2015

Introducing Raven: An Elegant Build for Java


There's a first step that every single Java project has to go through: setting up a build system. And often before that, choosing a build system. There hasn't been much improvement in the Java world in this area for quite a while; Ant 1.1 was released in July 2000 and Maven was created in 2004.

his lack of innovation could seem strange: for non trivial projects (which most end up being after some time), writing build scripts can take a lot of time. Given that builds usually aren't shipped to users, the time spent on their maintenance can seem like time lost... but a sub-optimal build system will actually make you lose time. So down the road, the best build is the one that saves you the most time when writing, debugging, and maintaining your scripts.

I started working on Raven because I was deeply dissatisfied with the solutions available in the Java world. And from what I've heard from other developers, I'm not the only one.

Now I'm going to say something controversial: both Ant and Maven have their strengths and weaknesses, but these tools are just toys compared to a full scripting environment. Think conditions, loops, exceptions, complex data structures. Most of all, think of all the details that you forgot to think about, all the little quirks and peculiarities that appear on most projects. What is going to be most powerful to solve these problems, a simple XML grammar or a full and powerful scripting language (not to mention Turing complete)? Would you rather write copy source, target or 3 lines of XML? And what fallback do you have when you're not within the boundaries imposed by the tool?

Getting Practical

Raven is based on the Ruby dynamic language and its most prominent build tool, Rake. Don't worry, you don't have to know either to read this article or start using Raven, you can learn little by little, starting simple. Rake itself is a little bit like Ant, it lets you define tasks and the dependencies between them. Only its syntax is much sweeter. For example:

task "hello" do
    puts "Hello"
task "world" => "hello" do
    puts "World"

If you have Rake installed, put this in a file named Rakefile and execute rake world in a command in the same directory as the file. It will do what you would expect. Note that the syntax could be even more terse by using { ... } blocks on one line instead of do ... end but this demonstrates the most common case, where you'll have more than one line of code in your task body. And you can put pretty much any Ruby code within the task block (and even Java code as we'll be using JRuby), even rely on external libraries, instantiate objects, and call methods. Your build can be as simple or as complex as you need.

The limitation is that Rake only provides very generic tasks that just wrap some classic Ruby code but don't do anything much by themselves. You have to tell them what to do in the nested code. That's where Raven shines. To make the life of Java developers easier, Raven adds a set of predefined tasks that implement the most common needs for a Java build system, like compiling or resolving jar dependencies. Just like a library, but instead of being a set of classes and methods, it's a set of reusable tasks that are well-suited for Java builds. So, all the tasks you're going to see in the rest of this article are implemented by Raven.

But wait, I haven't told you how to install anything. The quickest way to get started is to use Raven packaged with JRuby (a Ruby interpreter written in Java), everything necessary is bundled in it.
  1. Download the Raven distribution prepackaged with JRuby.
  2. Unzip it on your disk and set the environment variable JRUBY_HOME to this location.
  3. Add %JRUBY_HOME%\bin to your PATH environment variable.
  4. Check your installation by typing jruby -v in a command window.
For a more complete installation using the native Ruby interpreter (it's much faster to start up).

A Simple Project

To show you how to use Raven, I'm going to start with a simple but still a real world example: building Apache Commons Net. The Apache Commons Net library implements the client side of many network protocols like FTP or SMTP. Right now, their build is based on Ant and is mildly complex, so it's a pretty good candidate for me to present Raven.

Raven being just a set of specific tasks (plus a bit more, but we'll see that later), the whole build is still directed by Rake. So, all of the code I'm going to show is part of a file named Rakefile that should be placed at the root of the Commons Net unpacked source distribution. When you start Rake, it always looks for that script.

This first snippet demonstrates initialization and dependency handling:

require "raven"
require "rake/clean"
CLEAN.include ["target", "dist"]

dependency "compile_deps" do |task|
  task.deps << "oro-oro"

The two first lines load Raven and a Rake subtask for cleaning. The require command in Ruby is a bit like import, only it can load either a whole library (like Raven) or a single file. The third line tells Rake which directories should be removed by the clean task.

Lines 5 to 7 demonstrate the usage of the Raven dependency task. Commons Net depends on the Jakarta ORO library, so we're adding a dependency on it. It's just about listing which set of libraries will be needed. Calling the task (by executing rake compile_deps) will actually trigger the library download from a default Raven repository and depending on it will propagate a proper classpath as we'll see later. Also note that you can specify more than one library at a time and also give version numbers (Raven uses the latest by default). All of these library declarations are valid within a dependency task:

  task.deps << ["springframework-spring-core", 
      { "taglibs-standard" => "1.1.2" }]
  task.deps << ["axis2-kernel", "axis2-codegen"]

The provided name should follow the Maven naming of groupId and artifactId separated by a dash. Browse the Raven repository to see which libraries are available. Partial names can also be provided when there's no ambiguity. Now that we're done with dependencies, let's see what compilation would look like:

javac "compile" => "compile_deps" do |task|
  task.build_path << "src/java"

jar "commons-net.jar" => "compile"

The javac task is another of the tasks that Raven provides. What it does is pretty simple to understand. The => notation declares the pre-requisite on the dependencies. From this Raven can automatically compute the classpath for you. Notice that we are also setting the build path. It needs to be explicit as Commons Net has its sources under src/java. If it were under src/main/java, no additional configuration would be needed, making this the sweet one-liner:

javac "compile" => "compile_deps"

Finally, once compilation is done, the previous snippet also packages everything in a jar. That's the role of the jar task. The produced jar file is directly named like the task, minimizing the number of parameters.

With everything I've explained so far, you should end up with a 10 line Rakefile located at the root of the Commons Net source distribution. To run the build, just execute rake commons-net.jar and everything should get built in a target directory. You could also add a default task so that just running rake would build your jar:

task "default" => "commons-net.jar"

Some More

Compiling and packaging is nice, but it's usually only the first step in a build. For example, the Commons Net Ant script also handles tests and Javadoc. How would you do this with Raven? Once again, it's pretty simple, really.

junit "test"=>["compile", "compile_deps", "test_deps"] do |task|
  task.build_path << "src/test"
  task.test_classes << "**/*"

javadoc "jdoc" do |task|
  task.build_path << "src/java"

You probably don't need much of an explanation to understand what this does. Just note that the settings inside the tasks are here because Commons Net directory structure doesn't follow the Raven defaults. If the tests were located under src/test/java and the test classes followed the Test* pattern, the tasks would just be empty.

There are a few other tasks that I won't detail much more here, but that you should know of, in case you would need one of them.

Builds a jar file containing your project sources

Builds a WAR file from your compiled classes and the additional web application resources located under src/main/webapp

Creates a library directory and places all your project dependencies in it, makes it very easy to construct a classpath for your command scripts (bat or sh)

On the Shoulders of Giants

To be complete, our real life example should include a way to build a distribution. The Commons Net original build has a dist task and, even if it didn't, distribution is a pretty common use case, perhaps even the most common. So, how would you go about doing it with Raven? Well, errr, you don't. There's nothing in Raven to help you build distributions. You see, there's no real standard way to make a distribution, it really depends on what you want to include. But don't worry, you're not left alone here. As I mentioned at the beginning of this article, Raven is built on top of Rake, which itself runs in a full Ruby interpreter. So our dist task is just going to be a simple Rake task:

lib_dir("dist:libs" => "compile_deps") do |task| = "dist/lib"

task "dist" => ["commons-net.jar", "dist:libs"] do
  cp ["LICENSE.txt", "NOTICE.txt", "target/commons-net.jar"], "dist""dist/README.txt", "w") { |f| f << "Built on #{}" }

The first line of code demonstrates the usage of the lib_dir task that I explained previously. Then comes the interesting bit. The dist task is a standard Rake task, it only checks for the prerequisites and executes the code body afterward. Here I'm just making sure that the jar has been built and the libraries are included in a lib sub-directory. The rest is pure and simple Ruby.

Rake pre-includes a Ruby module that handles all basic file operations. Things like cp (copy), mv (move), mkdir (make directory), or rm (remove). That's pretty handy in a build where you typically do a lot of file manipulations. So, the first line in my task block copies the license, the notice file, and the produced jar in the distro directory. The cp method, just like most of the others, accept arrays.

The second line demonstrates how you would go about tweaking some file content. I'm creating a new README file (the "w" flag means new file) and adding a simple timestamp in it. Don't be put off by the #{..} syntax inside the string, it's just a way to place the result of a computation of a variable value inside of a string (the equivalent of "Built on" + new Date().toString()). Typically you would append that type of information in your README using the w+ flag, but Commons Net doesn't have a README, so I'm just creating an empty one here.

With the dist task, our build is complete, I've shown you everything that was needed to replace the original Ant script. We've reduced a 170 lines build to a 20 lines one. That's almost 10 times less code to maintain. But to drive my point a little further, just let me give you one last example that would demonstrate the usage of a control structure:

MODULES = ["web", "business", "persistence"]

MODULES.each do |mod|
  javac "#{mod}:compile" => ["#{mod}_deps", "common_deps"]

This would create a compilation task for a given list of modules. No need to repeat, just iterate. You can even create a method and call it from a task with specific parameters. Very basic things when you're programming, but something we've lost with most current build tools.

I hope you're now starting to see how much power being based on a scripting language like Ruby gives to Raven. You have a pretty strong and terse basis with a set of Java-specific tasks provided by Raven, simple cases are very simple to write. For everything that doesn't fit in the framework, you have an elegant safety net (in place of a plugin framework).

Other Choices

Raven isn't the only one of its kind, it's my answer to the build problem and to the dissatisfaction I had with the currently available tools. Others came with other solutions coming from the same frustrations, and I don't pretend that my solution will be the best for everybody. So, there are a couple of alternatives, built on the same foundations as Raven, namely Rake, but with a different philosophy.

The first alternative would be Antwrap. I wouldn't actually consider it a replacement for Raven, so much as a very good complement. It lets you reuse all existing Ant tasks that have already been created, but with a much nicer syntax than XML. So, you could use Raven for everything that's already included and Antwrap when an existing Ant task does what you're looking for, all within the same script.

The second tool is Buildr. It's an Apache Incubator project and completely overlaps with Raven, so it could be a total replacement. The difference is in the philosophy: Raven is imperative, asking you to write how to build your project; Buildr is more declarative, you specify what your build looks like. So, said differently, those of you who prefer the style of Ant over Maven will prefer Raven, those who are more seduced by the Maven model will probably find Buildr more seducing. And I don't see this as a problem, software is also a matter of preferences and taste, you should just use the tool that makes you most comfortable.