ZeroTurnaround Blog

Product Updates, Company News & Fun

Apache Camel with JRebel: Instantly visible changes to routes and config definitions

Moving Data with Apache Camel

Some years ago I was working in the field of financial data integration for a famous bank in the Baltics. One of the most common tasks with these kinds of applications is to consume a file, parse some data out of it, crunch the numbers, and pump the results into some store. Back then, this was enough even for large multinationals.

Nowadays, however, you frequently find yourself needing to deal with something else as a source (as well as a target) for the data: examples are web services, databases, message queues, proprietary middleware and others.

Writing the connectors/adapters/handles for all kind of sources and data formats might be quite time consuming, while only a tiny bit of the work is dedicated to the business logic, e.g. “multiply two numbers and log the result”. Apache Camel was the tool I used in multiple projects and it helped me a lot by providing out of the box components for data integration. I didn’t have to write the cumbersome JMS logic, didn’t mess with the files and the result was always very neat.

One little issue that bothered me, however, was the turnaround time needed when I just wanted to make little changes and see what will happen in the application’s behavior. In our JRebel 5.1.2 release we added support for Apache Camel and we spent some time polishing it for JRebel 5.2.

Main point: now it’s possible to reconfigure Camel routes automatically without the need to restart the application after the changes.

Considering Data Flows with Camel

The main idea is to think of the data flow as if is going from point A to point B, and doing some other actions in between. Camel basically provides a DSL in the form of fluent API that you can use to construct the pipelines for data transformations.

Here’s the sentence what we can use to construct the simplest data pipeline:

“The data should go from A to B

And here’s how we can write it in Camel API:

from(“A”).to(“B”);

It is very easy to write, read and understand. So “A” and “B” are currently just the abstract source and destination, but we can easily replace these with any implementation provided by Apache Camel:

from(“file:src/).to(“file:dst/);

Just as simple as that! Now we said that the files should be transferred from src/ folder to dst/. What if we want to send the files into different destinations depending on the content. This is a good scenario for the content based router pattern:

from(“file:src/).choice()
  .when(xpath("/person/city = 'London'"))
    .to("file:uk").
  .otherwise()
    .to("file:others");

Assuming that we work with XML files, we decide to send the files to the ‘uk’ folder if the element is equal to “London”, otherwise we send the file to others folder.

There’s plenty of options how to add business logic in this program flow. The simplest would be to add an implementation of Processor interface into the pipeline:

from(“file:src/).process(new MyProcessor()).to(“file:dst/);

So MyProcessor class now implements Processor interfaces provided by Camel in order to be able to plug some custom processing logic into the pipeline. The benefit of this approach is that it is easy to grasp even very complex dataflows and understand what the application should be doing.

There’s of course a lot more what you can do with Camel for the data integration scenarios, but I wouldn’t spend time describing those since Camel follows Enterprise Integration Patterns to implement the scenarios. If you work in data integration field, do yourself a favour, study the patterns – it will make your life much easier.

JRebel rides the Camel

JRebel can restart Camel routes and update the configuration definitions so that it is now much easier to experiment with the integration scenarios. Let’s have a walk through a simple example of a Camel application with JRebel and see what we can do.

Create project from Maven archetype:

mvn archetype:generate \
  -DinteractiveMode=false \
  -DgroupId=org.zt \
  -DartifactId=filemonkey \
  -Dversion=1.0-SNAPSHOT \
  -DarchetypeGroupId=org.apache.camel.archetypes \
  -DarchetypeArtifactId=camel-archetype-java 
  -DarchetypeVersion=2.10.0

The command above generate a simple application that transfers files from one location to the others:

public class MyRouteBuilder extends RouteBuilder {
  public void configure() {
    from("file:src/data?move=.done")
      .choice()
        .when(xpath("/person/city = 'London'"))
          .to("file:target/messages/uk").
        .otherwise()
          .to("file:target/messages/others");
  }
}

Create rebel.xml configuration file

As this is a Maven project, we can use JRebel Maven plugin in order to generate the configuration file that is required by JRebel agent in order to be able to monitor project workspace. You only need to add the plugin definition as described in the documentation.

Set MAVEN_OPTS variable

To bootstrap JRebel, we need to set MAVEN_OPTS environment variable as follows:
In Windows: set MAVEN_OPTS=”-javaagent:/path/to/jrebel.jar -Drebel.camel_plugin=true”
In Linux or Mac: export MAVEN_OPTS=”-javaagent:/path/to/jrebel.jar -Drebel.camel_plugin=true”

Start the application

Now we can start the application and as we have configured JRebel maven plugin, it will generate rebel.xml configuration file and it will be picked up by JRebel agent which we configured via MAVEN_OPTS environment variable. The Maven archetype that we used to generate the example automatically configures exec-maven-plugin in pom.xml so it is possible to start the application as follows:

mvn clean package exec:java

JRebel Camel plugin integrates with RouteBuilder class and you should see from console messages that the integration kicks in:

JRebel-Camel: Monitoring RouteBuilder class filemonkey.MyRouteBuilder

Change file component configuration

The generated example uses Camel file component which behavior can be configured depending on what do we want to do with the files. The initial example just transfers the files from one directory to the others depending on the content value, but the processed files are kept in the same place. In real life we would rather move the files to some other directory where all the processed files are archived.

- from("file:src/data?noop=true")
+ from("file:src/data?move=.done")

We say now that the processed files should be moved to “.done” directory after passing the route. Once the change is made and we recompile our route builder class JRebel integration kicks in and reloads the route definition:
JRebel: Reloading class 'filemonkey.MyRouteBuilder'.
JRebel-Camel: Updating CamelContext due to change in RouteBuilder

INFO Starting to graceful shutdown 1 routes (timeout 300 seconds)
INFO Route: route1 shutdown complete, was consuming from: Endpoint[file://src/data?noop=true]
INFO Graceful shutdown of 1 routes completed in 0 seconds
INFO Route: route1 stopped, was consuming from: Endpoint[file://src/data?noop=true]
INFO Route: route1 shutdown and removed, was consuming from: Endpoint[file://src/data?noop=true]
JRebel-Camel: Recreating routes from RouteBuilder 'routes1'
JRebel-Camel: Populating...

INFO Endpoint is configured with noop=true so forcing endpoint to be idempotent as well
INFO Using default memory based idempotent repository with cache max size: 1000
INFO Route: route2 started and consuming from: Endpoint[file://src/data?move=.done&noop=true]

For one route it happens quite rapidly so if you stick with best practices and don’t over-engineer the routes in your Camel application, the turnaround cycle will be very quick with JRebel.

Now what if we want to add more options for data transfers in our über-advanced application? For instance, what if we want to detect files that have “Moscow” in the element:

when(xpath("/person/city = 'Moscow'"))
  .to("file:target/messages/ru")

So we just added one more conditional path to the flow and XML files will be put to the new directory in case they match the xpath criteria. JRebel will reconfigure the route once again as soon as the changed class is recompiled.

Adding more logic and more classes is also totally accepted. For instance, if we wanted to make more custom processing for the files that go into the “ru” folder. In that case, we can add a new class that implements the Processor interface and use process(...) method to register in our pipeline:

when(xpath("/person/city = 'Moscow'"))
  .process(new MyProcessor())
    .to("file:target/messages/ru")

JRebel will locate the new class, reload the route builder class and reconfigure the route – all automatically! But nothing is perfect, although Camel integration is quite bulky in JRebel and it basically reconstructs the route definition entirely, it can handle most of the change scenarios in the Camel application. Only in the case of bootstrapping large components or adding Spring to the existing Camel application will require a restart, although in an ideal world you won’t have to make these kinds of changes many times each hour or day.

Summary

The world of data integration is getting better over time, and it’s a joy to experiment with the data integration when the API is so simple and the user experience is so smooth, as with Apache Camel. This is a very nice integration framework and I’m really glad that JRebel now supports reloading of Camel routes on the fly. For you Camel fans out there, I hope you’ll try it out with JRebel 5.2, now available for your downloading pleasure (and you can still win that trip to Ireland, just for evaluating!)

So, if you have any comments, please write them below, or tweet @JRebel.

Get JRebel 5.2 & rock Apache Camel!

  • Abhishekh Padmanabhan

    You mention that “Only in the case of bootstrapping large components or adding Spring to
    the existing Camel application will require a restart, “. Is there a specific reason why this cannot be supported ?

  • arhan

    Because it requires altering CLASSPATH at runtime, and that is not possible atm.

  • Sorin

    How can I make the Camel plugin work for routes defined using the Spring XML DSL ?