Blog

Job chaining in Jenkins is the process of automatically starting other job(s) after the execution of a job. This approach lets you build multi-step automation pipelines or trigger the rebuild of a project if one of its dependencies is updated. In this article, we will look at a couple of plugins for Jenkins job chaining and see how to use them to build and visualize these pipelines.

• Out of the Box Solution
• Build Pipeline Plugin
• Parameterized Trigger Plugin
• Downstream Buildview Plugin
• Conlusions

Out of the box solution – Build Other Projects

Jenkins has a built-in feature to build other projects. It is in the Post-build Actions section. You can specify the projects that you want to build after this project is built (you can trigger more than one). So whenever project A is built you will trigger the building of project B. You can also specify the conditions when the other jobs are built. Most often you are interested in continuing with the pipeline only if the job is successful but your mileage might vary.

One thing to remember here is that this feature has two configuration location. You can configure project A and specify a post action as in the previous screenshot. Another option is to configure this from project B and say “build this project B only after project A is built”. You don’t have to fill out both, just change one and the other is updated. See the next screenshot for the second option.

Build Pipeline Plugin

Build Pipeline Plugin is one interesting plugin. The main features of this plugin is to provide visualization of the pipeline and also to provide manual trigger for continuous delivery purposes. The configuration is a separate Post Build action where you can configure which projects should be built after project A. By default the triggering is actually done manually by the end user! If you want certain steps of the pipeline to be automatic then you have to use the built-in job chaining (see the Out of the Box Solution for more details).

The pipeline plugin offers a very good visualization of the pipeline. By configuring a new Jenkins view and choosing which job is the first job in the pipeline you can get a visualization of the whole pipeline. In the screenshot, be sure to note that one of those steps is manual and the result are automatic. The manual one can be triggered from the very same view.

Parameterized Trigger Plugin

The Parameterized Trigger Plugin is another triggering plugin but with a twist: this plugin lets you configure more aspects of the triggering logic. It covers the basic Out of the Box Solution features and adds many more. The most important one is the option to trigger the next build with parameters. For example, by defining SOURCE_BUILD_NUMBER=${BUILD_NUMBER} you are able to use the variable $SOURCE_BUILD_NUMBER in project B. This way you can, for example, use the artifact built in the previous job to be fetched from your central artifact repository using the ${BUILD_NUMBER}.

Downstream Buildview Plugin

The Downstream Buildview Plugin plugin that does not do job chaining itself, but provides a means to visualize the pipeline. Similar to the Build Pipeline View but more dynamic. You can click on any item in the build history and have its pipeline visualized.

Conclusions

The main feature that makes Jenkins so good is that there is always an app plugin for what you need. Of course, the same fact also highlights its biggest weakness. It is rather difficult to choose the correct plugin and very often you need a couple of plugins to achieve your goal. The same is true for job chaining and visualization.

The job chaining features that we covered in this post all provide the minimum functionality – triggering other jobs. The Parameterized Trigger plugin is the most feature-rich, but lacks the manual triggering. The Build Pipeline only offers manual triggering and you need to figure out automatic triggering yourself (using the built-in feature for example).

From the visualization side, the Build Pipeline plugin is definitely the best looking. At the same time, the plugin does not support passing parameters (the latest alpha build is a bit better) and once the pipeline gets long it gets a bit ugly. We do like the part of defining a separate view and then always being on top of your pipeline. The Downstream Build View plugin gives you great flexibility and insight to job chaining, but does not enforce any kind of process.

So, there are the Jenkins plugins that we use at ZeroTurnaround for job chaining and visualization. DO you use the same tools? If not, can you recommend any others? Which are your favorites? Please leave comments below!

Last week was a great week for ZeroTurnaround. On Monday we grew by 10% head count wise and I plotted the trend. It was humongous. My calculations showed that we will reach 600 billion people by the end of the year. This introduced a lot of questions and couple of interesting threads. Also it meant that our first intern ever hired had a lot more on his plate than first expected. I’ve compiled one of the public threads just for you guys. It tells you how and why we have a document Will ZeroTurnaround reach Gravitational Singularity and also to state that we actually don’t pack people together 100% flesh to flesh.

Play! Framework is getting a lot of praise. It is growing in usage and popularity. You can spot presentations about it at conferences, numerous blog posts etc. Even I praised my favorite features of the Play! Framework couple of months ago, My Top 5 Play Framework Features. There is so much positive feedback about it, that I thought it would be a bit refreshing to list the features (um, “un-features”?) that really get on my nerves. Here is my list of Play! un-features that limit my productivity, encourage me to read slashdot.org, and really irk my inner geek…

Application Startup Time Grows with Application Size

If you create a new application in Play! and start your app via play run then it takes about 2 seconds for the application to start. Basically just the time to switch to your browser and hit refresh. Once your application grows in size the startup time also grows. Our application starts for 28s in development mode (on my MBP with a SSD). This is already enough time to take a quick peek at your Google Reader. Of course the same is true for any JEE application, at first they are blazingly fast but as time goes on the startup times just grows and grows.

Reloading 3rd Party Library Code

Play! Framework applications have support for really quick turnaround because a large part of the application code changes can be reloaded instantly. So you add a method, change params and you don’t need to restart the app. This is all freakin’ awesome, but the moment you happen to develop a library that you are using within your application, you need to make room for restarts. Every code change then requires a restart. Not so happy anymore (let me find my jrebel.jar file!)

Memory Hog

Play! Framework templates are quite the memory hog. You might not notice it in the beginning, but once you have some larger pages and more users, then you will see a spike in memory consumption. This is due to the design of sub-templates. The sub-template is read into a String variable on every request that uses it. This will spike memory usage to the effect of the HTML file size. You can bypass this by giving your app more memory, having smaller templates and moving static content into static files instead of templates. There is also a more effective memory usage on the way in the 1.2.x branch.

Fear of Upgrade

Upgrading has been a bumpy ride so far, (the minors not so much) and the leap from 1.0 » 1.2 was quite scary. Our tests started failing left and right once we packaged our application. We stumbled upon quite a few  bugs and the choices in the end were to either use our custom build of Play! framework, or downgrade. We chose the first option. Now another major version is looming and a lot has changed again–check out the graphics at the 2.0 page. I’m afraid that our products will need a major overhaul if we want to get on to the next bandwagon. It just scares me.

An Issue Here and There

Play! Framework is guaranteed to surprise you when you least expect it. Just the other day I got a report from a colleague, saying that he got a verify error from Play! code. Ok, great. What can I say, try to reproduce and file a bug report? Or what about these:

• You’re programming some feature and suddenly you start getting ClassCastExceptions. Oh well, I restart and I’m unable to reproduce. Not enough for a bug report, but there seems to be something fishy with Play! Framework reloading and Jobs. Oh well, lets hope it won’t happen again any time soon.
• Everything is super until one of the Jenkins nodes is showing failing tests. You dig for couple of hours and you find there is special handling of a folder with a dot preceding it, finally resolved.
• You find a controller method that produces OutOfMemoryErrors because of the dark bytecode magic that is happening behind the scenes. You’re able to reproduce it, but when submitting a bug report you try to reproduce on an empty application, which just won’t happen. You add a @ByBass to your own method and the problem goes away. Argh!

These small issues that pop up suprisingly end up stealing confidence and time. You start doubting the underlying system and lose the edge that this tool is supposed to give you. Just imagine the problems you will stumble on once Python, Groovy, Ivy is replaced by sbt, Scala, Akka.

So why do I still use Play! Framework?

When writing this piece @ekabanov asked my why on earth are using this framework then? My answer was, well, we would have just as many issues with any framework out there, but the nature of those issues would probably be different. You will always stumble upon bugs, you will always have problems when the application grows in size, and, of course, you will have problems when upgrading your dependencies. So far, there is nothing to be done.

I’d love to hear about your pet peeves with Play! Framework – please leave your comments below!

Last week was pretty good, thanks for asking. We’re all booked up for JavaOne (come geek out with us at booth #5510, and get a free JRebel or LiveRebel license) and the Jenkins User Conference in San Francisco in early October, and persistently edging towards our 1.1 release planned for the same time. Today we are releasing the first public version of it, Milestone 2. The release candidate is planned for next Friday. The main features of the 1.1 branch is full control of the application lifecycle. This means you can use LiveRebel to do your initial deploy, updates and undeploy. Also the CLI has matured a lot and has more *nix like tool features. For a complete changes check out the changelog. You can grab this release from the LiveRebel archives page.

While planning my trip to San Francisco for JavaOne (catch me there for a beer), I saw that there is a Jenkins User Conference coming up October 2nd, 2011, just prior to JavaOne. As I was signing up for the event, I started thinking that we’ve been using Hudson/Jenkins for years now. Through the years, certain Jenkins features/plugins have really paid off and been life savers for getting higher-quality software out quicker. So I thought, why not share this with you? Here is a list of my own and the team’s top 10 must-have Jenkins features/plugins.

• Easy Installation
• Easy Upgrades
• Scriptability
• View Filters
• Maven
• Amazon EC2
• HTML Publisher
• Throttle Builds
• Join
• Green Balls

(more…)

So, you’ve got Jenkins running nicely, and you’re proud of your Continuous Integration environment, but you’re looking to move towards Continuous Deployment. Today we’re announcing one step in that direction: combining Jenkins with LiveRebel.

Just a quick reminder, LiveRebel is a tool to manage your production updates. This means that instead of experiencing downtime or longs hours of session drain during the update you get instant, session preserving, no downtime nor OutOfMemoryError experiences. You also get to rollback changes instantly. Now imagine orchestrating this from your favourite continuous integration tool.

In May we released LiveRebel 1.0, and then went on to making it easy to configure and setup in different environments. We’ve covered how to use the CLI to script LiveRebel with bash and how to manage clustered Glassfish installation to run instant updates. Now we have turned to managing your deployments with your Jenkins/Hudson continous integration server.

You can now install the LiveRebel Deploy Plugin from the Jenkins plugin listing. A tick of a checkbox and a restart to your Jenkins installation is all that’s required. We’ve also published a How To Setup LiveRebel Deploy Plugin for Jenkins/Hudson doc, for your setup pleasure :). The plugin lets you run continuous deployments from Jenkins. This means you can turn most of your deployments into instant deployments that preserve state and don’t cause OutOfMemoryErrors.

Next features for the plugin are: support for Hudson (currently the installation is a bit more complicated) and the ability to use LiveRebel for the whole app lifecycle (currently we fallback to Cargo deploy plugin for initial deploy and not supported deploys). Stay tuned for updates!

It’s not surprising that directly after the release of LiveRebel 1.0 last month, we’ve been working hard on get the next version of LiveRebel ready – v1.0.1. To briefly sum up, we have been working on:

• Support for more Java EE Containers – WebSphere, WebLogic
• Real-life testing of 10GB+ of WAR files
• Started working on a Jenkins/Hudson plugin
• LiveRebel Scripting

Support for more Java EE Containers

Our main focus in this release has been to add support for more Java EE containers. This job involves adding the beta containers into our test suite and creating container-specific tests. Of course, these tests need to earn a PASS before we can take a container out of beta. With this release we bring IBM WebSphere and Oracle WebLogic fully out of the beta phase.

Real-life testing of 10GB+ of WAR files

At the same time, we’ve been using a lot of our internal tooling to analyze a large number of WAR files. We’ve gone over 10GB of web application versions to test our tooling on real-life projects. The results from this will be released in due time, once all the info is analyzed. This process has brought couple of fixes to our command-line tooling.

Jenkins/Hudson Plugin for LiveRebel

During this time we started work on the Jenkins/Hudson plugin. This means that you can use LiveRebel to deploy artefacts built with your CI to staging or production. This initiative is still in beta right now but this also has introduced some new internal features and fixes.

LiveRebel Scripting

One other direction we are moving to is the scripting of LiveRebel. We’ve released CLI documentation and also a how-to article on scripting LiveRebel with Bash. This department will have some new features in the following release and there are also articles in the pipeline on how to script against our REST API.

All in all, it has been a really busy month for us. Besides moving into a new office with Starcraft TV and hiring like crazy, we’ve been able to bring many fixes and changes to LiveRebel. Checkout the full changelog and head off to download your copy – LiveRebel is still free for 90 days here!

One of the features that we are looking forward to more people learning about in LiveRebel is the command line interface (CLI). You can run most of the LiveRebel functionality from the command line. There is a manual that covers most of the CLI aspects available on our LiveRebel documentation page. In this article we will use those bits and pieces to write a script that will run automatic updates.

We will be using bash as the scripting language. The end script is available from the LiveRebel Scripts repository.

Some assumptions:

• We assume that you have installed and configured LiveRebel, and you have access to the LiveRebel lr-cli.sh script (in the LiveRebel installation’s bin directory).
• We assume that you have a script to run a full redeploy/restart when LiveRebel does not support the change.

You’ll need a LiveRebel authorization token found inthe LiveRebel Web Console as shown here:

Let’s start with writing the script by defining the TOKEN and LiveRebel home directory and the lr-cli.sh locations.

#!/bin/bash
 
# LiveRebel Command Center token
TOKEN="ad7af931-3cf6-4000-967d-925dcfc44955"
# LiveRebel Installation Dir
LR_HOME=/Users/toomasr/Downloads/liverebel
CLI_SCRIPT=$LR_HOME/bin/lr-cli.sh

Next, let’s use the 1st parameter to the script as the WAR file that needs to be uploaded to the LiveRebel Command Center. We will take the output of the upload command into a variable called RESULT and later on use that to parse the response.

FILENAME=$1
RESULT=$($CLI_SCRIPT -token $TOKEN -url localhost:9001 upload -file $FILENAME)

Now to parse the response. The response has 3 different outcomes and based on the result we are also interested in the name of the application (the application name can be different from the WAR archive name, it is defined in the liverebel.xml file inside the WAR archive).

• Unable to connect to LiveRebel Command Center
• Application has already been uploaded
• The upload was a success

if [[ $RESULT == *refused* ]];
then
    echo "Unable to connect to LiveRebel. Are you sure your server is running?"
    echo "Exiting..."
    exit 1
elif [[ $RESULT == *ERROR* ]];
then
    VER=`echo $RESULT | grep -o "Version [^a-z]*"`
    VER=${VER#Version }
 
    APP=`echo $RESULT | grep -o "of application .* already exists"`
    APP=${APP#of application }
    APP=${APP% already exists}
else
    VER=`echo $RESULT | grep -o "Version: .*"`
    VER=${VER#Version: }
 
    APP=`echo $RESULT | grep -o "Application: .*Version: "`
    APP=${APP#Application: }
    APP=${APP% Version: }
fi

If everything went well, then by now we have either uploaded the WAR file or it already exists on the LiveRebel Command Center. We’ve also extracted the name of the application and the version. The next step is to figure out the active version of this application. We will later on use this information to run the compatibility test.

# find out the active version
RESULT=$($CLI_SCRIPT -token $TOKEN -url localhost:9001 vers -app $APP)
ACTIVE_VER=`echo $RESULT | grep -o "Active: .*"`
ACTIVE_VER=${ACTIVE_VER#Active: }
 
if [[ "" == $ACTIVE_VER ]];
then
    echo "The application is not deployed to any of the servers. Not doing anything"
    exit 1
fi
 
echo "Active ver $ACTIVE_VER, uploaded $VER"
 
if [[ $VER == $ACTIVE_VER ]];
then
    echo "You are trying to upload the same version. Not doing anything"
    exit 1
fi

We have the active version of the application now. Now let’s find out if the update from the active version to the version we just uploaded is supported by LiveRebel or not. If it is compatible, then run the update – otherwise use your fallback solution.

RESULT=$($CLI_SCRIPT -token $TOKEN -url localhost:9001 diff -app $APP -from $ACTIVE_VER -to $VER)
 
if [[ $RESULT == "Compatibility: compatible" ]];
then
    echo "Versions are compatible. Activating version $VER"
    $CLI_SCRIPT -token $TOKEN -url localhost:9001 update -app $APP -ver $VER
else
    # your own custom script to run the redeploy
fi

Voila! and we’re done. This article serves as an example how to interact with LiveRebel Command using the CLI. Your mileage may vary and you might want to invoke the LiveRebel script from your own master script. If that is the case, you will need to change a few things here and there. But just refer to the CLI documentation and use the commands one by one.

PS! We are working on a Jenkins/Hudson plugin that would do all this automatically. In case of incompatibility it would either use the Cargo plugin as fallback option. This plugin is available from the LiveRebel Deploy Plugin repository.

As we enter the final race to the stable release of LiveRebel we’re happy to announce that  the LiveRebel 1.0 RC2 have passed all tests and has been uploaded to liverebel.com from the ZeroTurnaround Tartu labs.

This release doesn’t introduce any new features. We’ve fixed several small issues, run our product on an even larger testsuite and tweaked the UI here and there. If you are testing out LiveRebel be sure to upgrade. We are on track for 1.0 release on 17th May, stay tuned.

We have just released the last milestone of LiveRebel Beta, M12. We will continue with release candidates from this point on. We were able to complete all the features critical for 1.0

• We redesigned the UI and reworked UX (thank you guys at Fraktal)
• We reduced runtime performance overhead to under 3%, as measured by SpecJVM2008 (a 5x improvement over previous milestones)
• We upgraded internal Play! Framework to version 1.2

The only feature that we have yet to deliver is LiveRebel Failover. You can have multiple Command Centers in charge of your cluster. This ensures that you don’t need to make changes to your cluster when something happens with your command center machine. This is something planned for RC1 due Friday the 29th.

In the meantime, check out our survey, How do you update your Java EE app in production? It’s a single page that takes a couple minutes to fill out…plus it lets you try to prove Jevgeni, our CTO, wrong about his assumptions!

Other than that please get a shiny new copy of LiveRebel at our download page. Be sure to do a fresh installation.

Older Entries »