Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

Conquering job hell and multiple app branches using Jenkins & Mercurial

Branching

In the past, I’ve written a lot about Jenkins, Mercurial (see all the cool links below) and other tools that we use at ZeroTurnaround. Today, I thought I’ll share with you how we manage multiple branches in Mercurial while still enabling a Continuous Integration experience for our development teams. I’ll concentrate on one of our products, LiveRebel (which, incidentally, we actually use to release over 30 applications of our own, twice a week, all without impacting our users. Dogfooding it baby!)

Our approach uses tools freely available on the market and some custom scripts. If you have multiple branches and struggling with CI, then this article might give you some ideas.

How we use the Branching Model

jenkins-logo

To set the stage, I really do believe that true Continuous Integration is with a single mainline that developers commit their code to. In reality, I doubt if there are many teams out there with the luxury of having a single mainline that everything can be merged to. At one point in any long-term project, there will be multiple version-based mainlines in existence, and branching is really the way to deal with this.

Let’s use the ZeroTurnaround product LiveRebel as an example, for which we have versions 2.6, 2.7, 3.0 and 3.1. Some of them are in active development. Others are not. But all of these branches get at least some development and releases, but at the same time only share roughly 70% of the functionality.

To make matters a bit more difficult, we also have a separate staging branch for each version branch. So, LiveRebel 3.0 also happens to have a 3.0 staging branch, and it’s this staging branch that we do a release off of. These staging branches came to be a couple of years ago when we were feeling that whenever we started a release we couldn’t commit anything to that branch.

Releases take time. Especially if your project is multi-module Maven project with huge artefacts that bring release time itself to about an hour. Besides the release time, we also don’t want to see any commits to the source tree during that time because they might blow something up and delay the release even more. So, to avoid a code freeze we introduced the staging branch, which is what we release from while we continue to code away on the version’s mainline. Simple, and works really well for us.

Playing nice with Jenkins and a gazillion jobs

Of course, when you have many branches in Mercurial (or Git, SVN, etc.), you will need a lot of jobs in Jenkins, naturally gaining the benefits of CI for all of them. Managing all this is another question, however, and I’ve discussed this at some length in my StackOverflow question about handling multiple branches in CI.

The first thing was to install the Jenkins Nested View Plugin. This plugin gives you multiple levels of views, i.e. the first level is the project/product name, the second level is mainly branch names and only then you see the job listing.

We opted to custom scripts to manage the inevitable job hell and we prepared scripts that copy the jobs from view to another. So, for example, when we kick off a new branch we also create a new set of jobs.

jenkins-autojobs-desc

We also have scripts that “sync” these views so that when you change some jobs in a view you can get those changes propogated to another view.

Of course, this all seems a lot of work for such a typical problem. Jenkins & Git (compared to Mercurial) can manage multiple branches within a single job, but this might not be a fit if your project has evolved and you need to build some branches differently. In any case, here are a couple of similar approaches:

  • Job DSL Plugin – manage job definitions and creation with a DSL and templates
  • Jenkins Auto Jobs – automatically create jobs from templates and branches in SCM.

Conclusions

Multiple branches and CI is not a solved problem. For the purists, it is easy; just use a single mainline or special integration branch. We’ve tried and failed at that and started using multiple branches and multiple sets of CI jobs for the branches.

The solution with Jenkins and these scripts is a bit nasty, and as the guys build LiveRebel we certainly don’t suggest adding more manual scripting to any process, but this is our own mess to deal with. Now that there are plugins available to manage your jobs it makes even more sense to try to scale Jenkins jobs to your branching habits.

Of course, I do hope there will be a cleaner solution to this problem in the following years!

Related Posts

Here are a bunch of links for great content that we’ve written in the past on Jenkins plugins, Mercurial extensions, job chaining and more. Check them out!

  • Oleg Šelajev

    We could look into https://wiki.jenkins-ci.org/display/JENKINS/Literate+Plugin which is supposedly able to build multiple branches of a single project. I don’t know if that’ll work out of the box with our jobs structure, but it’s something we should look into.

  • Andrzej Majewski

    Dealing with multiple branches and jobs is a nightmare. I have used following approach
    http://entagen.github.io/jenkins-build-per-branch/

    which works base on templates. It work very well as I could stop manually process of setting up jobs for every new branch. The jobs would come and go as new branches were created and deleted.