So you’re using Continuous Integration and Mercurial as your VCS. CI is great, you’re loving the feedback and feeling good after a green build. You’ve just got your project started and a couple of features are underway, but then you see a red build. “It’s nothing”, they’ll say, “just wanted to publish my changes, halfway through this big refactoring”. “One broken build is expected”, they’ll say, “just this one”.
Before long either developers stop publishing their changesets daily or you end up with a Continuous Misery instead of Continuous Integration, because the only color you see is red. About that time you (re)discover that branching is so easy in Mercurial, as easy as pushing or pulling changes from other repositories.
“I’ll just create a branch for these guys so they can easily publish their code to others, right?” and you create branches for every feature. The developers are happy and you CI build is always green, but soon you discover that your application is broken, despite the green lights on the build. “Oh wait, my CI jobs are all configured on the ‘default’ branch…”
If the above story sounds remotely familiar to you, then you’re not alone. Feature branches and Continuous Integration don’t play well together. CI purists think that feature branches are an abomination, but if you’re not a purist then you just want a working application and a clean mainline.
We value being pragmatic over being a purist. Sure, you could use feature toggles or branch by abstraction to emulate feature branches, but just running ‘hg branch “feature – x”’ is so much faster and simpler.
We’ve raised this topic once more, as a call for answers from the community, about how you guys handle such situations. This time however, we’d like your opinion about an idea that came to us.
Since our last visit with the “feature branches vs Continuous Integration” debate a few things have changed. Namely, a few projects have emerged that automatically help you clone jobs for feature branches in Jenkins:
- Jenkins autojobs supports Mercurial, Subversion and Git, but only takes in one job to clone for a feature branch
- Jenkins Build Per Branch supports only Git, but takes a set of jobs (identified by a prefix) to clone for a feature branch.
With the help of those projects you can start using CI for simpler projects, but what if your Maven project consists of multiple parts located in multiple repositories and you’d still like to use feature branches in them? And what if you don’t want to clog up your Jenkins views with an ever-changing array of jobs?
A unique problem arises of how to guarantee that your build tool doesn’t package your application from branch1 with dependencies from branch2 or branch3 or… The issue can be illustrated as follows:
I took this issue to heart and wrote a whole bachelors’ thesis out of it. In my thesis, I gave a brief introduction to both CI and feature branches, listing many alternatives to feature branches along the way. As mentioned earlier, going with feature branches may be the most practical way for parallel development and keeping the mainline clean.
The full thesis can be retrieved here and for people comfortable with CI and feature branches, reading chapters 3 & 5 should give just enough background to understand our proposed solution to these problems, but we’ll go over them shortly in this blog post as well.
We created a prototype Jenkins plugin called Feature Branch Notifier that works as a combination of a patched Mercurial plugin, new trigger plugin and a pre-build step, enabling us to detect updated branches with polling and change the branch the build will run on. Its configuration is similar to the default Mercurial plugin, with only one additional field called “Match branch names with” under advanced options, which can be used to filter branch names that new builds will launch on.
To enable the plugin you have to choose a new Source Code Management option called Mercurial (feature branch aware) and specify the branch as a special environment variable called $BRANCH.
Then you’ll need to select the new Feature branch aware Poll SCM option under Build Triggers and copy your trigger configuration from the usual Poll SCM trigger.
Lastly you need to check the Check and mark builds with feature branches checkbox, which enables a pre-build step that sets the value of the $BRANCH variable.
The way it works all together is that the new trigger plugin will search for updates across all your branches and fire new builds of the job if it finds any. Builds not running on the ‘default’ branch will have a special tag on them. The pre-build step looks for the tag in the build and sets the $BRANCH value accordingly, thus your build will run on different branches, which is indicated by adding the branch name to the end of the build in the Build History view.
To quickly launch new builds on a given feature branch, a new menu item is available on tagged builds to schedule a new build with the same branch tag.
The Integration Risk
By now you may be thinking that all this sounds fine, but what about integration risks? Builds running on different branches may prove that the code in that particular branch passes tests, but it gives no guarantees about being compatible with code in another branch.
These risks can be mitigated by borrowing the gatekeeper and branch updater notions from Bamboo. In principle they are different sides of the same coin. Gatekeeper merges the feature branch into mainline before each build and branch updater does the opposite, by merging the mainline into the feature branch. The principles of a gatekeeper can be illustrated as follows:
Both the gatekeeper and branch updater push the merge only after a successful build, meaning all of the plan’s jobs have successfully finished, which gives us the guarantee that the build is not broken if the merge is pushed to the shared code repository.
In Jenkins, executing shell scripts could simply mimic this functionality. One script that does the merge as the first build step and another that pushes the merge as the last build step, building and testing in the steps between. With the use of the “Feature Branch Notifier” plugin an additional check for the current value of $BRANCH should be added to only run the gatekeeper or branch updater, when the build is not running on the default branch i.e. when the $BRANCH variable is not ‘default’.
With the help of our plugin, you can now easily start launching builds on different branches without clogging up your views with different jobs. Though for many projects using components located in different repositories this might not suffice, as you still have the threat of combining your main application with a dependency from a random branch.
Our solution to that problem comes by using our prototype plugin in combination with some other plugins to create so called environment-locked multijobs. These configurations are complex and there’s only limited length to a blog post, though they are described in detail in chapter 5 of my thesis.
I hope some of you will find this solution at least interesting if not helpful and as always any feedback for the plugin or the solution in general is welcomed. Chapter 6 of my thesis also lists some needed improvements for this plugin and gives some ideas for alternative solutions as well.
In any case, using Continuous Integration shouldn’t exclude the use of branches in your project. It would be perfectly reasonable to move a couple of features to a separate branch and also develop on the mainline, when everybody is aware of the potential integration risks and active measures have been taken to get notified of incompatibilities as soon as possible.
Want to see more about Continuous Integration?
Visit our existing content on Continuous Integration with Why Devs <3 CI: A Guide to Loving Continuous Integration, a 30-page report by RebelLabs. Just click the button below to get access to the PDF! :)
[button label=”Get the PDF: ‘Why Devs <3 CI'" href="http://zeroturnaround.com/rebellabs/devs/why-developers-love-ci/" target="_blank" em="1"]