Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.

9 features you need to demand from a hosted continuous integration service

After the explosion of Github, where source code is located offsite, and the rise of cloud computing services such as Amazon Web Services, the next logical step is to move the build system itself to the cloud. We’ve talked about continuous integration before when we discussed the best Jenkins plugins that you want to install, how to visualize pipelines of the build jobs, and even created a report about how to get started and up to speed with Jenkins.

Continuous Integration, or Continuous Delivery if you take it one step further, are the final missing links into a fully automated software creation process that converts the code we write into a fully deliverable product artifact. With the code and the runtime being already on the cloud, it would only make sense to connect them together with a hosted build solution that would fetch the code and take it to production status.

Traditionally, companies have their build servers on premise. Jenkins is the most popular CI solution, getting as much as 70% adoption in our last technology landscape report, while Atlassian Bamboo is the second most used build server, although not that close to Jenkins at all. We’re running a similar report this year, so if you’re interested in seeing the changes to the Java landscape, please complete and share our 5 minute Developer productivity survey 2016.

Travis ci logo

The need to have on-site continuous integration servers changed quickly with the introduction of build servers that used the cloud. Travis is a great example of this and a well known solution that became very popular in the open-source world. A number of other vendors have since appeared that offer the same, if not better, functionality as Travis with several pricing schemes for both private and public repositories.

In this post we are just starting on a journey that will be the ultimate comparison of 14 hosted continuous integration services. While the topic might not be as fancy as the next container solution hitting the market, most available reviews only focus on 2 or 3 services and are often written in a biased way, like promoting a specific product. I decided to write an impartial review, since I am not affiliated with any of them in any way. From this series, you’ll know which continuous integration service to choose for your next project.

Today, we’ll look into some of the common traits for most of the hosted build services, and define what qualities you should be looking for when picking one. We’ll then apply this framework and talk in detail about every single one of the following hosted continuous integration products:

By the way, if you have a favorite in this list and want to see what I think about it sooner, rather than later, click on it in this survey and I’ll do my best to respect the results.

Powered by Typeform

Common pain points across CI services

With so many products to compare, I chose to focus on explicit Java support on the builds. Unfortunately, most of the issues I ran into were related to the format of yml files, particularly the lack of explicit Java support. It’s not particularly surprising, but a lot of products assume that you are a Python or Ruby developer, despite the constant dominance Java has on the TIOBE index and other programming language rankings.

The Cancer of YML files

Almost all products have a way to define your build in a yml file. I think that Travis was the first service that started this trend. Even though in theory the yml file is a good idea, in practice it gets complicated way too fast for no particular reason. It is a meta-build system that needs to be maintained along with your regular build files.

I see the need for a yml file if I need to do something special in my build or if I want to define an environment parameter, such as a mysql service for unit tests, but frankly I don’t understand why it is required for trivial tasks, like the basic compile step.

At least in the Java world there are not that many build systems. If my project contains a top level pom.xml file, there is a good chance that simply running mvn test will compile and test my code. So why do I need a yml file for this?

The syntax between yml files differs from service to service and not all of them have good documentation on the exact contents. If we talk about the levels of how comfortable yml file configuration is, here’s my take on what we could have:

  • The service autodetects your build system and runs the appropriate command
  • The service allows to you to create a yml file with a gui wizard
  • The service requires a yml file but you can override via gui options
  • The service requires a yml file and fails the build if there is none

I’ve tried multiple hosted solutions, and my repository history is now full of yml build file changes. This clutters the history and, frankly, annoys me that such a fundamental part of the system warrants so much effort on my part.

Lack of dependency caching

With Maven and Gradle your build will also download a lot of binary dependencies. Ideally these dependencies should be downloaded only once.

However all services start a new build with a fresh vm/container that has a clean state. This means that unless you do something about these dependencies before hand, each build will download them again and again. This makes the internet gods sad.

If your project is a big monolith, perhaps you don’t need to care as much about the dependency download as much because your build time might be heavily dominated by the compilation phase. On the other hand, if you have a lot of smaller projects, perhaps you follow the latest microservices trends, downloading all these dependencies can quickly become a problem. If you also employ the use of Gradle wrapper it also means that more time is spent on downloading Gradle itself for every build.

It is therefore evident that a caching mechanism is needed. Some services have special documentation pages that explain how you can cache dependencies. Some smarter services have built-in support for Maven and Gradle that automatically cache everything with no intervention from you.

Lack of love for Java applications

My final pain point is the lack of sane defaults for Java applications. For example why do I have to configure a path for my JUnit reports? If I have told you that I use Maven, then the path to junit reports is known in advance.

Jenkins has some smart support for showing my project’s code coverage and unit tests results without any special configuration needed. I did not find similar support in any CI service, even though most of them advertise themselves as a replacement for Jenkins.

What do I want from CI service

Now, I know that I might sound like a spoiled child saying “I want this” and “I want that”, and “My use-case isn’t properly supported”, but think about your typical project. If you pick a service that aims to provide a hosted continuous integration solution for your project, it has to be usable. Here’s a list of qualities and features that one can reasonably ask from such a service.

1. Connectivity


How easy it is to deliver your code into the service. Even though Github is the most popular solution, surprisingly not all projects use it. The base use-case is that the hosted CI solution integrates seamlessly with Github and BitBucket and you would be pleasantly surprised if other providers, say a self-hosted Gitlab are also supported.

2. First-time setup

Setting up a build server can be a time-consuming process, especially when your project is complex. Getting a prototype project to work on the CI might be easy, but you are not really done until your real world project is actually working, meaning that the artifact(s) that your build produce are equivalent to those from your local setup. I would expect a CI solution to guide me through my first steps with the project setup.

Some products have auto-detection capabilities that understands my project build system without additional configuration, such as .yml files. Kudos to those, although this is what I’d expect from any decent solution.

3. User Interface

Getting a build system up and running is just the beginning. Assuming you will have many different projects on your CI system, you’ll also need a way to monitor and manage them. Typically, the project requires a dashboard of sorts that organizes build reports and other useful navigation links.

I know it’s hard to define what makes a good interface. I believe however that everybody can instantly recognize a bad interface that prevents you from accomplishing your task. At a minimum, the hosted solution should give me an overview of all my builds and allow me to drill down into each specific build for more detailed information.

4. Build environment

build environment
Perhaps the most important functionality of the hosted continuous integration solution. It includes the programming languages supported by the product, but as I said already I usually focus on Java, the versions of compilers present, build systems and how easy it is to update or install new software. A product gets higher points if it has extensive configuration properties and allows me to define the build environment in an intuitive way.

5. Feedback

With the build in place, there are several things that can go wrong with it. Incorrect environment setup, failed unit tests, unresponsive external services and so on. What many teams would want is real-time monitoring of your builds, and excessive logging on things that went wrong. A solution might even be to allow the ssh connections to the build servers. This is a really nice feature to have, particularly when the build acts in a really unpredictable manner.

6. Post-build steps/deployments

Once your build completes, you’ll need to do something with the results. Maybe deploy the artifact to a staging server, send an email, upload it to FTP, update a bug issue or anything else that fits in with your process. A product gets high marks in this area if it offers several post-build hooks to allow you to take advantage of the build output.

Extra kudos, if the service supports integrations with specific cloud providers like Amazon, Heroku and CloudFoundry for example.

7. Enterprise features

enterprise features
Getting simple web projects to build might be easy, but as we mentioned not all projects are that simple (not all projects are web projects also). Support for multi-module builds, pipelines, pull requests, and having a huge number of active projects with long build history will be examined in this area. Products get high points if they cater to complex build processes. This is one of the categories that has of lot of diversity, as different companies have a different mindset on what exactly an “enterprise” wants.

8. Documentation

There is no such thing as too much documentation. Unfortunately not all companies embrace this idea. Giving access to quick tutorials is one thing, having a complete reference guide is another. Products get high points in my book if they come with a detailed explanation of how they work and what they do.

9. Ease of contact

It is very hard to rate “communication” in an objective way. But given how important the continuous integration environment can be, proper support channels are imperative. For example, we could easily be dealing with a continuous deployment pipeline, where a build actually means a release.

My expectations

In general, I was writing this post with the ultimate comparison of CI solutions in mind. So I listed my base expectations to the functionality that they provide and got something like the list of the features we discussed above.

Initially I had planned a complex build scenario with unit/integration tests, deployment stages and even multi-step builds. However as my research continued I saw that due to time constraints I needed to focus on the basic scenario. My base case is the following:

  • Fetches code from github
  • Compiles the code while giving me real time feedback on the build log
  • Archives my artifacts (i.e. war file)
  • Allows me to add post build steps (send an email, ftp something etc)
  • Has at least a reference guide for documentation
  • Is responsive via Twitter/Email or contact form

Doesn’t sound too complicated, does it? However, when I was looking at the available solutions and making notes, the results astonished me.


In this post I summarized the common experience Java developers get from interacting with hosted continuous integration solutions. This is an introduction to the ultimate comparison of CI tools and starting next week we’re going to review the hosted CI tools one by one and report our findings, expectations and opinion. In the end, it will be a comprehensive and neutral comparison of the services available, so you won’t need to try them all yourself.

Pick your favorite CI solution in the survey above, and we’ll make our best effort to review that first. The survey is going to be active for a week or so, until we lock the results and establish the order of the reviews.

Did I forget anything that is important about the continuous integration experience? Ping me on Twitter: @codepipes and let’s chat.