Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

The correct way to use integration tests in your build process

In my previous article on unit testing, I showed you how to mock all the dependencies of a single Java class in order to isolate it from the rest of the system. This way the class under test is in a controlled environment where you can test its own business logic with nothing else in the way.

However there are some times where you should NOT use mocking during testing. These are the times where you are actually interested in the interface of this class with its dependencies. You zoom out from a single class and you want to test a component of the system which not only is based on multiple classes but it might also use other systems such as a database or a queue.

For these cases you need an integration test. All the previous articles in the testing series talked about unit tests so far. And yes, there is a difference!

What is an integration test

Sometimes there is not a clear distinction on what is an integration test and what is a unit test.

My basic rule of thumb is that if

  1. a test uses the database
  2. a test uses the network
  3. a test uses an external system (e.g. a queue or a mail server)
  4. a test reads/writes files or performs other I/O

…then it is an integration test and not a unit test. I have seen several developers who talk about “tests” and either they mean both or just integration tests. Here is also a brief comparison between the two.

Unit test Integration test
Results depends only on Java code Results also depends on external systems
Easy to write and verify Setup of integration test might be complicated
A single class/unit is tested in isolation One or more components are tested
All dependencies are mocked if needed No mocking is used (or only unrelated components are mocked)
Test verifies only implementation of code Test verifies implementation of individual components and their interconnection behaviour when they are used together
A unit test uses only JUnit/TestNG and a mocking framework An integration test can use real containers and real DBs as well as special integration testings frameworks (e.g. Arquillian or DbUnit)
Mostly used by developers Integration tests are also useful to QA, DevOps, Help Desk
A failed unit test is always a regression (if the business has not changed) A failed integration test can also mean that the code is still correct but the environment has changed
Unit tests in an Enterprise application should last about 5 minutes Integration tests in an Enterprise application can last for hours

You should now know the difference between the two.

How to write an integration test

Writing an integration test is heavily dependent on your environment. The first thing that you should decide is the scope of your integration test. So if we keep following Matt from the previous article who keeps on working on the gigantic CRM, he could write integration tests for:

  • Verifying correct integration of the CRM with the printer (in a staging environment of course)
  • Verifying correct integration of the CRM with the mail server
  • Verifying correct reading/writing of invoices from/to the DB
  • Verifying the whole data flow of receiving an order, creating an invoice, saving it to the DB and mailing it to the client. This is an End-To-End integration test

Since some of the integration tests in the case of Matt, use a staging environment (e.g. the mail server) it is also important to document these dependencies so that his fellow developers know about them. I always hate it when I run the test suite on a new application and half the tests fail because my workstation has no network access to the testing database!

A second point with integration tests that must be accounted is the use of detailed logging. When a unit test fails it is very easy to understand why since the scope is very narrow. When an integration test fails, things are not so simple. Because by definition an integration tests is based on many components and a specific data flow, identifying the failure cause is not always straightforward.

My recommended way to alleviate this problem, is the use of detailed logging statements (that are always needed in an Enterprise application regardless of unit tests). This way, when an integration test fails you can examine the logs and understand if the issue is in the code or in an external resource used by the test.

Why integration tests should never run together with unit tests

Now we reach the most important point regarding unit tests. In a big enterprise application integration and unit tests MUST be handled differently. Here is an all too common scenario that I have personally seen multiple times.

Matt has created a lot of unit and integration tests. All of them are executed by Maven when the test goal is run. However during a server migration some of the integration tests stop working. However everyone on the team is busy and nobody fixes the IPs in the configuration files.

Soon after some integration tests that depend on an external system run really slowly. But nobody has time to investigate the cause. Developers no longer run tests before committing code because the test suite is very slow. More unit tests break as a result, since developers do not maintain them.

New developers come into the team. They start working on the CMS and soon find out that half the test suite is broken. Most of them do not even bother with unit tests anymore.

A valiant developer comes into the team and says that this madness must stop. He spends a day and finds out that the effort required to fix all tests is not realistic for the current timeframe. He also finds out that in several cases the unit tests are broken because of changes in the business requirements. So fixings the tests is not a straightforward process since somebody has to adapt them to new code.

By this point it is clear that tests are not actually used in this project. New developers simply declare that “writing unit tests is a waste of time” and they are right from their point of view, since nobody wants to work with a broken test suit

This is a scenario that we need to avoid.

Delegating integration tests to Maven Failsafe plugin

There are many ways to split unit and integration tests. My suggestion is to use the Maven failsafe plugin.

Unit tests should follow the naming convention introduced in the first part of this series. Unit test classes are named with “name of class + Test”. Then they are placed in the test directory of the Maven project structure.

The unit tests are executed automatically when the test goal is run.

Next you should add the failsafe plugin in your pom.xml file.

<project><build>
    <plugins><plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.13</version>
        <executions>
          <execution>
            <id>integration-test</id>
            <goals>
              <goal>integration-test</goal>
            </goals>
          </execution>
          <execution>
            <id>verify</id>
            <goals>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Your integration tests however hava a different naming convention. They are named as “name of class + IT”. IT stands for Integration Test.

Now the test goal will ignore them. Instead these tests will be executed by the integration-test goal which is a built-in goal into Maven. Here is a table that summarizes this split

Unit tests Integration Tests
Located in Maven test directory Maven test directory
Naming convention name of class + Test name of class + IT
Example class name BasketWeightTest.java InvoicingProcessorIT.java
Managed by Maven surefire plugin Maven failsafe plugin
Executed in test goal Yes No
Executed in integration-test goal No Yes

How to run integration tests in your build process

Now that all these changes are done you have great flexibility on how you run unit tests. Most importantly your build server (e.g. Jenkins) should contain a mixture of jobs that deal with both kinds of tests. Here is a overview of suggested jobs.

Job type Schedule Description Tests
Main build Every 15 minutes or half hour. Only compiles and runs unit tests. Should finish in 15-20 minutes max Only unit tests
Integration build Every 24 hours (usually at night) Runs integrations tests. Can run for 2-3 hours All tests
QA build Manually Deploys to a QA environment All tests

The suggested workflow is the following

  1. Developers run the test goal during development
  2. Developers run the test goal before any commit
  3. Developers run the integration-test goal before a major commit with many side effects
  4. Build server compiles code and runs the test goal every 15-30 minutes (main build)
  5. Build server compiles code and runs the integration-test goal every day (integration build)
  6. Build server compiles code and runs the integration-test goal before a release to QA

With this workflow it is clear that developers get fast feedback from the unit tests so anything that breaks can be fixed immediately. The boring process of running integration tests is left to the build server which runs them automatically in a well defined schedule.

Conclusion

In this post we have finally tackled integration tests. We showed you the differences with unit tests regarding test focus, external systems and running time.

We also hope we convinced you that they must be handled differently. Our suggested method for splitting tests is the maven failsafe plugin.

Finally we proposed some combinations of build jobs and how they run different types of tests.

Feel free to post your suggestions below regarding unit and integration tests.

  • http://zeroturnaround.com Jevgeni Kabanov

    Isn’t it an even better idea to run integration tests in dedicated environments orchestrated by Jenkins, configured by Chef and provisioned by Vagrant or virtualization?

  • Alex Soto

    Jevgeni you are partially right you can do it too. But I usually run integration tests similar to the production one but without virtualization (for example using AS embedded server or embedded DB without replication, mirroring, ..), but when I jump into virtualization (if I cannot have a replica of the production environment) is in tests like acceptance tests, deployment tests, …

    In my case this way works well because I’m not mocking all my “dependencies” systematically (http://antoniogoncalves.org/2012/11/27/launching-the-nomock-movement/).

    Well this is my approach, which in my case fits perfectly.
    Alex.

  • Alex Soto

    Hi Kostis really nice post which summarizes very well how to run integration tests. Only pointing out that you can achieve the same without using failsafe plugin, you can use surefire with exclusions, but of course failsafe approach is also good.
    Good post.
    Alex.

  • http://twitter.com/wytten Dale Wyttenbach

    An alternative to the failsafe plugin is to use JUnit 4 Category annotation to label tests as integration tests, and then use maven profiles to exclude them from developer builds, but run them nightly in the CI server. This week I also had the epiphany that once you exclude integration tests you can apply JUnit 4 parallelism to your mock object tests to make them run even faster.

  • anonymous

    Hello,
    it would be great if you could add in your blogs links to the previous articles in the series.
    This would make it much easier to read the whole thing…

  • anonymous

    Sorry, to be clear, put in one place links to all other articles, not just the previous one.

  • Kostis Kapelonis

    Yes, JUnit categories is another option. With this you can even split your unit tests even further. Like unit tests, slow integration tests, fast integration tests, tests that need something special etc. I have never tried JUnit parallelism, good to know!

  • Kostis Kapelonis

    Yes there are multiple way to do the split. Dale also recommended JUnit4 categories. What I really like about the failsafe plugin is the naming convention. A test is instantly recognizable by its name.

  • Kostis Kapelonis

    You can click on my name directly below the title of this post and you will get the full list of the blog posts.

  • Kostis Kapelonis

    I think it depends on the application you write. There are Enterprise applications that don’t need anything special. They are just CRUD web interfaces over a relational database.

  • http://twitter.com/redlabbe balder

    Yes, the ultimum integration testing! Build done -> launch VirtualMachines with jenkins, optionally controlled with Vagrant and setup with puppet or chef, deploy apps/db and what’s needed for the app and test against that. Tests done, kill environments and release resources.
    Always a clean slate!

  • Anil Kumar

    hey can u tell me that how to import build solutions from build server on integration server..
    u can also mail me documents regarding this problem on my email id
    my email id is : aboys888@gmail.com

  • http://agung-setiawan.com/ Agung Setiawan

    i’m a little bit confuse. When i’m writing test that use configuration files like spring-security.xml and web application context do i still call it unit test?

  • Othon

    Hi Kostis,

    I really enjoyed reading your articles about testing. You did a great job explaining unit test, integration test and test coverage. The dos and don’ts and when and where use what type of test. Thanks a lot for this.

    I’m bookmarking your articles for future references.

    Thanks

  • Stein

    Very good blog, but I strongly do disagree with the time span on how you fragment stuff.
    The reasoning why integration tests should be in its own build, I do not agree on. When they are in their own build, its harder to detect, the integration tests will often be delegated to teams that knows next to nothing on how to solve them, or you end up with an “integration testing team”, and they take longer time to fix.

    In my opinion, integration tests should be in the main build, and the developer’s should run the build often – as an absolute minimum it should be merged locally and ran before pushing from local machine. That way the developer that is doing the changes, also have to take responsibility for changes in integration tests, and can not be ignored.

    Now, if you have a 15-20 minutes build on an application without integration tests, and/or the integration tests take 2-3 hours, I would say that you have a completely different issue that really should be addressed! In my oppinion, a build – with integration tests, should not take more than 5 minutes, and you should consider splitting into more applications when you pass 3 minute build. But as I stated, this is my opinion…

  • John

    It seems to me that your application in not really large. You can’t possibly test a banking system with integration tests in less than 5 minutes.

  • Stein

    I have been working with quite huge systems like national statistics, booking, international cargo systems, and what I have noticed is that even though the systems are quite large in total, a lot can be done to make several smaller pieces that work together. The code also gets cleaner, so its really a win-win situation I think.

    If a banking systems is chopped into smaller blocks, I think its quite possible to run a full set of tests within five minutes. Also it sounds like its too many integration tests.

  • John

    I am not sure I understand what you propose. By your suggestion I split my big banking system in 30 modules where each takes 5 minutes to build. My team is 10 developers who commit multiple changes. Some of the changes might be in the domain classes (thus affecting the whole system).

    When I make I release I would have to build these 30 modules in order (because of the dependencies) and still pay 30* 5 minutes = 2,5 hours.

    So what is that gain? I still need to wait for 2.5 hours after each commit to make sure that nothing is indeed broken (i.e. until all integration tests pass)

    (Also if you think that a banking system has too many integration tests, you obviously don’t know the inner workings of a bank)

  • Stein

    Well… I do suggest that dividing the system into 30 modules would be an option, yes. If you have to deploy all 30 modules every time you make a change, then perhaps they are put together wrong.

    From my experience, when systems are broken up in smaller modules with a good interface between them, then you only have to release a couple of those modules. For instance if one of the modules is front-end of the customer part of the banking system, where they can see their account information, you can make changes to that application and deploy it without touching the back-end part of the application, or releasing the administration part of the whole.

    That will effect the whole release process since you can run changes much faster through test and QA process, and can help the cause thowards continous delivery.

    And yes, I have been working with banking systems too, and large systems that gather whole nation’s transactional data. Politically it was one of the most challenging systems I have ever been working on – though not that hard to divide into working modules, but national people register were way more complex to split into working modules.

  • John

    I still don’t understand how it is possible to do what you say. Let’s say that I make a single change that affects modules 3,5 (I know this information because I am a human). Then another developer makes a changes that affects modules 5,7,4,23 (out of the 30 modules)

    How does the build server know that for the first commit only 2 (out of 30 modules) need to be built and for the second commit only 4 out of 30 modules need to be built?

    So at the moment my build server should need to build all 30 modules (not deploy them). Hence I have to wait 2,5 hours.

    I never said that all 30 modules need to be deployed every time. I am saying that they need to be compiled and unit tested. Also the article talks about Java, but my system could be on Groovy or another dynamic language (where I have to run unit tests to verify correct behaviour)

  • Stein

    A very interesting discussion. You will sometimes have exceptions to the rule, like the rule that a method should not be more than 20 lines. Sometimes its just not practical, so in some cases you can accept larger modules with longer build-time, though I still mean that if a module have longer than 3 minutes build-time, it should be checked and see if something can be done.

    The reason is that if you have 100 developers, building twice a day with 15 minutes build, you waste day’s worth of work every day, not to mention that developers gets frustrated.

    To answer your post:
    Most build-servers will notice when there’s a change to a module, and then it will be built.
    The rest of the library that is not touched or affected by the updated module, is – for that module, a third party product and does not need to be built. As long as there’s no compability issues or functionality you need in that module, I don’t see why you need to update straight away.
    Example: You don’t update StringUtil or Spring at once its a new version out, and you certainly don’t run all the unit tests built into Hibernate ;)

  • John

    Ok. then we agree.

    When I was talking about rebuilds, I was talking about runtime dependencies. Not compile time (which is indeed very easy for a build server to handle).

  • Ciro Anacleto

    Insightful. Great article. Thank you !

  • Mark

    10-20 minutes build? isn’t that too long…?

  • Jinesh Doshi

    Hi blader, would you able to share any link/tutorial on how this is done. Would really appreciate your help in this! Thanks!

  • Jinesh Doshi

    Hi Jevgeni, would you able to share any link/tutorial on how this is done. Would really appreciate your help in this! Thanks!

  • balder

    Now, 2 years later. I tend to propose a slightly different approach for what I described in the comment.
    Since then, we have docker!
    I currently don’t have a link yet. I haven’t actually written it down.

    In short: each application should live in it’s own reset-able docker image. You launch your ‘environment’ from the initial state you decided it should be in. ( docker images FTW! they are easily updated and saved, or just discarded after use and checked out/build from scratch again. ) Update only that applications that need to be changed, run all your system integration tests and deploy to production (if all goes well :-)