Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

The Build Tool Report: Turnaround Times using Ant, Maven, Eclipse, IntelliJ, and NetBeans

Some time ago we ran a survey asking a few questions about the build process, specifically the tools that are used to do incremental builds and how much time those builds take. We had over 600 responses, so now it’s time to count the results.

This is the first time that we’ve published results on the incremental build process, so the information is more likely to serve as a guide than an authoritative information source. That being said, the information is still quite interesting, and if it serves to start a conversation that improves the process of even one team, then we’re proud to have helped out. If you haven’t answered the 3-question survey yet, take two minutes and go for it – and do let your community know about it – as more answers trickle in we’ll update this post with the new data. If you’d like to play with the results on your own we‘ve provided all the data and our calculations in a handy Excel sheet that you can download here.


The first question in the survey was “What build tool do you use for incremental builds on your largest current project?” The breakdown follows:

Chart 1: “Which build tool is used most often for incremental builds?”

chart1
This does not include tools that scored less than 10 answers in the survey. Those are:

  • Buildr
  • Shell scripts
  • javac
  • Make
  • NAnt
  • Savant
  • Hudson
  • PHP

Maven and Ant are responsible for over half of the incremental builds in the wild. It also seems that Maven has overtaken Ant by popularity, although you should take it with a grain of salt as the questions were not phrased for this purpose. Unfortunately I couldn’t find any external data to confirm or contradict these results. Please do let us know if you know of other surveys on this topic.

The other half of incremental builds are done inside IDEs. Although some of those are just driving Ant or Maven inside the IDE, there is also a large number of developers that use the IDE as the primary build tool during development (we’ll analyze this in detail in the end of the report).

It might be valuable to reiterate here that this is a poll where respondents are self-selecting, and therefore it may not accurately reflect the actual marketplace when it comes to determining market share.  Since we’re more interested in the incremental build process itself, that’s something we can live with.  That being said, there is a chance that people who have fast (or slow) builds may be more likely to complete a poll like this.  Since we’re not sure if this is serious enough to sway any results, we’ll just display the data, and let you decide.  So, while 53% of developers are using Ant or Maven for their incremental builds, everyone else uses their IDEs: Eclipse is dominating the IDE landscape with 32% , followed by IntelliJ IDEA at 10% and NetBeans at 5%.

With that in mind, we asked, “How long does an incremental build take?”

Chart 2: “How long does an incremental build take?”

chart2

This is relatively good news. Nearly half of our respondents (44%) indicated that their incremental build process takes less than 30 seconds. 40% of respondents have incremental builds lasting from 1 to 3 minutes. Only 16% of incremental builds last over 4 minutes.

The average length of a build is 1.9 minutes with the standard deviation of 2.8.

To finish up, we asked, “In an hour of coding, how many times do you run an incremental build?”

Chart 3: “How often do Java developers run incremental builds?”

chart3

A healthy 31% of respondents don’t have to run the build at all (e.g. it’s run automatically on save). The rest of the numbers are all over the place, so we’ll wait with the analysis until we can put them in context.

The average number of incremental builds an hour is 3.9 times, with a standard deviation of 4.1.

It’s time to crunch some data. We assigned numeric values to each of the intervals (e.g. “2.5″ for the “2-3″ interval) and multiplied the number of incremental builds an hour by the amount of time one incremental build takes (basically, Chart 2 times Chart 3), thus finding the approximate amount of time respondents spend building in each hour of development. This was done per respondent, so if someone said that their build takes 4-5 minutes and they build twice an hour, we’d see a result of 9 minutes per hour. We broke down the data into the following intervals:

Chart 4: “Time spent on incremental builds during an hour of coding”

chart4

The average total time taken by incremental builds in an hour is exactly 6 minutes, but the standard deviation is 10.1, rendering this number unreliable. We can, however, divide the respondents in three quite well defined groups:

  • Less than 1 minute an hour. 34% of respondents basically don’t spend any time on the incremental build process.
  • 1 to 5 minutes an hour. 34% of respondents spend a “reasonable” amount of time on incremental builds – under 5 minutes an hour. This group spends an average of 3 minutes an hour on incremental builds, which corresponds to about 5% of total development time.
  • Over 5 minutes an hour. 32% of respondents spend over 5 minutes an hour on incremental builds. The weighted average in this group is over 13 minutes an hour. This group of developers is spending about 22% of their development time on incremental builds.

Chart 5: “Time spent on incremental builds, per hour, by build tool”

chart5

It is clear from this chart that Ant and Maven take significantly more time than IDE builds. Both take about 8 minutes an hour, which corresponds to 13% of total development time. There seems to be little difference between the two, perhaps because the projects where you have to use Ant or Maven for incremental builds are large and complex.

Eclipse is definitely the fastest with 2.9 minutes an hour, which corresponds to about 5% of total development time. Eclipse is the only IDE supporting true incremental build on save using a fast embedded compiler, so these results are expected. It is likely that the true number is even lower, as some of the respondents use Eclipse to launch the Ant or Maven builds (we’ll return to this with the next chart).

IntelliJ IDEA falls in between with 5.7 minutes an hour and about 10% of total development time. It does not have true background compilation and often the IDE builds just launch Ant or Maven behind the scenes.

Chart 6: “Incremental build length breakdown per build tool”

chart6

This final chart shows the proportion of respondents in one of the three groups we defined with Chart 4 broken down per build tool.

Finally some things become clear. About 61% of Eclipse builds happen instantly (taking less than 1 minute per hour). We can assume that those are the respondents using compile-on-save, whereas the rest use Eclipse to launch Ant or Maven builds. Considering that 32% of respondents are using Eclipse (see Chart 1), this means that about 20% of all of the respondents use Eclipse with compile-on-save and thus benefit from the instant incremental builds.

The breakdowns of Ant and Maven times are quite similar, with Maven being slightly slower. Moreover the breakdown for IntelliJ IDEA is also similar, supporting the hypotheses that IntelliJ IDEA builds launch Ant or Maven in the background. The proportion itself likely corresponds to the size and complexity of projects, with smaller ones building quickly and larger ones taking build time of over 5 minutes and hour.

It is hard to draw any simple conclusions from this survey, as different groups have different problems, so instead let’s put together the most interesting and reliable numbers we got:

  • 56% of respondents have builds that last over half a minute, each.
  • 20% of respondents use Eclipse with compile-on-save and thus benefit from the instant incremental builds.
  • 34% of respondents spend an average of 3 minutes an hour on incremental builds, which corresponds to about 5% of total development time, or 1.5 weeks per year (40-hour weeks).
  • 32% of respondents spend more than 5 minutes per hour on incremental builds.  Of the developers spending more than 5 mins per hour, 13 minutes per hour is the average amount of time spent.  13 mins per hour equals 22% of total development time, or 6.5 weeks per year (40 hour workweeks).
  • Weeks spent on incremental builds are calculated by assuming 48 work weeks per year (minus vacation) and an average of 5 hours of development time per day.
  • Java developers spend 1.5 to 6.5 work weeks a year (with an average of 3.8 work weeks,  or 152 hours, annually) waiting for builds, unless they are using Eclipse with compile-on-save.
  • http://www.squins.com/ Kees van dieren

    Its easy to fasten maven builds using the maven cli plugin.

    Read all about it here: http://www.mojavelinux.com/blog/archives/2009/05/a_gamechanging_maven_2_plugin_you_absolutely_must_use/index.php

  • http://www.squins.com Kees van dieren

    Its easy to fasten maven builds using the maven cli plugin.

    Read all about it here: http://www.mojavelinux.com/blog/archives/2009/05/a_gamechanging_maven_2_plugin_you_absolutely_must_use/index.php

  • Marc Stock

    Your information regarding IntelliJ is incorrect (or at least not up-to-date). IntelliJ supports incremental compilation just like Eclipse does. There’s a compiler setting where you can tell it whether to use it or not. See this FAQ for more info:http://www.jetbrains.com/idea/documentation/migration_faq.html

  • Marc Stock

    Your information regarding IntelliJ is incorrect (or at least not up-to-date). IntelliJ supports incremental compilation just like Eclipse does. There’s a compiler setting where you can tell it whether to use it or not. See this FAQ for more info:http://www.jetbrains.com/idea/documentation/migration_faq.html

  • magnus

    Netbeans uses Ant to build projects – So how do these 5% of people use netbeans to build projects without being part of the 21% using Ant??

  • magnus

    Netbeans uses Ant to build projects – So how do these 5% of people use netbeans to build projects without being part of the 21% using Ant??

  • Toomas Römer

    @magnus They were counted separately. We made a distinction between build tools and IDE tools. For Eclipse we also cannot say with 100% accuracy that JDT is used, maybe the users orchestrate ANT or Maven from it but report it as Eclipse.

  • Toomas Römer

    @magnus They were counted separately. We made a distinction between build tools and IDE tools. For Eclipse we also cannot say with 100% accuracy that JDT is used, maybe the users orchestrate ANT or Maven from it but report it as Eclipse.

  • http://www.netbeans.org/ Tori

    The NetBeans IDE has had compile-on-save since NetBeans 6.5.
    http://www.netbeans.org/community/releases/65/

  • http://www.netbeans.org Tori

    The NetBeans IDE has had compile-on-save since NetBeans 6.5.
    http://www.netbeans.org/community/releases/65/

  • Martin Dürrmeier

    While this report seems highly interesting I wonder if there are any related reports at all.
    Can’t belive there are none!
    I’m currently writing my Bachelor Thesis about how to optimize the compile-debug-deploy cycle and did not find a single one (besides this side)

  • Martin Dürrmeier

    While this report seems highly interesting I wonder if there are any related reports at all.
    Can’t belive there are none!
    I’m currently writing my Bachelor Thesis about how to optimize the compile-debug-deploy cycle and did not find a single one (besides this side)

  • Martin Dürrmeier

    Regarding the IDE as a “build tool”
    The F5 Key is not a build process:
    http://www.codinghorror.com/blog/archives/000988.html

  • Martin Dürrmeier

    Regarding the IDE as a “build tool”
    The F5 Key is not a build process:
    http://www.codinghorror.com/blog/archives/000988.html

  • Franz See

    When comparing ant against maven, the sizes of the projects are usually different. Ant projects tend to be huge while Maven projects tend to be small.

    Building your single super ant project is slower than building a single maven project/module. However, building all maven projects/modules is slower to building that single super ant project.

  • Franz See

    When comparing ant against maven, the sizes of the projects are usually different. Ant projects tend to be huge while Maven projects tend to be small.

    Building your single super ant project is slower than building a single maven project/module. However, building all maven projects/modules is slower to building that single super ant project.

  • http://pentiumpanties.blogspot.com/ Kush

    This is a very useful study for devlopers who have yet not decided which build tool to use. Since eclipse is more than a build tool and maven and ant can both be integrated into it. it is a clear choice

  • http://pentiumpanties.blogspot.com Kush

    This is a very useful study for devlopers who have yet not decided which build tool to use. Since eclipse is more than a build tool and maven and ant can both be integrated into it. it is a clear choice

  • Wes

    The survey doesn’t take into account Maven is not just a build tool but a project management tool. All other configurations require more work to manage project dependencies. I can count the hours engineers have spent looking for a jar file or trying to figure out what the last guy was thinking when they laid out the project. These are the real time eaters in projects

  • Wes

    The survey doesn’t take into account Maven is not just a build tool but a project management tool. All other configurations require more work to manage project dependencies. I can count the hours engineers have spent looking for a jar file or trying to figure out what the last guy was thinking when they laid out the project. These are the real time eaters in projects

  • Niko Mahle

    Hi, it would be very interesting how much percentage of the maven users use still Maven 1 ?

  • Niko Mahle

    Hi, it would be very interesting how much percentage of the maven users use still Maven 1 ?

  • what_are_you_saying

    “The average total time taken by incremental builds in an hour is exactly 6 minutes, but the standard deviation is 10.1, rendering this number unreliable. We can, however, divide the respondents in three quite well defined groups:”

    You must be joking. An incremental compiler, such as the one in the Eclipse IDE compiles in the background in response to a code modification. Rarely is a complete rebuild necessary, that is, unless you have filled your IDE with lots of crap like Maven, which this website seems to be trying to promote.

    Ant is by definition not “incremental”. Why don’t you go back to school, or learn VB?

  • what_are_you_saying

    “The average total time taken by incremental builds in an hour is exactly 6 minutes, but the standard deviation is 10.1, rendering this number unreliable. We can, however, divide the respondents in three quite well defined groups:”

    You must be joking. An incremental compiler, such as the one in the Eclipse IDE compiles in the background in response to a code modification. Rarely is a complete rebuild necessary, that is, unless you have filled your IDE with lots of crap like Maven, which this website seems to be trying to promote.

    Ant is by definition not “incremental”. Why don’t you go back to school, or learn VB?

  • what_are_you_saying

    Ant is frequently cited as ** breaking ** the incremental compiler. Are you people out of your mind? Or just severely “challenged”?

  • what_are_you_saying

    Ant is frequently cited as ** breaking ** the incremental compiler. Are you people out of your mind? Or just severely “challenged”?

  • http://www.zeroturnaround.com/ Toomas Römer

    @what_are_you_saying

    You have wrong level of abstraction here. By incremental builds we mean builds done in development for the developer to see results and not clean full builds.

    You’re splitting hairs by saying that the true incremental builds are Eclipse JDT delta compilations vs ant/maven javac incremental builds. Invoke ant on a source tree, change one file and reinvoke. Only one new file will be compiled (unless the referenced .class files also need compilation).

    Ant is said to break the the “incremental” (or the IDE) compiler because once you use command line compilation + IDE compilation you get into trouble. JDT and javac compile enums for example differently, anonymous inner classes can have different numbering scheme, compilation errors can be compiled into runtime exceptions vs not compiling at all. This is the sad story of using 2 compilers and one output path. You run into problems.

    We don’t advocate any specific build tool. We like the shortest turnaround possible and the best tools for the specific job. For this survey we tried to cover as many tools as possible.

  • http://www.zeroturnaround.com Toomas Römer

    @what_are_you_saying

    You have wrong level of abstraction here. By incremental builds we mean builds done in development for the developer to see results and not clean full builds.

    You’re splitting hairs by saying that the true incremental builds are Eclipse JDT delta compilations vs ant/maven javac incremental builds. Invoke ant on a source tree, change one file and reinvoke. Only one new file will be compiled (unless the referenced .class files also need compilation).

    Ant is said to break the the “incremental” (or the IDE) compiler because once you use command line compilation + IDE compilation you get into trouble. JDT and javac compile enums for example differently, anonymous inner classes can have different numbering scheme, compilation errors can be compiled into runtime exceptions vs not compiling at all. This is the sad story of using 2 compilers and one output path. You run into problems.

    We don’t advocate any specific build tool. We like the shortest turnaround possible and the best tools for the specific job. For this survey we tried to cover as many tools as possible.

  • Angelo Schneider

    Sorry, but I strongly doubt anyone using Ant or Maven is doing an “incremental build” with it. They all do full builds, hence the long build times.

  • Kingwincelas2

    Maven sucks HUGE, SMELLY HORSE BALLS!!!!!

  • heena goyal

    Hey,

    Nice piece of information provided . I wanted to ask a question

    i am from Electronics background and the company i m placed in is an IT company as for now i have no other option i have to join this company only but the main problem is this that they work on java and i don’t know anything about java. I want to learn java online so saw many videos and even searched and came across http://www.wiziq.com/course/12145-the-6-week-complete-java-primer-with-training-certificate but i dont know how it is going to work for me. If anybody has taken this course please let me know..