Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

Survey Results: The Java EE Container Redeploy & Restart Report – measuring Turnaround Time

A couple of months ago we ran a survey asking a few questions about Java EE development, containers and redeploy times. Now that over 1100 people have responded it’s time to update the results. Since we’ve had more time to analyze them we also hope to provide a few insights into the data including a more detailed container breakdown.

If you’d like to play with the results on your own we‘ve provided all the data and our calculations in a handy Excel sheet that you can download here. As a note, if you haven’t answered the 3-question survey yet, take two minutes and go for it. As more answers trickle in we’ll update this post with the new data.

The first question in the survey was “What container do you use on your largest current project?”. The breakdown follows:

Chart 1: Which Container is used most often?

chart1

We didn’t include the containers that scored less than 10 answers in the survey. These included:

  • (2) Adobe JRun
  • (1) Geronimo
  • (1) Iona
  • (7) TmaxSoft JEUS
  • (5) JONAS
  • (1) Pramati Server
  • (4) SAP NetWeaver
  • (2) Sybase EAServer

Unsurprisingly Apache Tomcat holds the lead, closely followed by JBoss. Together the OSS servers total over 70% of the responses. Although I wouldn’t go as far as to translate these numbers directly into market share, we did find studies that found similar results, courtesy of SD Times – Slight differences though: we asked which ONE container you use for your largest projects, and theirs appear to allow people to choose multiple containers.

The 2nd question we asked was, “How long does it take to restart your container and redeploy your app?”. The breakdown follows (the horizontal axis is in minutes):

Chart 2: “How long does it take to restart your container and redeploy your app?”

chart2

We previously assumed that the redeploy/restart phase lasted approximately 1 minute.  Though 38% of respondents agreed or said it was quicker than 1 minute, 62% suggested that this process lasts around 2 or more minutes.  The calculated average is approximately 2.5 minutes –indicating that we underestimated the amount of time spent on the redeploy phase.

Moving along, we asked, “In an hour of coding, how many times do you redeploy?”. The breakdown follows:

Chart 3: “How Many Times do you Redeploy in an Hour of Coding?”

chart3

The distribution seems to be close to normal, excepting the jump in the end. The average is slightly over five times an hour, which coincides with the measurements we did before.

For those who answered, “I never redeploy”, we asked some of them how they did this. Responses were:

  • “I’m not the guy that does the redeploys”
  • “We develop on embedded jetty & activemq & atomikos in debug mode instead of nearly unusable target OracleAS. So of course we need redeploy or restart jetty as usual, but not the OAS.”
  • “I’m in the very early stages of a project so a lot of out time is spent coding and testing without redeploying – typically I redeploy 3-4 times per hour”
  • “We use JavaRebel and it’s awesome” (this obviously came from someone trying to charm our development team – who are still blushing. Note – it’s now called JRebel.)

Next we did some data crunching. We assigned numeric values to each of the intervals (e.g. “3.5″ for the “3-4″ interval) and multiplied the number of redeploys an hour by the amount of time one redeploy takes (basically, Chart 2 times Chart 3), thus finding the approximate amount of time respondents spend redeploying in each hour of development. We broke down the data into five minute intervals:

Chart 4: How Much Time Does a Java Developer Spend Redeploying in an Hour of Coding? (using raw data before improving accuracy)

chart4

The average is about twelve and a half minutes per hour, accounting for more than 20% of total development time. However, the standard deviation is over 14, which means that the actual per cent varies greatly. We wanted to show more accurate data, so by looking closely and grouping results, we found this:

  • Less than 5 minutes an hour. Looking into this group we see that it is slightly skewed by the respondents who indicated that they never redeploy. We asked the people who “never redeploy” how they do it (see comments above), and they indicated that their responses were often not effective for tracking the actual redeploy time that we are trying to measure. Therefore, it makes sense to remove that data from the chart.
  • 5 to 15 minutes an hour. 45% of respondents fall into this category.
  • 20 to 25 minutes an hour.
  • Over 30 minutes an hour. Looking closer at the data we see that some of the respondents indicated that they redeploy over 60 minutes an hour (choosing both “More than 10 redeploys per hour” and “More than 5 minutes per redeploy” for Charts 2 and 3). This is quite impossible.  In fact, we believe any number over 40 minutes per hour seems to be against reason. It makes sense to remove all this data from the chart too.

The chart with the updated data looks as follows:

Chart 5: How Much Time Does a Java Developer Spend Redeploying in an Hour of Coding? (more accurate data)

chart5

The average is now ten and a half minutes per hour with a standard deviation of 8, which is more trustworthy. This translates to about 17.5% of total development time, which is considerably more than we expected. We will use this “cleaned up data” for the rest of the post.

Later we’ll describe how to calculate this over a year, but here are some quick figures:

  • 5 mins per coding hour becomes 6000 minutes annually (2.5, 40-hour workweeks)
  • 9 mins per coding hour becomes 10800 minutes annually (4.5 weeks)
  • 14 mins per coding hour becomes 16800 mins annually (7 weeks)
  • 21% of respondents report spending more than 14 mins per hour on redeploys
  • 71% of respondents report spending more than 2.5 full weeks per year, redeploying.

After looking at this issue in general, we investigated the data on a per-container basis.  This is what we found:

Chart 6: Organized By Container, How Much Time is Spent Redeploying?

chart6

There are few surprises here. Jetty is leading the pack with only 5.8 minutes per hour spent redeploying and IBM WebSphere is trailing with more than twice more — 13.8 minutes per hour. One thing to note is that although Jetty startup is undoubtedly faster than IBM WebSphere it is likely that most of the difference is due to the size of the applications deployed and the technologies used in them.

Next we have the same chart, but numbers displayed as a percent of development time (time spent per hour divided by 60).  Perhaps it’s useful to describe how we define “Development Time”.  For us, Development time is what you spend actually writing code, with an IDE, running your server and the Java language. This doesn’t include meetings, gathering requirements, smoking or any other activities that don’t involve a compiler. We’ve asked around, and it sounds like the industry assumes that 5 hours per day are spent coding.

Chart 7: Organized by Container, What Percent of Development Time is Spent Redeploying?

chart7

To put those numbers in perspective we can also calculate the amount of time spent redeploying a year. To do that, we’ll make some assumptions:

  1. We assume 48 work weeks  per year.  If you’re lucky enough to get more than 4 weeks of vacation, don’t rub it in.
  2. We assume that 5 hours per day is spent doing development. The other 3 hours are reserved for non-development activities.

The following chart displays the number of 40-hour work weeks a year spent redeploying:

Chart 8: How many 40-hour work weeks are spent on the redeploy phase, over a year?

chart8

Depending on the container, we’re looking at 3 to 7 work weeks per year, spent exclusively on the redeploy phase. The average over all of the data is slightly over 5 work weeks a year, but the standard deviation of 4 makes the range of 3-7 more reliable.

Our final graph shows the amount of time spent redeploying per container in more detail. Instead of using averages, we break down the proportion of respondents that redeploy into groups: less than 5 minutes an hour, 5 to 14 minutes and hour, 15 to 29 minutes and hour and over 30 minutes an hour.  We expect that this may show the size of projects that use different containers, and potentially shed light on the time your project spends redeploying.

Chart 9: Java EE container market penetration.

chart9

We interpret this chart as:

  • Jetty is only used in projects that redeploy quickly. This makes all kind of sense, considering that Jetty doesn’t even support redeployment and instead has extremely fast container startup.
  • Apache Tomcat and GlassFish are used in same types of project. Both are posed as fully functional yet lightweight alternatives to the classic heavy application servers. Although Tomcat is much more popular today, GlassFish is growing in popularity in the same market share.
  • JBoss, Oracle Weblogic and IBM WebSphere compete for pretty much the same market segment. The majority of their projects are large and complex, and the redeploy times reflect that.
  • Another interesting fact is that the proportion of projects in the 5-14 group is pretty stable for all containers. This likely corresponds to medium-sized projects that are not actively contested by either lightweight or classical application servers. The only exception is Jetty, which is heavily skewed to the lightweight side.

Coming up next: We’re taking a look at incremental build times and comparing Ant, Maven, Eclipse, IntelliJ, and NetBeans.  If you’re interested, take a minute to participate here. Thanks!

  • Henry

    I wish you guys could also solve my “deploys” in Android development….then my (programming) life would be set.

  • Henry

    I wish you guys could also solve my “deploys” in Android development….then my (programming) life would be set.

  • Toomas Römer

    @Henry What are the redeploy times for Android emulator?

  • Toomas Römer

    @Henry What are the redeploy times for Android emulator?

  • http://www.webtide.com/ Jan Bartel

    As a Jetty committer it’s nice to see our efforts to keep jetty small, fast and flexible are paying off!

    One thing I have to correct though is that Jetty does indeed support redeployment. In fact we support it 4 ways: in the standalone container via our ContextDeployer, in the maven jetty plugin, via jmx using the WebAppContext mbean and programatically when running jetty embedded.

    cheers
    Jan

  • http://www.webtide.com Jan Bartel

    As a Jetty committer it’s nice to see our efforts to keep jetty small, fast and flexible are paying off!

    One thing I have to correct though is that Jetty does indeed support redeployment. In fact we support it 4 ways: in the standalone container via our ContextDeployer, in the maven jetty plugin, via jmx using the WebAppContext mbean and programatically when running jetty embedded.

    cheers
    Jan

  • Lars Clausen

    The comparison of containers based on how much time developers spend on redeploying them is rather misleading — that assumes that the frequency of redeployment is independent of container. Here’s a breakdown of average deployment time for the deployment systems. Caveats about project size still applies, of course – would have been nice to know the number of classes/number of beans/other size measurement.

    Server Redeploy average
    Jetty 1,153
    Apache Tomcat 1,669
    Caucho Resin 1,735
    GlassFish 1,900
    JBoss 2,705
    Oracle OC4J 2,924
    Oracle Weblogic 3,606
    IBM WebSphere 3,845

  • Lars Clausen

    The comparison of containers based on how much time developers spend on redeploying them is rather misleading — that assumes that the frequency of redeployment is independent of container. Here’s a breakdown of average deployment time for the deployment systems. Caveats about project size still applies, of course – would have been nice to know the number of classes/number of beans/other size measurement.

    Server Redeploy average
    Jetty 1,153
    Apache Tomcat 1,669
    Caucho Resin 1,735
    GlassFish 1,900
    JBoss 2,705
    Oracle OC4J 2,924
    Oracle Weblogic 3,606
    IBM WebSphere 3,845

  • http://www.zeroturnaround.com/ Toomas Römer

    @Jan

    Thanks for the correction and the speedy container!

  • http://www.zeroturnaround.com Toomas Römer

    @Jan

    Thanks for the correction and the speedy container!

  • http://www.ekabanov.net/ Jevgeni Kabanov

    @Lars

    Not sure if there’s a big difference in the lineup. Plus why would frequency of redeployment depend on container?

  • http://www.ekabanov.net Jevgeni Kabanov

    @Lars

    Not sure if there’s a big difference in the lineup. Plus why would frequency of redeployment depend on container?

  • http://www.searchfull.net/ searchfull

    If your compile-deploy cycle lasts more than 10 seconds, you’re doing it very wrong. Make your war smaller and more modular, use a IDE plugin for deployment, use lazy initializations, and separately-deployed resource pools.

  • http://www.searchfull.net searchfull

    If your compile-deploy cycle lasts more than 10 seconds, you’re doing it very wrong. Make your war smaller and more modular, use a IDE plugin for deployment, use lazy initializations, and separately-deployed resource pools.

  • George

    So those who choose Weblogic and Websphere not only choose to pay Oracle/IBM instead of get free software, they choose to waste developers’ time as well. The good old fashioned “choosing IBM won’t get you fired” turns out to be irresponsible incompetent choice. These companies deserve to suffer…..

  • George

    So those who choose Weblogic and Websphere not only choose to pay Oracle/IBM instead of get free software, they choose to waste developers’ time as well. The good old fashioned “choosing IBM won’t get you fired” turns out to be irresponsible incompetent choice. These companies deserve to suffer…..

  • Denis Robert

    @searchfull: Ever work with Websphere? 10 seconds doesn’t even get you past the first steps in deployment. Deploying on websphere has sometimes taken me up to half an hour, for something I would consider a smallish app. Even with perfect optimization, you’d be lucky to get your deploy time under 5 minutes.

    Not to mention the number of times a clean restart is required, even though IBM would like you to believe it never is…

  • Denis Robert

    @searchfull: Ever work with Websphere? 10 seconds doesn’t even get you past the first steps in deployment. Deploying on websphere has sometimes taken me up to half an hour, for something I would consider a smallish app. Even with perfect optimization, you’d be lucky to get your deploy time under 5 minutes.

    Not to mention the number of times a clean restart is required, even though IBM would like you to believe it never is…

  • melbeltagy

    We’re using Websphere and we don’t redeploy at all. We run in debug mode, using RAD from within our workspace.
    Whatever changes happens are deployed automatically by WAS for Java classes and JSPs. Even for EJBs.
    We only have to restart the server if we change an EJB’s method’s signature, which is rarely.
    Restart time takes about a minute or two, max. Is that an issue? Of course not.

  • melbeltagy

    We’re using Websphere and we don’t redeploy at all. We run in debug mode, using RAD from within our workspace.
    Whatever changes happens are deployed automatically by WAS for Java classes and JSPs. Even for EJBs.
    We only have to restart the server if we change an EJB’s method’s signature, which is rarely.
    Restart time takes about a minute or two, max. Is that an issue? Of course not.

  • http://www.modularhomesnetwork.com/ modular homes

    Last year we ran two surveys asking developers how much time they spend … Given that, the averages should apply pretty nicely to Java EE development!

  • Mike

    Products from IBM / Oracle offer things like “support contracts” which can kind of come in handy during crisis situations for web applications that are actually business critical, guy.

  • AB

    You’ve never had to write truly enterprise-scale applications. When you do, you’ll find that WL and WS scale exceptionally well and the freebies are lacking. For SMB scenarios, though, you can generally get by with freebies.