Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

How to Measure the Effects of Development + Operations improvements, an OpenSpace conversation

In Part 1 of this post, we covered What is DevOps and outlined the ideas behind a process and toolchain who’s purpose is to Deliver Functionality to End Users.

How can you track the effects of changes made to the development and operations side of your organization?

Whether DevOps related or not, it’s tough to get buy-in from the suits, team leads, or frankly, nearly anyone — when you only have fuzzy, subjective, “it just works better” commentary. At ZeroTurnaround, we’re small and fast, but I get the impression that at larger firms, engineers need to defend themselves from management intervention and somehow convince the higher-ups that spending some extra cash in their department will make real effects on the firm. I’m told that engineering teams (whether development or operations) are seen as “cost-centres”, whereas they’re often providing real value, that if properly communicated to management, could change the entire perception of their team / department / organization.

At DevOpsDays Boston 2011, I participated in an Open Space where a group of about 20 people spoke about metrics for tracking the effects of changes in your Dev / Ops environments – whether due to process, tooling, mentality, or other factors. We wrote down pretty much everything that we thought of, so here’s a short version of that list, with some commentary:

  • Mean Time Between Failure (MTBF)
  • Mean Time to Recover
  • Mean Time to Rollback
  • How do we measure – we released x days earlier…?
  • How many issues (changes/feature adds) got resolved in period x?  Deliveries / time
  • # deployments to prod : # incidents created
  • Tie a cost of incidents to a % of revenue (an example given was, if the cost of incidents is <3% of revenue, then they're doing a good job, and should focus more on quicker delivery) “size” of deployment
  • Cost of Deployment = time to deploy * people involved or waiting
  • Length of cycle time (# of features in a period of time) vs # of outages
  • Amount of dev time spent on features vs bug fixes
  • DevOps “Karma” measurements –
    ◦ measuring happiness of people – avg team member smileys / month   :) vs :(
    ◦ Ability to attract key people / retaining top talent
    ◦ Dilbert count – it was suggested that more dilbert comics are posted on cubicle walls = a worse environment to work in.  This was countered by the argument that a sarcastic environment can still be highly effective and enjoyable.
  • # of bugs that come out of a release
  • Visibility of app status (latency, # hits)
  • # of early catches – incidents that were prevented due to having the people
  • % “value” releases vs % “internal” releases (where an internal release = bug fixes or eliminating technical debt to helps ensure everything runs smoothly
  • “Ripple Effect” – # changes coming out of development that required an operational procedure or tooling change (dev –> ops), then measure the amount of time that developers spent helping operations people (ops –> dev)
  • Planned vs Unplanned Changes (visible ops guys say that the number of unplanned changes should be zero)… possible to measure # of system logons
  • Ability to value IT staff’s value (“I saved the company x $$”)
  • What’s the cost of moving a unit of value from development to the end user? Seen as a manufacturing process: when you know that value, you know whether it’s worth it to pay more to get it out faster

An idea that came from this open space was that: If you want to do something, there should be a metric associated with it. If there’s no metric, it doesn’t get done.

What are your experiences with using these / other metrics in your environment?


David BoothDavid Booth is the CEO of ZeroTurnaround (www.zeroturnaround.com), a software tooling company that focuses on productivity. David’s professional interests include Java, Agile, Marketing and community building. Meanwhile, when not collecting airline reward miles, he spends time training his Hungarian Vizsla, Mako. You can connect with David on LinkedIn.


  • Dguimbellot

    COnsider parsing incidents by defect categories? then assign a defect category or few to implementation/deployment?

    when we did our analysis we discovered that 27% of all incidents were deployment related.

    additionally we associated a cost of downtime associated with a deployment. there was no way to do a deployment without taking down a system. so we could do ROI correlation 

  • http://www.facebook.com/muralidharan.valuthur Muralidharan Valuthur

    How about measuring # of days between a feature dev complete and actual release ? These are the days where in the investment of development has been kept idle