Imagine a bacon-wrapped Ferrari. Still not better than our free technical reports.
See all our reports

If and when you should use Test-Driven Development

Have you ever argued about the efficiency of test-driven development in your day job? I certainly have, and on both sides of the argument. I have worked in the software development sector for 9 years as a developer and an architect on seven different projects ranging from mobile apps to a custom made telecom self services. I’m fed up of the pointless arguments – the ones based on solo experiences and anecdotal evidence. In this article I try to remedy this situation and clarify TDD in an objective way. Arguments should be based on a common understanding of what TDD is and your current project situation at hand: people involved, technology, goal, deadline, etc.

In this article we’ll discuss the following topics:

  • Benefits of TDD in short
  • Graphical User Interface and Test-Driven Development
  • Against Test-Driven Development
  • Why TDD isn’t practiced more in real life?

A short list of the benefits TDD offers

Test-driven development is increasingly widespread and there is good empirical evidence that it’s a beneficial practice. TDD reduces the number of bugs in production and improves code quality. In other words it makes code easier to maintain and understand. Also, it provides automated tests for regression testing. The evidence about TDD from the industry:


Although, it doesn’t mean that TDD should be used by everyone everywhere in every situation. This will be discussed further in chapter “Against Test-Driven Development” of this post.

Moreover, it is not advised to develop a graphical user interface (GUI) in a test-driven way. One could look at this topic in more depth on StackOverflow, like in this question about Does TDD apply well when developing an UI? or the academic community. Both agree that developing GUIs in a test-driven way is not straightforward. According to the aforementioned sources, automating GUI tests during the development phase costs more than manual testing. It’s even costlier if one tries to automate in a TDD way because GUI test automation tools are focused on existing systems. It would be pragmatic to automate GUI regression tests after the development phase is finished. In general, GUI test automation is made costly by these aspects:

  • Complexity – the number of possible user interactions from different application states, grows exponentially as the GUI functionality increases.
  • Verification – as the complexity increases it becomes harder to create automated test scripts that cover a sufficient number of cases.
  • Change – GUIs change very often during development, so test script maintenance becomes a burden.

This brings us to the next chapter, how to incorporate TDD and GUI development.

Graphical User Interfaces and Test-Driven Development

If developing a GUI in a test-driven way is not beneficial, then how should one use TDD with an application that have a GUI? One of the leading proponents of TDD, Kent Beck, suggests to create a thin view layer together with underlying model and develop application in a TDD way from that model instead. Some software patterns have spawned from this idea:

The goal of these patterns is to make the view layer as dumb as possible, so testing it isn’t required. All of these patterns work similarly: the view will have an underlying model that has fields for each view element or state. The view will also delegate all the actions to the model. If an update to the view is needed, the model will request it. This allows developing the application from that model using TDD.

This approach covers most of the code with automated tests, although, doubts remain if the actual view is integrated with the underlying model correctly. To overcome this uncertainty, a few tests are recommended that invoke the GUI, for example through the browser for web applications, and verify that the connections with the underlying model are correct. The fact that there are only a few tests that invoke the GUI means that maintaining those are easy. Moreover, because there are only a few GUI tests, they don’t slow down the application test cycle. These tests don’t have to be written in a TDD fashion of course.

Arguments against Test-Driven Development

Test-driven development can backfire when the environment is not suitable or it is used incorrectly. One should consider these topics when planning to use TDD:

  • Cost of implementing functionality
  • Test code requires maintenance as well as production code
  • Too much TDD makes code more complicated than necessary

Let’s look at them in more details.

Cost of implementing functionality

The main drawback is that it takes significantly more time to implement functionality in a test-driven way than in conventional manual testing way. The cost depends on a couple of things. Consider for example these issues:

  • Experience with TDD among developers – less experience means more cost.
  • Existing code support for automated testing – trying to automate testing on a framework not meant for that, will probably cost more than the benefit it provides.
  • Organizational support for TDD – It might be easy to convince your department to start using TDD, but if your project depends from another department you might not reap all the benefits of TDD. For example, if you need a running copy of their component with all the test data for running integration tests, but they are not willing to provide it to you, then you simply cannot run integration tests. Organizational support also means resources for setting up an initial TDD scaffold.

When creating a proof of concept it is usually faster and cheaper to use manual testing. Furthermore, when time to market is important, then it’s worth considering creating a quick version without automated tests. It is a slippery road, of course, because future changes will be more error prone, thus harder and slower to implement because of a lack of automated regression tests.

Test code requires maintenance as well as production code

The second problem of TDD is that test code grows linearly with production code. All lines of code require maintenance, which means cost. The cost is easiest to acknowledge when a change to existing functionality is made. The tests have to be modified as well as production code. This becomes a problem when multiple tests execute the same line of production code. This usually happens when there are too many high-level tests. The highest level test is an end-to-end test, which starts by invoking the GUI and goes all the way down to the data storage level. But it’s not only the highest-level test that can cause problems.

Ideal software testing pyramid

Tests that execute a whole component, for example a web service, are quite high-level too, compared to a unit test. The ice cream cone anti-pattern is a nice illustration of TDD gone wrong. An ice cream cone emerges when you invert the ideal test automation pyramid. You can make ice cream by creating more integration tests than unit tests and more GUI tests than integration tests and even more manual regression tests than your automated GUI tests.

Software Testing Ice-Cream Cone Anti-Pattern

Too much TDD makes code more complicated than necessary

A third problem emerges when one applies too much TDD – so much so that the tests will make the code more complex than necessary. There are good examples, albeit from the Ruby world of how this can happen. Here are main points from that article:

  • It’s wrong to decouple the application from the Rails framework, so that tests could run faster and code could be easier to mock. This looks good from the testing perspective, but from the production code perspective the code gets more complex. It contradicts one of the TDD goals – to design better production code.
  • Unit testing MVC controllers doesn’t make sense. These should be integration tested instead.
  • Don’t try to eliminate databases from model tests. Booting the database takes only 1.2 seconds and you can use transactional text fixtures to generate different scenarios.
  • “The answer to how can I make it better, is how can I make it clearer, not how can I test it faster or more isolated” — D.H.Hansson.

Why TDD isn’t practiced more in real life?

In short, it’s impossible to show connection with TDD and improved productivity. That’s because measuring software creation productivity is impossible. Here’s a nice overview of failed attempts and their reasons. They propose a solution, but in my opinion it is often way too complicated to work in real life.

The longer version is that there is a conflict between the long term and short term goals for software creation teams. In the short term the fastest way is to develop functionality, test it manually, fix the bugs, test again and so on, until the functionality is ready. There are no arguments that TDD would be faster in the short term. However, in the long term some problems arise with the manual testing method:

  • Time for manual regression testing increases as the project grows, which makes release cycles longer.
  • One could freeze regression testing time in order to freeze release cycle time, but then more and more bugs go to production.
  • Code quality suffers which makes adding new functionality slower as the project grows.

Even though the benefits are clear in the long term, these benefits don’t run companies – deadlines do. When an upcoming hard deadline approaches, the development team will be focused on the short term. At that time the inexperienced team members hope to fix things in the future, experienced members have already abandoned that hope.

Conclusion

In this post we’ve discussed how TDD is becoming more and more widespread for the following reasons:

  • reducing the number of bugs in production
  • improving code quality
  • providing you with the automated tests for regression testing

However, before choosing to use TDD in an existing project, you have to consider the unique context of your project in detail. Ask yourself these questions and then decide:

  • How much will TDD increase the cost of implementing your functionality?
  • What kind and how many automated tests do you plan to create?
  • Will TDD be used correctly? Will TDD simplify your production code or make it more complex?

After all, here’s a short rule of thumb: A GUI shouldn’t be built with TDD. A thin view layer between a GUI and application logic should be created instead. If automated regression tests are needed, GUI test automation tools can be used after development.

What’s your take on TDD? Share it in the comments below and let’s put our heads together to figure out the best approach.

PS. I’d love to thank my friend Allar Tammik for helping me a lot with reviewing the article!

  • Kevin Wright

    “A GUI shouldn’t be built with TDD” – I disagree. Development of a GUI very much *should* be driven by tests, though the “tests” in this case would be end-user feedback.

    Unit tests are not the only type of test…

  • frhd

    My thoughts exactly. Depending on the kind of GUI-type app, they even come first.

  • Joosep Simm

    User feedback is useful of course, but how do you plan to automate it?
    And how would you collect the feedback before implementation?

  • Kevin Wright

    1. Just as all tests are not unit tests, it’s also true that not all tests are automated tests.
    2. Feedback of this nature is best handled through mockups.

  • Joosep Simm

    I think we talk about different things then. TDD is about automated tests. Sure, the philosophy to “design a test before action” can be applied elsewhere too. But I wouldn’t call UX testing on mockups TDD practice.

  • David Leppik

    The fundamental issue is that UI requirements are psychological, not mathematical. Automated tests work when there are well-defined (i.e. quantifiable) requirements. Either it works or it doesn’t. Not: it works 95% of the time for non-colorblind English speakers who came to your website with certain expectations.

    I see this distinction as the reason that MVC is so popular. The stuff that can only be improved through trial and error (which includes usability tests and rapid iteration) goes in the Controller. The well-defined requirements go into the Model. And every program has some amount of both.

    The trick is that as a program matures, you can move stuff from the Controller into the Model. Once—or if—you get to the point where properties of your design components are unlikely to change, then you formalize them into reusable classes with testable traits. For example, buttons have well-known characteristics and state models. A radio button can have only one selection, and cannot be changed when disabled. You can test those programmatically.

    Angular (https://angularjs.org/) encourages UI components to have unit tests for their states. Since it’s baked into the framework design, it’s not too burdensome. The tests will help with debugging, especially for edge cases that are hard to reproduce. But they won’t answer the big question of whether the users can actually figure out how to use the UI, or even if they will find it useful.

  • Arni Leibovits

    Tere, Joosep! Great article! I agree with most of it, and we at our company follow this kind of philosophy.

    However, I would argue regarding one point – that TDD would not be faster in the short term. Of course this depends on what you are developing. However, when you’re dealing with complex logic having a large combination of states and variables, I believe TDD can actually speed up development, as you get instant feedback, and you’re able to test every possible case. Thoroughly testing such logic manually would be tedious, and finding some combination of cases might not even be possible.
    However, of course, manual testing has it’s place. :)

  • Joosep Simm

    I agree 100%

    Also, the speed difference depends a lot from developer experience with TDD and the framework/IDE support.

  • Jiggy

    I agree! Well documented points. Thanks for sharing.