Evaluating the efficacy of test-driven development: industrial case studies

ABSTRACT

This paper discusses software development using the Test Driven Development (TDD) methodology in two different environments (Windows and MSN divisions) at Microsoft. In both these case studies we measure the various context, product and outcome measures to compare and evaluate the efficacy of TDD. We observed a significant increase in quality of the code (greater than two times) for projects developed using TDD compared to similar projects developed in the same organization in a non-TDD fashion. The projects also took at least 15% extra upfront time for writing the tests. Additionally, the unit tests have served as auto documentation for the code when libraries/APIs had to be used as well as for code maintenance.

REFERENCES

• 1: "IEEE Std 982.2-1988 IEEE guide for the use of IEEE standard dictionary of measures to produce reliable software," 1988.
• 2: Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999 [doi>10.1109/32.799939]
• 3: Kent Beck, Extreme programming explained: embrace change, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 4: Beck, Test Driven Development: By Example, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 5: Jeffrey Carver , Letizia Jaccheri , Sandro Morasca , Forrest Shull, Issues in Using Students in Empirical Studies in Software Engineering Education, Proceedings of the 9th International Symposium on Software Metrics, p.239, September 03-05, 2003
• 6: Bill Curtis, Modeling the software process: three problems overcome with behavioral models of the software development process (panel session), Proceedings of the 11th international conference on Software engineering, p.398-399, May 1989, Pittsburgh, Pennsylvania, United States [doi>10.1145/74587.74643]
• 7: Hakan Erdogmus , Maurizio Morisio , Marco Torchiano, On the Effectiveness of the Test-First Approach to Programming, IEEE Transactions on Software Engineering, v.31 n.3, p.226-237, March 2005 [doi>10.1109/TSE.2005.37]
• 8: D. Gelperin and W. Hetzel, "Software Quality Engineering", Proceedings of Fourth International Conference on Software Testing, Washington D.C., June 1987.
• 9: Boby George , Laurie Williams, An initial investigation of test driven development in industry, Proceedings of the 2003 ACM symposium on Applied computing, March 09-12, 2003, Melbourne, Florida [doi>10.1145/952532.952753]
• 10: B. George and L. Williams, "A Structured Experiment of Test-Driven Development", Information and Software Technology (IST), 46(5), pp. 337--342, 2003.
• 11: Watts S. Humphrey, Managing the software process, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 12: Craig Larman , Victor R. Basili, Iterative and Incremental Development: A Brief History, Computer, v.36 n.6, p.47-56, June 2003 [doi>10.1109/MC.2003.1204375]
• 13: M. M. Müller and O. Hagner, "Experiment about Test-first Programming", Proceedings of Conference on Empirical Assessment in Software Engineering (EASE), 2002.
• 14: Matthias M. Müller , Walter F. Tichy, Case study: extreme programming in a university environment, Proceedings of the 23rd International Conference on Software Engineering, p.537-544, May 12-19, 2001, Toronto, Ontario, Canada
• 15: L. Williams, Krebs, W., Layman, L., "Extreme Programming Evaluation Framework for Object-Oriented Languages -- Version 1.1," Technical Report, North Carolina State University, NCSU CSC TR-2003-20, 2003.
• 16: Laurie Williams , E. Michael Maximilien , Mladen Vouk, Test-Driven Development as a Defect-Reduction Practice, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.34, November 17-21, 2003

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
      D.2.8 Metrics
          Subjects: Process metrics; Product metrics

Keywords:
software quality, test-driven development