An experimental evaluation of continuous testing during development

ABSTRACT

Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It is intended to reduce the time and energy required to keep code well-tested and prevent regression errors from persisting uncaught for long periods of time. This paper reports on a controlled human experiment to evaluate whether students using continuous testing are more successful in completing programming assignments. We also summarize users' subjective impressions and discuss why the results may generalize.The experiment indicates that the tool has a statistically significant effect on success in completing a programming task, but no such effect on time worked. Participants using continuous testing were three times more likely to complete the task before the deadline than those without. Participants using continuous compilation were twice as likely to complete the task, providing empirical support to a common feature in modern development environments. Most participants found continuous testing to be useful and believed that it helped them write better code faster, and 90% would recommend the tool to others. The participants did not find the tool distracting, and intuitively developed ways of incorporating the feedback into their workflow.

REFERENCES

• 1: Victor R. Basili, The role of experimentation in software engineering: past, current, and future, Proceedings of the 18th international conference on Software engineering, p.442-449, March 25-29, 1996, Berlin, Germany
• 2: Kent Beck, Extreme programming explained: embrace change, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 3: Beck, Test Driven Development: By Example, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 4: Margaret Burnett , Curtis Cook , Omkar Pendse , Gregg Rothermel , Jay Summet , Chris Wallace, End-user software engineering with assertions in the spreadsheet paradigm, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 5: Bruce Childers , Jack W. Davidson , Mary Lou Soffa, Continuous Compilation: A New Approach to Aggressive and Adaptive Code Transformation, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, p.205.1, April 22-26, 2003
• 6: Allen Cypher , Daniel C. Halbert , David Kurlander , Henry Lieberman , David Maulsby , Brad A. Myers , Alan Turransky, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 7: Alastair Dunsmore , Marc Roper , Murray Wood, Further investigations into the development and evaluation of reading techniques for object-oriented code inspection, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581349]
• 8: Eclipse. http://www.eclipse.org.
• 9: Emacs. http://www.emacs.org.
• 10: M. Jean Harrold , Rajiv Gupta , Mary Lou Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.3, p.270-285, July 1993 [doi>10.1145/152388.152391]
• 11: Peter Henderson , Mark Weiser, Continuous execution: the VisiProg environment, Proceedings of the 8th international conference on Software engineering, p.68-74, August 28-30, 1985, London, England
• 12: JetBrains IntelliJ IDEA. http://www.intellij.com/idea/.
• 13: JUnit. http://www.junit.org.
• 14: Michael Karasick, The architecture of Montana: an open and extensible programming environment with an incremental C++ compiler, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, p.131-142, November 01-05, 1998, Lake Buena Vista, Florida, United States
• 15: Tessa A. Lau , Pedro Domingos , Daniel S. Weld, Version Space Algebra and its Application to Programming by Demonstration, Proceedings of the Seventeenth International Conference on Machine Learning, p.527-534, June 29-July 02, 2000
• 16: H. K. N. Leung and L. White. Insights into regression testing. In Proceedings of the Conference on Software Maintenance, pages 60--69, Miami, FL, Oct. 16-19, 1989.
• 17: Robert C. Miller , Brad A. Myers , David Garlan, Lightweight structure in text, 2002
• 18: Daniel L. Moody , Guttorm Sindre , Terje Brasethvik , Arne Sølvberg, Evaluating the quality of information models: empirical testing of a conceptual model quality framework, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 19: Jeremy W. Nimmer , Michael D. Ernst, Invariant inference for static checking:, Proceedings of the 10th ACM SIGSOFT symposium on Foundations of software engineering, November 18-22, 2002, Charleston, South Carolina, USA [doi>10.1145/587051.587054]
• 20: Robert P. Nix, Editing by example, ACM Transactions on Programming Languages and Systems (TOPLAS), v.7 n.4, p.600-621, Oct. 1985 [doi>10.1145/4472.4476]
• 21: Alessandro Orso , Taweesup Apiwattanapong , Mary Jean Harrold, Leveraging field data for impact analysis and regression testing, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 22: Alessandro Orso , Donglin Liang , Mary Jean Harrold , Richard Lipton, Gamma system: continuous evolution of software after deployment, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 23: Christina Pavlopoulou , Michal Young, Residual test coverage monitoring, Proceedings of the 21st international conference on Software engineering, p.277-284, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302637]
• 24: Michael P. Plezbert , Ron K. Cytron, Does “just in time” = “better late than never”?, Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.120-131, January 15-17, 1997, Paris, France [doi>10.1145/263699.263713]
• 25: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
• 26: Gregg Rothermel , Roland J. Untch , Chengyun Chu, Prioritizing Test Cases For Regression Testing, IEEE Transactions on Software Engineering, v.27 n.10, p.929-948, October 2001 [doi>10.1109/32.962562]
• 27: D. Saff. Automated continuous testing to speed software development. Master's thesis, MIT Department of Electrical Engineering and Computer Science, Cambridge, MA, Feb. 3, 2004.
• 28: David Saff , Michael D. Ernst, Reducing wasted development time via continuous testing, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.281, November 17-21, 2003
• 29: David Saff , Michael D. Ernst, Mock object creation for test factoring, Proceedings of the ACM-SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, June 07-08, 2004, Washington DC, USA [doi>10.1145/996821.996838]
• 30: D. Saff and M. D. Ernst. Continuous testing in Eclipse. In 2nd Eclipse Technology Exchange Workshop (eTX), Barcelona, Spain, Mar. 30, 2004.
• 31: Mayer D. Schwartz , Norman M. Delisle , Vimal S. Begwani, Incremental compilation in Magpie, Proceedings of the 1984 SIGPLAN symposium on Compiler construction, p.122-131, June 17-22, 1984, Montreal, Canada
• 32: Shel Siegel, Object oriented software testing: a hierarchical approach, John Wiley & Sons, Inc., New York, NY, 1996
• 33: M. L. Soffa. Continuous testing. Personal communication, Feb. 2003.
• 34: Amitabh Srivastava , Jay Thiagarajan, Effectively prioritizing tests in development environment, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 35: Tacoma community college elementary algebra syllabus. http://www.tacoma.ctc.edu/home/jkellerm/MATH090/default.htm.
• 36: W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997
• 37: Andreas Zeller, Yesterday, my program worked. Today, it does not. Why?, Proceedings of the 7th European software engineering conference held jointly with the 7th ACM SIGSOFT international symposium on Foundations of software engineering, p.253-267, September 06-10, 1999, Toulouse, France
• 38: Marc K. Zimmerman , Kristina Lundqvist , Nancy Leveson, Investigating the readability of state-based formal requirements specification languages, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581347]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments
          Subjects: Integrated environments

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Keywords:
continuous compilation, continuous testing, test-first development, unit testing