An empirical study of the effect of time constraints on the cost-benefits of regression testing

ABSTRACT

Regression testing is an expensive process used to validate modified software. Test case prioritization techniques improve the cost-effectiveness of regression testing by ordering test cases such that those that are more important are run earlier in the testing process. Many prioritization techniques have been proposed and evidence shows that they can be beneficial. It has been suggested, however, that the time constraints that can be imposed on regression testing by various software development processes can strongly affect the behavior of prioritization techniques. Therefore, we conducted an experiment to assess the effects of time constraints on the costs and benefits of prioritization techniques. Our results show that time constraints can indeed play a significant role in determining both the cost-effectiveness of prioritization, and the relative cost-benefit tradeoffs among techniques, with important implications for the use of prioritization in practice.

REFERENCES

• 1: J. H. Andrews , L. C. Briand , Y. Labiche, Is mutation an appropriate tool for testing experiments?, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062530]
• 2: L. Briand and J. Wüst. Empirical studies of quality models in object-oriented systems. Adv. Comp., 56:98--167, 2002.
• 3: Yih-Farn Chen , David S. Rosenblum , Kiem-Phong Vo, TestTube: a system for selective regression testing, Proceedings of the 16th international conference on Software engineering, p.211-220, May 16-21, 1994, Sorrento, Italy
• 4: Shyam R. Chidamber , Chris F. Kemerer, Towards a metrics suite for object oriented design, Conference proceedings on Object-oriented programming systems, languages, and applications, p.197-211, October 06-11, 1991, Phoenix, Arizona, United States
• 5: M. S. Christian Collberg, Ginger Myles. An empirical study of Java bytecode programs. Technical Report TR04-11, Dept. of Computer Science, University of Arizona, 2004.
• 6: Hyunsook Do , Sebastian Elbaum , Gregg Rothermel, Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact, Empirical Software Engineering, v.10 n.4, p.405-435, October 2005 [doi>10.1007/s10664-005-3861-2]
• 7: Hyunsook Do , Gregg Rothermel, An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models, Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, November 05-11, 2006, Portland, Oregon, USA [doi>10.1145/1181775.1181793]
• 8: Hyunsook Do , Gregg Rothermel, On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques, IEEE Transactions on Software Engineering, v.32 n.9, p.733-752, September 2006 [doi>10.1109/TSE.2006.92]
• 9: Hyunsook Do , Gregg Rothermel, Using sensitivity analysis to create simplified economic models for regression testing, Proceedings of the 2008 international symposium on Software testing and analysis, July 20-24, 2008, Seattle, WA, USA [doi>10.1145/1390630.1390639]
• 10: Sebastian Elbaum , Alexey Malishevsky , Gregg Rothermel, Incorporating varying test costs and fault severities into test case prioritization, Proceedings of the 23rd International Conference on Software Engineering, p.329-338, May 12-19, 2001, Toronto, Ontario, Canada
• 11: Sebastian Elbaum , Alexey G. Malishevsky , Gregg Rothermel, Test Case Prioritization: A Family of Empirical Studies, IEEE Transactions on Software Engineering, v.28 n.2, p.159-182, February 2002 [doi>10.1109/32.988497]
• 12: Dennis Jeffrey , Neelam Gupta, Test Case Prioritization Using Relevant Slices, Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06), p.411-420, September 17-21, 2006 [doi>10.1109/COMPSAC.2006.80]
• 13: Finn V. Jensen, Introduction to Bayesian Networks, Springer-Verlag New York, Inc., Secaucus, NJ, 1996
• 14: Capers Jones, Applied software measurement (2nd ed.): assuring productivity and quality, McGraw-Hill, Inc., Hightstown, NJ, 1996
• 15: Jung-Min Kim , Adam Porter, A history-based test prioritization technique for regression testing in resource constrained environments, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581357]
• 16: A. Kinneer, M. Dwyer, and G. Rothermel. Sofya: A flexible framework for development of dynamic program analysis for Java software. Technical Report TR-UNL-CSE-2006-0006, University of Nebraska--Lincoln, Apr. 2006.
• 17: David Leon , Andy Podgurski, A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.442, November 17-21, 2003
• 18: Zheng Li , Mark Harman , Robert M. Hierons, Search Algorithms for Regression Test Case Prioritization, IEEE Transactions on Software Engineering, v.33 n.4, p.225-237, April 2007 [doi>10.1109/TSE.2007.38]
• 19: Modeling the Cost-Benefits Tradeoffs for Regression Testing Techniques, Proceedings of the International Conference on Software Maintenance (ICSM'02), p.204, October 03-06, 2002
• 20: S. Mirarab and L. Tahvildari. A prioritization approach for software test cases on Bayesian Networks. In Found. App. Softw. Eng., LNCS 4422-0276, pages 276--290, Mar. 2007.
• 21: Siavash Mirarab , Ladan Tahvildari, An Empirical Study on Bayesian Network-based Approach for Test Case Prioritization, Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation, p.278-287, April 09-11, 2008 [doi>10.1109/ICST.2008.57]
• 22: Akira K. Onoma , Wei-Tek Tsai , Mustafa Poonawala , Hiroshi Suganuma, Regression testing in an industrial environment, Communications of the ACM, v.41 n.5, p.81-86, May 1998 [doi>10.1145/274946.274960]
• 23: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 24: Dewayne E. Perry , Carol S. Stieg, Software Faults in Evolving a Large, Real-Time System: a Case Study, Proceedings of the 4th European Software Engineering Conference on Software Engineering, p.48-67, September 13-17, 1993
• 25: F. L. Ramsey and D. W. Schafer. The Statistical Sleuth. Duxbury Press, 1st edition, 1997.
• 26: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
• 27: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 28: Gregg Rothermel , Roland J. Untch , Chengyun Chu, Prioritizing Test Cases For Regression Testing, IEEE Transactions on Software Engineering, v.27 n.10, p.929-948, October 2001 [doi>10.1109/32.962562]
• 29: Forrest Shull , Vic Basili , Barry Boehm , ,A. Winsor Brown , Patricia Costa , Mikael Lindvall , Dan Port , Ioana Rus , Roseanne Tesoriero , Marvin Zelkowitz, What We Have Learned About Fighting Defects, Proceedings of the 8th International Symposium on Software Metrics, p.249, June 04-07, 2002
• 30: Amitabh Srivastava , Jay Thiagarajan, Effectively prioritizing tests in development environment, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 31: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 32: Kristen R. Walcott , Mary Lou Soffa , Gregory M. Kapfhammer , Robert S. Roos, TimeAware test suite prioritization, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146240]
• 33: W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Experimentation, Measurement, Verification

Keywords:
empirical studies, regression testing, test case prioritization