An empirical study of the effects of test-suite reduction on fault localization

ABSTRACT

Fault-localization techniques that utilize information about all test cases in a test suite have been presented. These techniques use various approaches to identify the likely faulty part(s) of a program, based on information about the execution of the program with the test suite. Researchers have begun to investigate the impact that the composition of the test suite has on the effectiveness of these fault-localization techniques. In this paper, we present the first experiment on one aspect of test-suite composition--test-suite reduction. Our experiment studies the impact of the test-suite reduction on the effectiveness of fault-localization techniques. In our experiment, we apply 10 test-suite reduction strategies to test suites for eight subject programs. We then measure the differences between the effectiveness of four existing fault-localization techniques on the unreduced and reduced test suites. We also measure the reduction in test-suite size of the 10 test-suite reduction strategies. Our experiment shows that fault-localization effectiveness varies depending on the test-suite reduction strategy used, and it demonstrates the trade-offs between test-suite reduction and fault-localization effectiveness.

REFERENCES

• 1: Rui Abreu , Peter Zoeteweij , Arjan J. C. van Gemund, On the Accuracy of Spectrum-based Fault Localization, Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, p.89-98, September 10-14, 2007
• 2: H. Agrawal, J. Horgan, S. London, and W. Wong. Fault localization using execution slices and dataflow tests. In Proceedings of IEEE Software Reliability Engineering, pages 143--151, 1995.
• 3: Aristotle Research Group. Aristotle analysis system, 2007. http://www.cc.gatech.edu/aristotle/.
• 4: Benoit Baudry , Franck Fleurey , Yves Le Traon, Improving test suites for efficient fault localization, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134299]
• 5: Tsong Yueh Chen , Man Fai Lau, Dividing strategies for the optimization of a test suite, Information Processing Letters, v.60 n.3, p.135-141, Nov. 11, 1996 [doi>10.1016/S0020-0190(96)00135-4]
• 6: Holger Cleve , Andreas Zeller, Locating causes of program failures, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062522]
• 7: Dan Hao , Ying Pan , Lu Zhang , Wei Zhao , Hong Mei , Jiasu Sun, A similarity-aware approach to testing based fault localization, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101953]
• 8: M. Jean Harrold , Rajiv Gupta , Mary Lou Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.3, p.270-285, July 1993 [doi>10.1145/152388.152391]
• 9: J. R. Horgan and S. A. London. ATAC: A data flow coverage testing tool for C. In Proceedings of the Symp. on Assessment of Quality Software Development Tools, pages 2--10, May 1992.
• 10: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 11: James A. Jones , Mary Jean Harrold, Empirical evaluation of the tarantula automatic fault-localization technique, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101949]
• 12: James A. Jones , Mary Jean Harrold , John Stasko, Visualization of test information to assist fault localization, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581397]
• 13: Ben Liblit , Mayur Naik , Alice X. Zheng , Alex Aiken , Michael I. Jordan, Scalable statistical bug isolation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 14: Chao Liu , Xifeng Yan , Long Fei , Jiawei Han , Samuel P. Midkiff, SOBER: statistical model-based bug localization, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 15: J. Offutt, J. Pan, and J. M. Voas. Procedures for reducing the size of coverage-based test sets. In Proceedings of the International Conference on Testing Comp. Software, pages 111--123, June 1995.
• 16: M. Renieris and S. Reiss. Fault localization with nearest neighbor queries. In Proceedings of the International Conference on Automated Software Engineering, pages 30--39, Montreal, Quebec, October 2003.
• 17: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

General Terms:
Experimentation, Reliability

Keywords:
empirical study, fault localization, test-suite reduction