Improving test suites for efficient fault localization

ABSTRACT

The need for testing-for-diagnosis strategies has been identified for a long time, but the explicit link from testing to diagnosis (fault localization) is rare. Analyzing the type of information needed for efficient fault localization, we identify the attribute (called Dynamic Basic Block) that restricts the accuracy of a diagnosis algorithm. Based on this attribute, a test-for-diagnosis criterion is proposed and validated through rigorous case studies: it shows that a test suite can be improved to reach a high level of diagnosis accuracy. So, the dilemma between a reduced testing effort (with as few test cases as possible) and the diagnosis accuracy (that needs as much test cases as possible to get more information) is partly solved by selecting test cases that are dedicated to diagnosis.

REFERENCES

• 1: H. Agrawal, J. Horgan, S. London, and W. Wong. Fault Localization using Execution Slices and Dataflow Tests. Proceedings of ISSRE'95 (Int. Symposium on Software Reliability Engineering), Toulouse, France, October 1995.
• 2: Benoit Baudry , Franck Fleurey , Jean-Marc Jézéquel , Yves Le Traon, Automatic Test Cases Optimization Using a Bacteriological Adaptation Model: Application to .NET Components, Proceedings of the 17th IEEE international conference on Automated software engineering, p.253, September 23-27, 2002
• 3: Benoit Baudry , Franck Fleurey , Jean-Marc Jezequel , Yves Le Traon, Automatic Test Case Optimization: A Bacteriologic Algorithm, IEEE Software, v.22 n.2, p.76-82, March 2005 [doi>10.1109/MS.2005.30]
• 4: Holger Cleve , Andreas Zeller, Locating causes of program failures, Proceedings of the 27th international conference on Software engineering, p.342-351, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062522]
• 5: V. Dallmeier, C. Lindig, and A. Zeller. Lightweight Defect Localization for Java. Proceedings of ECOOP'05 (European Conference on Object-Oriented Programming), Glasgow, Scotland, July 2005.
• 6: R. DeMillo, R. Lipton, and F. Sayward. Hints on Test Data Selection: Help For The Practicing Programmer. IEEE Computer, 1978. 11(4): 34--41.
• 7: F. Fleurey, B. Baudry, and Y. Le Traon. Improving Test Cases for Accurate Diagnosis. Accessed on: May 2005.http://www.irisa.fr/triskell/results/Diagnosis/index.htm
• 8: James A. Jones , Mary Jean Harrold, Empirical evaluation of the tarantula automatic fault-localization technique, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101949]
• 9: James A. Jones , Mary Jean Harrold , John Stasko, Visualization of test information to assist fault localization, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581397]
• 10: Maisaa Khalil , Yves Le Traon , Chantal Robach, Towards an automatic diagnosis for high-level design validation, Proceedings of the 1998 IEEE International Test Conference, p.1010, October 18-22, 1998
• 11: B. Korel. Dynamic method for software test data generation. Software Testing, Verification and Reliability, 1992. 2(4): 203--213.
• 12: Bertrand Meyer, Object-oriented software construction (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1997
• 13: C. C. Michael , G. McGraw , M. A. Schatz, Generating Software Test Data by Evolution, IEEE Transactions on Software Engineering, v.27 n.12, p.1085-1110, December 2001 [doi>10.1109/32.988709]
• 14: A. Jefferson Offutt , Ammei Lee , Gregg Rothermel , Roland H. Untch , Christian Zapf, An experimental determination of sufficient mutant operators, ACM Transactions on Software Engineering and Methodology (TOSEM), v.5 n.2, p.99-118, April 1996 [doi>10.1145/227607.227610]
• 15: M.L. Rosenzweig. Species Diversity In Space and Time. Cambridge University Press, 1995.
• 16: J. Wegener, A. Baresel, and H. Stahmer. Evolutionary Test Environment for Automatic Structural Testing. Information and Software Technology, 2001. 43(14): 841--854.
• 17: Andreas Zeller , Ralf Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE Transactions on Software Engineering, v.28 n.2, p.183-200, February 2002 [doi>10.1109/32.988498]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

General Terms:
Measurement, Verification

Keywords:
diagnosis, mutation analysis, test generation