Is mutation an appropriate tool for testing experiments?

ABSTRACT

The empirical assessment of test techniques plays an important role in software testing research. One common practice is to instrument faults, either manually or by using mutation operators. The latter allows the systematic, repeatable seeding of large numbers of faults; however, we do not know whether empirical results obtained this way lead to valid, representative conclusions. This paper investigates this important question based on a number of programs with comprehensive pools of test cases and known faults. It is concluded that, based on the data available thus far, the use of mutation operators is yielding trustworthy results (generated mutants are similar to real faults). Mutants appear however to be different from hand-seeded faults that seem to be harder to detect than real faults.

REFERENCES

• 1: James H. Andrews , Yingjun Zhang, General Test Result Checking with Log File Analysis, IEEE Transactions on Software Engineering, v.29 n.7, p.634-648, July 2003 [doi>10.1109/TSE.2003.1214327]
• 2: L. C. Briand , Y. Labiche , Y. Wang, Using Simulation to Empirically Investigate Test Coverage Criteria Based on Statechart, Proceedings of the 26th International Conference on Software Engineering, p.86-95, May 23-28, 2004
• 3: T. A. Budd and D. Angluin, "Two Notions of Correctness and their Relation to Testing," Acta Informatica, vol. 18 (1), pp. 31--45, 1982.
• 4: D. T. Campbell and J. C. Stanley, Experimental and Quasi-Experimental Designs for Research, Houghton Mifflin Company, 1990.
• 5: W. Chen, R. H. Untch, G. Rothermel, S. Elbaum and J. von Ronne, "Can fault-exposure-potential estimates improve the fault detection abilities of test suites?," Software Testing, Verification and Reliability, vol. 12 (4), pp. 197--218, 2002.
• 6: R. A. DeMillo, R. J. Lipton and F. G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," IEEE Computer, vol. 11 (4), pp. 34--41, 1978.
• 7: J. L. Devore, Probability and Statistics for Engineering and the Sciences, Duxbury Press, 5th Edition, 1999.
• 8: H. Do, G. Rothermel and S. Elbaum, "Infrastructure support for controlled experimentation with software testing and regression testing techniques," Oregon State University, Corvallis, OR, USA, Technical report 04-06-01, January, 2004.
• 9: Phyllis G. Frankl , Oleg Iakounenko, Further empirical studies of test effectiveness, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, p.153-162, November 01-05, 1998, Lake Buena Vista, Florida, United States
• 10: Phyllis G. Frankl , Stewart N. Weiss, An experimental comparison of the effectiveness of the all-uses and all-edges adequacy criteria, Proceedings of the symposium on Testing, analysis, and verification, p.154-164, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120821]
• 11: Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.2, p.184-208, April 2001 [doi>10.1145/367008.367020]
• 12: R. G. Hamlet, "Testing programs with the aid of a compiler," IEEE Transactions on Software Engineering, vol. 3 (4), pp. 279--290, 1977.
• 13: Michael Harder , Jeff Mellen , Michael D. Ernst, Improving test suites via operational abstraction, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 14: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 15: S. Kim, J. A. Clark and J. A. McDermid, "Investigating the Effectiveness of Object-Oriented Testing Strategies with the Mutation Method," Software Testing, Verification and Reliability, vol. 11 (3), pp. 207--225, 2001.
• 16: A. M. Memon, I. Banerjee and A. Nagarajan, "What Test Oracle Should I use for Effective GUI Testing?," Proc. IEEE International Conference on Automated Software Engineering (ASE'03), Montreal, Quebec, Canada, pp. 164--173, October, 2003.
• 17: A. Jefferson Offutt, Investigations of the software testing coupling effect, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.1, p.5-20, Jan. 1992 [doi>10.1145/125489.125473]
• 18: A. Jefferson Offutt , Ammei Lee , Gregg Rothermel , Roland H. Untch , Christian Zapf, An experimental determination of sufficient mutant operators, ACM Transactions on Software Engineering and Methodology (TOSEM), v.5 n.2, p.99-118, April 1996 [doi>10.1145/227607.227610]
• 19: A. J. Offutt and J. Pan, "Detecting Equivalent Mutants and the Feasible Path Problem," Software Testing, Verification, and Reliability, vol. 7 (3), pp. 165--192, 1997.
• 20: A. J. Offutt and R. H. Untch, "Mutation 2000: Uniting the Orthogonal," Proc. Mutation, San Jose, CA, USA, pp. 45--55, October, 2000.
• 21: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 22: J. Rice, Mathematical Statistics and Data Analysis, Duxbury press, 2nd Edition, 1995.
• 23: Gregg Rothermel , Mary Jean Harrold, Empirical Studies of a Safe Regression Test Selection Technique, IEEE Transactions on Software Engineering, v.24 n.6, p.401-419, June 1998 [doi>10.1109/32.689399]
• 24: P. Thévenod-Fosse, H. Waeselynck and Y. Crouzet, "An experimental study on software structural testing: deterministic versus random input generation," Proc. 21st International Symposium on Fault-Tolerant Computing, Montreal, Canada, pp. 410--417, June, 1991.
• 25: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998
• 26: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

Keywords:
hand-seeded faults, mutants, real faults