Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a mutation is not detected by the test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the program's semantics unchanged-and thus cannot be detected by any test. Such equivalent mutants have to be eliminated manually, which is tedious.

We assess the impact of mutations by checking dynamic invariants. In an evaluation of our JAVALANCHE framework on seven industrial-size programs, we found that mutations that violate invariants are significantly more likely to be detectable by a test suite. As a consequence, mutations with impact on invariants should be focused upon when improving test suites. With less than 3% of equivalent mutants, our approach provides an efficient, precise, and fully automatic measure of the adequacy of a test suite.

REFERENCES

• 1: K. Adamopoulos, M. Harman, and R. M. Hierons. How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In Genetic and Evolutionary Computation - GECCO 2004, volume 3103 of Lecture Notes in Computer Science, pages 1338--1349, Seattle, Washington, 2004. Springer.
• 2: J. H. Andrews , L. C. Briand , Y. Labiche, Is mutation an appropriate tool for testing experiments?, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062530]
• 3: D. Baldwin and F. Sayward. Heuristics for determining equivalence of program mutations. Technical Report 276, Yale University, Department of Computer Science, 1979.
• 4: V. Dallmeier, C. Lindig, and A. Zeller. Lightweight defect localization for Java. In ECOOP '05: Proceedings of 19th European Conference on Object-Oriented Programming, number 3586 in Lecture Notes in Computer Science, pages 528--550. Springer, 2005.
• 5: R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Offutt. An extended overview of the Mothra software testing environment. In Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, pages 142--151, Ban, Alberta, 1988. IEEE Computer Society Press.
• 6: R. A. DeMillo , R. J. Lipton , F. G. Sayward, Hints on Test Data Selection: Help for the Practicing Programmer, Computer, v.11 n.4, p.34-41, April 1978 [doi>10.1109/C-M.1978.218136]
• 7: Michael D. Ernst , Jake Cockrell , William G. Griswold , David Notkin, Dynamically Discovering Likely Program Invariants to Support Program Evolution, IEEE Transactions on Software Engineering, v.27 n.2, p.99-123, February 2001 [doi>10.1109/32.908957]
• 8: Phyllis G. Frankl , Stewart N. Weiss , Cang Hu, All-uses vs mutation testing: an experimental comparison of effectiveness, Journal of Systems and Software, v.38 n.3, p.235-253, Sept. 1997 [doi>10.1016/S0164-1212(96)00154-9]
• 9: B. J. M. Grün, D. Schuler, and A. Zeller. The impact of equivalent mutants. Technical report, Saarland University, 2009. Short paper submitted to Mutation 2009: International Workshop on Mutation Analysis.
• 10: Sudheendra Hangal , Monica S. Lam, Tracking down software bugs using automatic anomaly detection, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581377]
• 11: R. Hierons and M. Harman. Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability, 9(4):233--262, 1999.
• 12: W. E. Howden, Weak Mutation Testing and Completeness of Test Sets, IEEE Transactions on Software Engineering, v.8 n.4, p.371-379, July 1982 [doi>10.1109/TSE.1982.235571]
• 13: http://asm.objectweb.org/. ASM 2.2.1 GPL, 2008.
• 14: Yu-Seung Ma , Jeff Offutt , Yong-Rae Kwon, MuJava: a mutation system for java, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134425]
• 15: Stephen McCamant , Michael D. Ernst, Predicting problems caused by component upgrades, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 16: Bertrand Meyer, Object-Oriented Software Construction, Prentice-Hall, Inc., Upper Saddle River, NJ, 1988
• 17: Akbar Siami Namin , James H. Andrews , Duncan J. Murdoch, Sufficient mutation operators for measuring test effectiveness, Proceedings of the 30th international conference on Software engineering, May 10-18, 2008, Leipzig, Germany [doi>10.1145/1368088.1368136]
• 18: A. J. Offutt and W. M. Craft. Using compiler optimiziation techniques to detect equivalent mutants. Software Testing, Verification, and Reliability, 4:131--154, 1994.
• 19: A. Jefferson Offutt , Ammei Lee , Gregg Rothermel , Roland H. Untch , Christian Zapf, An experimental determination of sufficient mutant operators, ACM Transactions on Software Engineering and Methodology (TOSEM), v.5 n.2, p.99-118, April 1996 [doi>10.1145/227607.227610]
• 20: A. J. Offutt and J. Pan. Detecting equivalent mutants and the feasible path problem. In COMPASS '96: Proceedings 11th Conference on Computer Assurance, pages 224--236, Gathersburg, MD, 1996.
• 21: A. J. Offutt and R. H. Untch. Mutation 2000: Uniting the orthogonal, pages 34--44. Kluwer Academic Publishers, Norwell, MA, USA, 2001.
• 22: C. Pacheco and M. D. Ernst. Eclat: Automatic generation and classification of test inputs. In ECOOP '05: Procceedings of the 9th European Conference on Object-Oriented Programming, pages 504--527, Glasgow, Scotland, 2005.
• 23: Suzette Person , Matthew B. Dwyer , Sebastian Elbaum , Corina S. Pǎsǎreanu, Differential symbolic execution, Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, November 09-14, 2008, Atlanta, Georgia [doi>10.1145/1453101.1453131]
• 24: Barbara G. Ryder , Frank Tip, Change impact analysis for object-oriented programs, Proceedings of the 2001 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.46-53, June 2001, Snowbird, Utah, United States [doi>10.1145/379605.379661]
• 25: Rok Sosič , David Abramson, Guard: a relative debugger, Software—Practice & Experience, v.27 n.2, p.185-206, Feb. 1997 [doi>10.1002/(SICI)1097-024X(199702)27:2<185::AID-SPE79>3.0.CO;2-D]
• 26: Roland H. Untch , A. Jefferson Offutt , Mary Jean Harrold, Mutation analysis using mutant schemata, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.139-148, June 28-30, 1993, Cambridge, Massachusetts, United States
• 27: Patrick Joseph Walsh, A measure of test case completeness (software, engineering), State University of New York at Binghamton, Binghamton, NY, 1985

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Tracing

General Terms:
Experimentation

Keywords:
dynamic invariants, mutation testing