Using perturbation analysis to measure variation in the information content of test sets

ABSTRACT

We define the information content of test set T with respect to a program P to be the degree to which the behavior of P on T approximates the overall behavior of P. Informally, the higher the information content of a test set, the greater the likelihood an error in the data state of a program will be manifested under testing.Perturbation analysis injects errors into the data state of an executing program and traces the impact of those errors on the intervening states and the program's output. The injection is performed by perturbation functions that randomly change the program's data state. Using perturbation analysis we demonstrate that different test sets may satisfy the same testing criterion but have significantly different information content.We believe that "consistency of information content" is a crucial measure of the quality of a testing strategy. We show how perturbation analysis may be used to assess individual testing strategies and to compare different testing strategies.The "coupling effect" of mutation testing implies that there is little variation among mutation-adequate test sets for a program. This implication is investigated for two simple programs by analyzing the variation among several mutation-adequate test sets.

REFERENCES

• 1: R. A. DeMillo, D. S. Guindi, K. King, W. M. Mc- Cracken, and A. J. Offutt. An extended overview of the mothra software testing environment. Second Workshop on Software Testing, Vahdation, and Analyszs, pages pp. 142-151, July 1988.
• 2: T. Goradia. Dynamic impact analysis. ISSTA '93, pages 171-181, June 1993.
• 3: L. J. Morell, A Theory of Fault-Based Testing, IEEE Transactions on Software Engineering, v.16 n.8, p.844-857, August 1990 [doi>10.1109/32.57623]
• 4: Larry Morell , Branson Murrill, Semantic metrics through error flow analysis, Journal of Systems and Software, v.20 n.3, p.253-265, March 1993 [doi>10.1016/0164-1212(93)90069-A]
• 5: Jeffrey M. Voas, PIE: A Dynamic Failure-Based Technique, IEEE Transactions on Software Engineering, v.18 n.8, p.717-727, August 1992 [doi>10.1109/32.153381]
• 6: Jeffrey Voas , Larry Morrel , Keith Miller, Predicting Where Faults Can Hide from Testing, IEEE Software, v.8 n.2, p.41-48, March 1991 [doi>10.1109/52.73748]
• 7: J. M. Voas and K. W. Miller. The revealing power of a test case. Software Testzng, Verification and Reliability, 2:25-42, 1992.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Tracing

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Design, Theory, Verification