Further empirical studies of test effectiveness

ABSTRACT

This paper reports on an empirical evaluation of the fault-detecting ability of two white-box software testing techniques: decision coverage (branch testing) and the all-uses data flow testing criterion. Each subject program was tested using a very large number of randomly generated test sets. For each test set, the extent to which it satisfied the given testing criterion was measured and it was determined whether or not the test set detected a program fault. These data were used to explore the relationship between the coverage achieved by test sets and the likelihood that they will detect a fault.Previous experiments of this nature have used relatively small subject programs and/or have used programs with seeded faults. In contrast, the subjects used here were eight versions of an antenna configuration program written for the European Space Agency, each consisting of over 10,000 lines of C code.For each of the subject programs studied, the likelihood of detecting a fault increased sharply as very high coverage levels were reached. Thus, this data supports the belief that these testing techniques can be more effective than random testing. However, the magnitudes of the increases were rather inconsistent and it was difficult to achieve high coverage levels.

REFERENCES

• 1: Phyllis Frankl , Dick Hamlet , Bev Littlewood , Lorenzo Strigini, Choosing a testing method to deliver reliability, Proceedings of the 19th international conference on Software engineering, p.68-78, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253244]
• 2: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581]
• 3: Phyllis G. Frankl , Stewart N. Weiss , Cang Hu, All-uses vs mutation testing: an experimental comparison of effectiveness, Journal of Systems and Software, v.38 n.3, p.235-253, Sept. 1997 [doi>10.1016/S0164-1212(96)00154-9]
• 4: P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194]
• 5: P. G. Frankl , E. J. Weyuker, Provable Improvements on Branch Testing, IEEE Transactions on Software Engineering, v.19 n.10, p.962-975, October 1993 [doi>10.1109/32.245738]
• 6: J. R. Horgan , S. London, Data flow coverage and the C language, Proceedings of the symposium on Testing, analysis, and verification, p.87-97, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120815]
• 7: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 8: A. Mathur and W. E. Wong. Au empirical comparison of mutation and data flow-based test adequacy criteria. Technical Report SERGTR-135-P, Software Engineering Research Center, Purdue University, Mar. 1993.
• 9: A. Pasquini, A. Crespo, and P. MatreRa. Sensitivity of reliability-growth models to operational profiles errors vs testing accuracy. IEEE Transactions on Reliability, R-45(4):531-540, Dec. 1996.
• 10: Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 11: B. Rosner. Fundamentals ofBiostatistics. PWS-KENT, Boston, Mass., 1990.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Design, Experimentation, Measurement, Performance, Reliability, Theory