Testability, fault size and the domain-to-range ratio: An eternal triangle

ABSTRACT

A number of different concepts have been proposed that, loosely speaking, revolve around the notion of software testability. Indeed, the concept of testability itself has been interpreted in a variety of ways by the software community. One interpretation is concerned with the extent of the modifications a program component requires, in terms of its input and output variables, so that the entire behaviour of the component is observable and controllable. Another interpretation is the ease with which faults, if present in a program, can be revealed by the testing process and the propagation, infection and execution (PIE) model has been proposed as a method of estimating this. It has been suggested that this particular interpretation of testability might be linked with the metric domain-to-range ratio (DRR), i.e. the ratio of the cardinality of the set of all inputs (the domain) to the cardinality of the set of all outputs (the range). This paper reports work in progress exploring some of the connections between the concepts mentioned. In particular, a simple mathematical link is established between domain-to-range ratio and the observability and controllability aspects of testability. In addition, the PIE model is re-considered and a relationship with fault size is observed. This leads to the suggestion that it might be more straightforward to estimate PIE testability by an adaptation of traditional mutation analysis. The latter suggestion exemplifies the main goals of the work described here, namely to seek greater understanding of testability in general and, ultimately, to find easier ways of determining it.

REFERENCES

• 1: Richard Bache , Monika Mullerburg, Measures of testability as a basis for quality assurance, Software Engineering Journal, v.5 n.2, p.86-92, Mar. 1990
• 2: Bainbridge, J., "Defining testability metrics axiomatically", Software Testing, Verification and Reliability, 4(2), 63-80 (June 1994).
• 3: Antonia Bertolino , Lorenzo Strigini, On the Use of Testability Measures for Dependability Assessment, IEEE Transactions on Software Engineering, v.22 n.2, p.97-108, February 1996 [doi>10.1109/32.485220]
• 4: B. W. Boehm , J. R. Brown , M. Lipow, Quantitative evaluation of software quality, Proceedings of the 2nd international conference on Software engineering, p.592-605, October 13-15, 1976, San Francisco, California, United States
• 5: DeMillo, R.A., Guindi, D.S., McCracken, W.M., Offutt, A.J. and King, K.N., "An extended overview of the Mothra software testing environment", Proc. Second Workshop on Software Testing, Verification and Analysis, Banff, Canada, IEEE Press, pp. 142-151 (July 1988).
• 6: Roy S. Freedman, Testability of Software Components, IEEE Transactions on Software Engineering, v.17 n.6, p.553-564, June 1991 [doi>10.1109/32.87281]
• 7: Michael A. Friedman , Jeffrey M. Voas, Software Assessment: Reliability, Safety, Testability, John Wiley & Sons, Inc., New York, NY, 1995
• 8: Tarak Goradia, Dynamic impact analysis: a cost-effective technique to enforce error-propagation, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.171-181, June 28-30, 1993, Cambridge, Massachusetts, United States
• 9: Dick Hamlet , Jeff Voas, Faults on its sleeve: amplifying software reliability testing, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.89-98, June 28-30, 1993, Cambridge, Massachusetts, United States
• 10: How Tai Wah, K.S., "A theoretical study of fault coupling", Software Testing, Verification and Reliability, 10(1), 3-45 (March 2000).
• 11: L. J. Morell, A Theory of Fault-Based Testing, IEEE Transactions on Software Engineering, v.16 n.8, p.844-857, August 1990 [doi>10.1109/32.57623]
• 12: A. Jefferson Offutt , J. Huffman Hayes, A semantic model of program faults, Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis, p.195-200, January 08-10, 1996, San Diego, California, United States
• 13: Debra J. Richardson , Margaret C. Thompson, An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection, IEEE Transactions on Software Engineering, v.19 n.6, p.533-553, June 1993 [doi>10.1109/32.232020]
• 14: Gregg Rothermel , Roland H. Untch , Chengyun Chu , Mary Jean Harrold, Test Case Prioritization: An Empirical Study, Proceedings of the IEEE International Conference on Software Maintenance, p.179, August 30-September 03, 1999
• 15: Jeffrey M. Voas, PIE: A Dynamic Failure-Based Technique, IEEE Transactions on Software Engineering, v.18 n.8, p.717-727, August 1992 [doi>10.1109/32.153381]
• 16: Jeffrey M. Voas , Keith W. Miller, Semantic metrics for software testability, Journal of Systems and Software, v.20 n.3, p.207-216, March 1993 [doi>10.1016/0164-1212(93)90064-5]
• 17: Jeffrey M. Voas , Keith W. Miller, Software Testability: The New Verification, IEEE Software, v.12 n.3, p.17-28, May 1995 [doi>10.1109/52.382180]
• 18: Voas, J.M., Miller, K.W. and Noonan, R., "Designing programs that do not hide data state errors during random black-box testing", Proc. 5th Int. Conf. on Putting into Practice Methods and Tools for Information System Design, Nantes, France (Sept. 1992).
• 19: Jeffrey Voas , Larry Morrel , Keith Miller, Predicting Where Faults Can Hide from Testing, IEEE Software, v.8 n.2, p.41-48, March 1991 [doi>10.1109/52.73748]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS

General Terms:
Design, Languages, Performance, Theory

Keywords:
controllability, domain-to-range ratio, fault size, observability, testability