Predicting dependability by testing

ABSTRACT

In assessing the quality of software, we would like to make engineering judgements similar to those based on statistical quality control. Ideally, we want to support statements like: "The confidence that this program's result at X is correct is p," where X is a particular vector of inputs, and confidence p is obtained from measurements of the software (perhaps involving X). For the theory to be useful, it must be feasible to predict values of p near 1 for many programs, for most values of X.Blum's theory of self-checking/correcting programs has exactly the right character, but it applies to only a few unusual problems. Conventional software reliability theory is widely applicable, but it yields only confidence in a failure intensity, and the measurements required to support a correctness-like failure intensity (say 10^-9/demand) are infeasible. Voas's sensitivity theory remedies these problems of reliability theory, but his model is too simple to be very plausible. In this paper we combine these ideas: reliability, sensitivity, and self-checking, to obtain new results on "dependability," plausible predictions of software quality.

REFERENCES

• BF91: Ricky W. Butler , George B. Finelli, The infeasibility of experimental quantification of life-critical software reliability, Proceedings of the conference on Software for citical systems, p.66-76, December 04-06, 1991, New Orleans, Louisiana, United States
• BK89: M. Blum , S. Kanna, Designing programs that check their work, Proceedings of the twenty-first annual ACM symposium on Theory of computing, p.86-97, May 14-17, 1989, Seattle, Washington, United States [doi>10.1145/73007.73015]
• BS95: Antonia Bertolino , Lorenzo Strigini, Using testability measures for dependability assessment, Proceedings of the 17th international conference on Software engineering, p.61-70, April 24-28, 1995, Seattle, Washington, United States [doi>10.1145/225014.225020]
• BW94: M. Blum and H. Wasserman. Program resultchecking: A theory of testing meets a test of theory. In 35th Annual Symposium on Foundations of Computer Science, pages 382-391~ Santa Fe, NM, 1994.
• Ham94: D. Hamlet. Random testing. In J. Marciniak, editor, Encyclopedia of Software Engineering, pages 970-978. Wiley, New York, 1994.
• HV93: Dick Hamlet , Jeff Voas, Faults on its sleeve: amplifying software reliability testing, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.89-98, June 28-30, 1993, Cambridge, Massachusetts, United States
• Sho83: Martin L. Shooman, Software engineering: reliability, development, and management., McGraw-Hill, Inc., New York, NY, 1983
• TLN78: R. Thayer, M. Lipow, and E. Nelson. Software Reliability. North-Holland, New York, NY, 1978.
• VM92: J. M. Voas and K. W. Miller. Improving the software development process using testability research. In Third International Symposium on Software Reliability Engineering, pages 114-121, Research Triangle Park, NC, 1992.
• VM95: Jeffrey M. Voas , Keith W. Miller, Software Testability: The New Verification, IEEE Software, v.12 n.3, p.17-28, May 1995 [doi>10.1109/52.382180]
• Voa92: Jeffrey M. Voas, PIE: A Dynamic Failure-Based Technique, IEEE Transactions on Software Engineering, v.18 n.8, p.717-727, August 1992 [doi>10.1109/32.153381]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Correctness proofs

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Combined

General Terms:
Design, Human Factors, Reliability, Theory