Comparison of delivered reliability of branch, data flow and operational testing: A case study

ABSTRACT

Many analytical and empirical studies of software testing effectiveness have used the probability that a test set exposes at least one fault as the measure of effectiveness. That measure is useful for evaluating testing techniques when the goal of testing is to gain confidence that the program is free from faults. However, if the goal of testing is to improve the reliability of the program (by discovering and removing those faults that are most likely to cause failures when the software is in the field) then the measure of test effectiveness must distinguish between those faults that are likely to cause failures and those that are unlikely to do so. Delivered reliability was previously introduced as a means of comparing testing techniques in that setting. This paper empirically compares reliability delivered by three testing techniques, branch testing, the all-uses data flow testing criterion, and operational testing. The subject program is a moderate-sized C-program (about 10,000 LOC) produced by professional programmers and containing naturally occurring faults.

REFERENCES

• 1: R. W. Butler , G. B. Finelli, The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software, IEEE Transactions on Software Engineering, v.19 n.1, p.3-12, January 1993 [doi>10.1109/32.210303]
• 2: Tsong Yueh Chen , Yuen Tak Yu, On the Expected Number of Failures Detected by Subdomain Testing and Random Testing, IEEE Transactions on Software Engineering, v.22 n.2, p.109-119, February 1996 [doi>10.1109/32.485221]
• 3: J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering, SE-10(7):438{444, July 1984.
• 4: Phyllis G. Frankl , Richard G. Hamlet , Bev Littlewood , Lorenzo Strigini, Evaluating Testing Methods by Delivered Reliability, IEEE Transactions on Software Engineering, v.24 n.8, p.586-601, August 1998 [doi>10.1109/32.707695]
• 5: Phyllis G. Frankl , Oleg Iakounenko, Further empirical studies of test effectiveness, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, p.153-162, November 01-05, 1998, Lake Buena Vista, Florida, United States
• 6: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581]
• 7: Phyllis G. Frankl , Stewart N. Weiss , Cang Hu, All-uses vs mutation testing: an experimental comparison of effectiveness, Journal of Systems and Software, v.38 n.3, p.235-253, Sept. 1997 [doi>10.1016/S0164-1212(96)00154-9]
• 8: P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194]
• 9: P. G. Frankl , E. J. Weyuker, A Formal Analysis of the Fault-Detecting Ability of Testing Methods, IEEE Transactions on Software Engineering, v.19 n.3, p.202-213, March 1993 [doi>10.1109/32.221133]
• 10: P. G. Frankl , E. J. Weyuker, Provable Improvements on Branch Testing, IEEE Transactions on Software Engineering, v.19 n.10, p.962-975, October 1993 [doi>10.1109/32.245738]
• 11: Dick Hamlet , Ross Taylor, Partition Testing Does Not Inspire Confidence (Program Testing), IEEE Transactions on Software Engineering, v.16 n.12, p.1402-1411, December 1990 [doi>10.1109/32.62448]
• 12: J. R. Horgan , S. London, Data flow coverage and the C language, Proceedings of the symposium on Testing, analysis, and verification, p.87-97, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120815]
• 13: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 14: Bev Littlewood , Lorenzo Strigini, Validation of ultrahigh dependability for software-based systems, Communications of the ACM, v.36 n.11, p.69-80, Nov. 1993 [doi>10.1145/163359.163373]
• 15: Michael R. Lyu, Handbook of software reliability and system reliability, McGraw-Hill, Inc., Hightstown, NJ, 1996
• 16: A. Jefferson Offutt, Investigations of the software testing coupling effect, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.1, p.5-20, Jan. 1992 [doi>10.1145/125489.125473]
• 17: A. J. Offutt , S. D. Lee, An Empirical Evaluation of Weak Mutation, IEEE Transactions on Software Engineering, v.20 n.5, p.337-344, May 1994 [doi>10.1109/32.286422]
• 18: A. Pasquini, A. Crespo, and P. Matrella. Sensitivity of reliability-growth models to operational pro~les errors vs testing accuracy. IEEE Transactions on Reliability, R-45(4):531{540, Dec. 1996.
• 19: Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 20: Elaine J. Weyuker , Bingchiang Jeng, Analyzing Partition Testing Strategies, IEEE Transactions on Software Engineering, v.17 n.7, p.703-711, July 1991 [doi>10.1109/32.83906]
• 21: W. E. Wong, J. R. Horgan, S. London, and A. P. Mathur. E~ect of test set size and block coverage on the fault detection e~ectiveness. Technical Report SERC-TR-153-P, SERC, 1994.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

General Terms:
Design, Experimentation, Measurement, Performance, Reliability, Theory

Keywords:
software reliability, software testing