A controlled experiment in program testing and code walkthroughs/inspections

ABSTRACT

This paper describes an experiment in program testing, employing 59 highly experienced data processing professionals using seven methods to test a small PL/I program. The results show that the popular code walkthrough/inspection method was as effective as other computer-based methods in finding errors and that the most effective methods (in terms of errors found and cost) employed pairs of subjects who tested the program independently and then pooled their findings. The study also shows that there is a tremendous amount of variability among subjects and that the ability to detect certain types of errors varies from method to method.

REFERENCES

• 1: Fagan, M.E. Design and code inspections to reduce errors in program development. IBM Syst. J. 15, 3 (1976), 182-211.
• 2: Goodenough, J.B., and Gerhart, S.L. Toward a theory of test data selection. 1EEE Trans. Software Eng. SE-1, 2 (1975), 156-173.
• 3: Gould, J.D. Some psychological evidence on how people debug computer programs. Int. J. Man-Machine Studies 7, 2 (1975), 151-182.
• 4: Gould, J.D., and Drongowski, P. An exploratory study of computer program debugging. Human Factors 16, 3 (1974), 258-277.
• 5: An investigatory study into human problem solving capabilities as they relate to programmer efficiency, ACM SIGCPR Computer Personnel, v.3 n.3, p.10-14, Fall 1972 [doi>10.1145/1036508.1036511]
• 6: William C. Hetzel, An experimental analysis of program verification methods., 1976
• 7: Howden, W.E. Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans. Software Eng. SE-3, 4 (1977), 266-278.
• 8: Jelinski, Z., and Moranda, P.B. Applications of a probabilitybased model to a code reading experiment. Rec. 1973 IEEE Symp. Comptr. Software Reliability, IEEE, New York, 1973, pp. 78-81.
• 9: Glenford J. Myers, Softwear Reliability, John Wiley & Sons, Inc., New York, NY, 1976
• 10: Naur, P. Programming by action clusters. BIT 9, 3 (1969), 250-258.
• 11: Shneiderman, B. Experimental testing in programming languages, stylistic considerations and design techniques. Proc. AFIPS 1975 NCC, AFIPS Press, Montvale, N.J., 1975, pp. 653--656.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Code inspections and walk-throughs

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

  H. Information Systems
  H.1 MODELS AND PRINCIPLES

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software development

General Terms:
Design, Experimentation, Human Factors, Management, Performance, Reliability, Theory

Keywords:
code inspections, code walkthroughs, debugging, personnel selection, program verification, software reliability, testing