Understanding the sources of variation in software inspections

ABSTRACT

In a previous experiment, we determined how various changes in three structural elements of the software inspection process (team size and the number and sequencing of sessions) altered effectiveness and interval. Our results showed that such changes did not significantly influence the defect detection rate, but that certain combinations of changes dramatically increased the inspection interval. We also observed a large amount of unexplained variance in the data, indicating that other factors must be affecting inspection performance. The nature and extent of these other factors now have to be determined to ensure that they had not biased our earlier results. Also, identifying these other factors might suggest additional ways to improve the efficiency of inspections. Acting on the hypothesis that the “inputs” into the inspection process (reviewers, authors, and code units) were significant sources of variation, we modeled their effects on inspection performance. We found that they were responsible for much more variation in detect detection than was process structure. This leads us to conclude that better defect detection techniques, not better process structures, are the key to improving inspection effectiveness. The combined effects of process inputs and process structure on the inspection interval accounted for only a small percentage of the variance in inspection interval. Therefore, there must be other factors which need to be identified.

REFERENCES

• 1: BASILI, V.R. AND MILLS, H.D. 1982. Understanding and documenting programs. IEEE Trans. Softw. Eng. SE-8, 3 (May), 270-283.
• 2: BASILI, V. R. AND WEISS, D.M. 1984. A methodology for collecting valid software engineering data. IEEE Trans. Softw. Eng. SE-IO, 6 (Nov.), 728-738.
• 3: BECKER, R. A., CHAMBERS, J. M., AND WILKS, A.R. 1988. The New S Language. Wadsworth and Brooks/Cole.
• 4: David B. Bisant , James R. Lyle, A Two-Person Inspection Method to Improve Programming Productivity, IEEE Transactions on Software Engineering, v.15 n.10, p.1294-1304, October 1989 [doi>10.1109/TSE.1989.559782]
• 5: Box, G.E., HUNTER, W.G., AND HUNTER, J.S. 1978. Statistics for Experimenters. John Wiley and Sons, New York.
• 6: John M. Chambers , Trevor J. Hastie, Statistical Models in S, CRC Press, Inc., Boca Raton, FL, 1991
• 7: CHAMBERS, J. M., CLEVELAND, W. S., KLEINER, B., AND TUCKEY, P.A. 1983. Graphical Methods for Data Analysis. Chapman and Hall.
• 8: CHATFIELD, C. 1995. Model uncertainty, data mining and statistical inference. J. Roy. Stat. Soc. 158, 3. Series A.
• 9: CURTIS, B. 1981. Substantiating programmer variability. Proc. IEEE 69, 7 (July), 846.
• 10: DENNIS, A. R. AND VALACICH, J.S. 1993. Computer brainstorms: More heads are better than one. J. Appl. Psychol. 78, 4 (Apr.), 531-537.
• 11: Stephen G. Eick , Clive R. Loader , M. David Long , Lawrence G. Votta , Scott Vander Wiel, Estimating software fault content before coding, Proceedings of the 14th international conference on Software engineering, p.59-65, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143090]
• 12: FAGAN, M.E. 1976. Design and code inspections to reduce errors in program development. IBM Syst. J. 15, 3, 182-211.
• 13: Philip M. Johnson, An instrumented approach to improving software quality through formal technical review, Proceedings of the 16th international conference on Software engineering, p.113-122, May 16-21, 1994, Sorrento, Italy
• 14: KENNY, D.A. 1979. Correlation and Causality. John Wiley and Sons, New York.
• 15: John C. Knight , E. Ann Myers, An improved inspection technique, Communications of the ACM, v.36 n.11, p.51-61, Nov. 1993 [doi>10.1145/163359.163366]
• 16: KYLE, B. 1995. Successful Industrial Experimentation. VCH Publishers, Chapt. 5.
• 17: LADD, D.A. AND RAMMING, J.C. 1992. Software research and switch software. In the International Conference on Communications Technology (Beijing, China.)
• 18: LADD, D. A. AND RAMMING, J.C. 1994. Two application languages in software production. In the USENIX Symposium on Very High Level Languages. USENIX Assoc., Berkeley, Calif.
• 19: MCCULLAGH, P. AND NELDER, J.A. 1989. Generalized Linear Models. 2nd ed. Chapman and Hall.
• 20: J. F. Nunamaker , Alan R. Dennis , Joseph S. Valacich , Douglas Vogel , Joey F. George, Electronic meeting systems, Communications of the ACM, v.34 n.7, p.40-61, July 1991 [doi>10.1145/105783.105793]
• 21: David L. Parnas , David M. Weiss, Active design reviews: principles and practices, Proceedings of the 8th international conference on Software engineering, p.132-136, August 28-30, 1985, London, England
• 22: Lawrence G. Votta , Adam Porter, Experimental software engineering: a report on the state of the art, Proceedings of the 17th international conference on Software engineering, p.277-279, April 24-28, 1995, Seattle, Washington, United States [doi>10.1145/225014.225040]
• 23: PERRY, D. E., STAUDENMAYER, N. A., AND VOTTA, L.G. 1995b. Understanding and improving time usage in software development. In Trends in Software. Vol. 5, Software Process, A. Wolf and A. Fuggetta, Eds. John Wiley and Sons, New York.
• 24: Adam A. Porter , Lawrence G. Votta, Jr. , Victor R. Basili, Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment, IEEE Transactions on Software Engineering, v.21 n.6, p.563-575, June 1995 [doi>10.1109/32.391380]
• 25: A. Porter , H. Siy , C. A. Toman , L. G. Votta, An experiment to assess the cost-benefits of code inspections in large scale software development, Proceedings of the 3rd ACM SIGSOFT symposium on Foundations of software engineering, p.92-103, October 12-15, 1995, Washington, D.C., United States
• 26: G. Michael Schneider , Johnny Martin , W. T. Tsai, An experimental study of fault detection in user requirements documents, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.2, p.188-204, April 1992 [doi>10.1145/128894.128897]
• 27: Scott A. Vander Wiel , Lawrence G. Votta, Assessing Software Designs Using Capture-Recapture Methods, IEEE Transactions on Software Engineering, v.19 n.11, p.1045-1054, November 1993 [doi>10.1109/32.256852]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Code inspections and walk-throughs

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software process models (e.g., CMM, ISO, PSP)

General Terms:
Design, Experimentation, Measurement, Performance, Theory

Keywords:
empirical studies, software inspection, software process, statistical models