The effect of the number of inspectors on the defect estimates produced by capture-recapture models

ABSTRACT

Inspections can be made more cost-effective by using capture-recapture methods to estimate post-inspection defects. Previous capture-recapture studies of inspections used relatively small data sets compared with those used in biology and wildlife research (the origin of the models). A common belief is that capture-recapture models underestimate the number of defects but their performance can be improved with data from more inspectors. This increase has not been evaluated in detail. This paper evaluates new estimators from biology not been previously applied to inspections. Using a data from seventy-three inspectors, we analyze the effect of the number of inspectors on the quality of estimates. Contrary to previous findings indicating that Jackknife is the best estimator, our results show that the SC estimators are better suited to software inspections. Our results also provide a detailed analysis of the number of inspectors necessary to obtain estimates within 5% to 20% of the actual.

REFERENCES

• 1: A. Frank Ackerman , Lynne S. Buchwald , Frank H. Lewski, Software Inspections: An Effective Verification Process, IEEE Software, v.6 n.3, p.31-36, May 1989 [doi>10.1109/52.28121]
• 2: Lionel C. Briand , Khaled El Emam , Bernd G. Freimut, A Comparison and Integration of Capture-Recapture Models and the Detection Profile Method, Proceedings of the The Ninth International Symposium on Software Reliability Engineering, p.32, November 04-07, 1998
• 3: Lionel C. Briand , Khaled El Emam , Bernd G. Freimut , Oliver Laitenberger, A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content, IEEE Transactions on Software Engineering, v.26 n.6, p.518-540, June 2000 [doi>10.1109/32.852741]
• 4: Burnham, K.P. and Overtom, W.S., "Estimation of the Size of a Closed Population When Capture Probabilities Vary Among Animals." Biometrica, 1978. 65: 625--633.
• 5: Chao, A., "Estimation the population Size for Capture-Recapture Data with Unequal Catchability." Biometrics, 1987. 43(4): 783--791.
• 6: Chao, A., "Estimating Animal Abundance with Capture Frequency Data." Journal of Wildlife Management, 1988. 52(2): 295--300.
• 7: Chao, A. and Yeng, H.C., Program CARE-2 (for Capture-Recapture Part.2), http://chao.stat.nthu.edu.tw, 2003.
• 8: Darroch, J.N., "The Multiple-Recapture Conensus 1: Estimation of a Closed Population." Biometrika, 1958. 45: 343--359.
• 9: Nader B. Ebrahimi, On the Statistical Analysis of the Number of Errors Remaining in a Software Design Document after Inspection, IEEE Transactions on Software Engineering, v.23 n.8, p.529-532, August 1997 [doi>10.1109/32.624308]
• 10: Stephen G. Eick , Clive R. Loader , M. David Long , Lawrence G. Votta , Scott Vander Wiel, Estimating software fault content before coding, Proceedings of the 14th international conference on Software engineering, p.59-65, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143090]
• 11: Eick, S., Loader, C., Weil, S.V., and Votta, L. "How Many Errors Remain in a Software Design after Inspection". In Proceedings of the 25th Symposium on the Interface. 1993:
• 12: Khaled El Emam , Oliver Laitenberger , Thomas Harbich, The application of subjective estimates of effectiveness to controlling software inspections, Journal of Systems and Software, v.54 n.2, p.119-136, Oct. 15 2000 [doi>10.1016/S0164-1212(00)00032-7]
• 13: Khaled El Emam , Oliver Laitenberger, Evaluating Capture-Recapture Models with Two Inspectors, IEEE Transactions on Software Engineering, v.27 n.9, p.851-864, September 2001 [doi>10.1109/32.950319]
• 14: Lee, S.M. and Chao, A., "Estimating Population Size via Sample Coverage for Closed Capture-Recapture Models." Biometrics, 1994. 50: 88--97.
• 15: Miller, J., "Estimating the Number of Remaining Defects after Inspection." Software Testing, Verification and Reliability, 1999. 9(3): 167--189.
• 16: Otis, D., Burnham, K., White, G., and Anderson, D., "Statistical Inference from Capture Data on Closed Animal Population." Wildlife Monograph, 1978. 64: 1--135.
• 17: Petersson, H., Thelin, T., Runeson, P., and Wohlin, C., "Capture-Recapture in Software Inspections after 10 Years Research - Theory, Evaluation and Application." Journal of Systems and Software, 2003.
• 18: Per Runeson , Claes Wohlin, An Experimental Evaluation of an Experience-Based Capture-RecaptureMethod in Software Code Inspections, Empirical Software Engineering, v.3 n.4, p.381-406, December 1998 [doi>10.1023/A:1009728205264]
• 19: Forrest Shull , Jeffrey Carver , Guilherme H. Travassos, An empirical methodology for introducing software processes, Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering, September 10-14, 2001, Vienna, Austria
• 20: Thelin, T. and Runeson, P. "Capture-Recapture Estimators for perspective Based Reading - A Simulating Experiment". In Proceedings of the International Conference on Product Focused Software Process Improvement. 1999:
• 21: Thelin, T., Petersson, P., and Runeson, P., "Confidence Intervals for Capture-Recapture Estimations in Software Inspections." Journal of Information and Software Technology, 2002. 44(12): 683--702.
• 22: Scott A. Vander Wiel , Lawrence G. Votta, Assessing Software Designs Using Capture-Recapture Methods, IEEE Transactions on Software Engineering, v.19 n.11, p.1045-1054, November 1993 [doi>10.1109/32.256852]
• 23: White, G.C., Anderson, D.R., Burnham, K.p., and Otis, D.l., Capture-Recapture and Removal Methods for Sampling Closed Populations, Los Alomos National Laboratory, 1982.
• 24: Wohlin, C., Runeson, P., and Brantestam, J., "An Experimental Evaluation of Capture-Recapture in Software Inspections." Software Testing, Verification and Reliability, 1995. 5(4): 213--232.
• 25: Claes Wohlin , Per Runeson, Defect content estimations from review data, Proceedings of the 20th international conference on Software engineering, p.400-409, April 19-25, 1998, Kyoto, Japan
• 26: Yip, P.S.F., "A Martingale Estimating Equation for a Capture-Recapture Experiment in Discrete Time." Biometrics, 1991. 47: 1081--1088.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification

General Terms:
Documentation, Experimentation, Human Factors, Management, Measurement, Verification

Keywords:
capture-recapture models, inspections, requirements, validation and verification