Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects

ABSTRACT

Project managers can use capture-recapture models to manage the inspection process by estimating the number of defects present in an artifact and determining whether a reinspection is necessary. Researchers have previously evaluated capture-recapture models on artifacts with a known number of defects. Before applying capture-recapture models in real development, an evaluation of those models on naturally-occurring defects is imperative. The data in this study is drawn from two inspections of real requirements documents (that later guided implementation) created as part of a capstone course (i.e. with naturally occurring defects). The major results show that: a) estimators improve from being negatively biased after one inspection to being positively biased after two inspections, b) the results contradict the earlier result that a model that includes two sources of variation is a significant improvement over models with one source of variation, and c) estimates are useful in determining the need for artifact reinspection.

REFERENCES

• 1: A. Frank Ackerman , Lynne S. Buchwald , Frank H. Lewski, Software Inspections: An Effective Verification Process, IEEE Software, v.6 n.3, p.31-36, May 1989 [doi>10.1109/52.28121]
• 2: Lionel C. Briand , Khaled El Emam , Bernd G. Freimut , Oliver Laitenberger, A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content, IEEE Transactions on Software Engineering, v.26 n.6, p.518-540, June 2000 [doi>10.1109/32.852741]
• 3: Burnham, K. P. and Overtom, W. S., "Estimation of the Size of a Closed Population When Capture Probabilities Vary among Animals." Biometrics, 1978. 65:625--633.
• 4: Chao, A., "Estimation the population Size for Capture-Recapture Data with Unequal Catchability." Biometrics, 1987. 43(4): 783--791.
• 5: Chao, A., "Estimating Animal Abundance with Capture Frequency Data." Journal of Wildlife Management, 1988. 52(2): 295--300
• 6: Chao, A. and Yeng, H. C., Program CARE-2 (for Capture-Recapture Part.2), http://chao.stat.nthu.edu.tw
• 7: Darroch, J. N., "The Multiple-Recapture Conensus 1: Estimation of a Closed Population." Biometrika, 1958. 45: 343--359.
• 8: Stephen G. Eick , Clive R. Loader , M. David Long , Lawrence G. Votta , Scott Vander Wiel, Estimating software fault content before coding, Proceedings of the 14th international conference on Software engineering, p.59-65, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143090]
• 9: Eick, S., Loader, C., Weil, S. V., and Votta, L. "How Many Errors Remain in a Software Design after Inspection". In Proceedings of the 25th Symposium on the Interface. 1993.
• 10: Khaled El Emam , Oliver Laitenberger , Thomas Harbich, The application of subjective estimates of effectiveness to controlling software inspections, Journal of Systems and Software, v.54 n.2, p.119-136, Oct. 15 2000 [doi>10.1016/S0164-1212(00)00032-7]
• 11: Khaled El Emam , Oliver Laitenberger, Evaluating Capture-Recapture Models with Two Inspectors, IEEE Transactions on Software Engineering, v.27 n.9, p.851-864, September 2001 [doi>10.1109/32.950319]
• 12: Lee, S. M. and Chao, A., "Estimating Population Size via Sample Coverage for Closed Capture-Recapture Models." Biometrics, 1994. 50: 88--97.
• 13: Miller, J., "Estimating the Number of Remaining Defects after Inspection." Software Testing, Verification and Reliability, 1999. 9(3): 167--189.
• 14: John D. Musa , Anthony Iannino , Kazuhira Okumoto, Software reliability: measurement, prediction, application, McGraw-Hill, Inc., New York, NY, 1987
• 15: Otis, D., Burnham, K., White, G., and Anderson, D., "Statistical Inference from Capture Data on Closed Animal Population." Wildlife Monograph, 1978. 64: 1--135.
• 16: Petersson, H., Thelin, T., Runeson, P., Wohlin, C. "Capture-Recapture in Software Inspections after 10 Years Research - Theory, Evaluation, and Application." The Journal of Systems and Software, 72(2):249--264.
• 17: Per Runeson , Claes Wohlin, An Experimental Evaluation of an Experience-Based Capture-RecaptureMethod in Software Code Inspections, Empirical Software Engineering, v.3 n.4, p.381-406, December 1998 [doi>10.1023/A:1009728205264]
• 18: Thelin, T., Petersson, P., and Runeson, P., "Confidence Intervals for Capture-Recapture Estimations in Software Inspections." Journal of Information and Software Technology, 2002. 44(12): 683--702.
• 19: Gursimran S. Walia , Jeffrey Carver , Thomas Philip, Requirement error abstraction and classification: an empirical study, Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, September 21-22, 2006, Rio de Janeiro, Brazil [doi>10.1145/1159733.1159784]
• 20: Gursimran Singh Walia , Jeffrey C. Carver , Nachiappan Nagappan, The effect of the number of inspectors on the defect estimates produced by capture-recapture models, Proceedings of the 30th international conference on Software engineering, May 10-18, 2008, Leipzig, Germany [doi>10.1145/1368088.1368134]
• 21: Scott A. Vander Wiel , Lawrence G. Votta, Assessing Software Designs Using Capture-Recapture Methods, IEEE Transactions on Software Engineering, v.19 n.11, p.1045-1054, November 1993 [doi>10.1109/32.256852]
• 22: White, G. C., Anderson, D. R., Burnham, K. p., and Otis, D. l., Capture-Recapture and Removal Methods for Sampling Closed Populations, Los Alamos National Laboratory, 1982.
• 23: Wohlin, C., Runeson, P., and Brantestam, J., "An Experimental Evaluation of Capture-Recapture in Software Inspections." Software Testing, Verification and Reliability, 1995. 5(4): 213--232.
• 24: Claes Wohlin , Per Runeson, Defect content estimations from review data, Proceedings of the 20th international conference on Software engineering, p.400-409, April 19-25, 1998, Kyoto, Japan
• 25: Yip, P. S. F., "A Martingale Estimating Equation for a Capture-Recapture Experiment in Discrete Time." Biometrics, 1991. 47: 1081--1088.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
      D.2.4 Software/Program Verification
      D.2.8 Metrics
      D.2.9 Management

General Terms:
Documentation, Experimentation, Human Factors, Management, Measurement, Verification

Keywords:
capture-recapture models, defect estimation, empirical study, requirements, software inspections, validation and verification