Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Quantitative failure estimates for software systems are traditionally made at end of testing using software reliability growth modeling. A persistent problem with most kinds of failure estimation methods and models is the dependency on historical data. This paper presents a method for estimating the total amount of failures possible to provoke from a unit, without historical data dependency. The method combines the results from having several developers testing the same unit with capture-recapture models to create an estimate of 'remaining' number of failures. The evaluation of the approach consists of two steps: first a pre-study where the tools and methods are tested in a large open source project, followed by an add-on to a project at a medium sized software company. The evaluation was a success. An estimate was created, and it can be used both as a quality gatekeeper for units and input to functional and system testing.

REFERENCES

• 1: Anne Chao, 1998. Capture-Recapture Models, Encyclopaedia of Biostatistics, Editors: Armitage & Colton, Wiley, New York.
• 2: Harlan D. Mills, 1972. On the Statistical Validation of Computer Programs, Technical report FSC-72-6015, IBM Federal Systems Division.
• 3: Stephen G. Eick , Clive R. Loader , M. David Long , Lawrence G. Votta , Scott Vander Wiel, Estimating software fault content before coding, Proceedings of the 14th international conference on Software engineering, p.59-65, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143090]
• 4: Håkan Petersson, Thomas Thelin, Per Runeson, and Claes Wohlin, 2004, Capture-recapture in software inspections after 10 years research-theory, evaluation and application, Journal of Systems and Software 72,2 (July, 2004), pp. 249--264.
• 5: Catherine Stringfellow, Anneliese Andrews, Claes Wohlin and Håkan Petersson, Estimating the number of components with defects post-release that showed no defects in testing, Software Testing, Verification and Reliability 12,2 (2002) pp. 93--122.
• 6: Mark C. K. Yang and Anne Chao, Reliability-estimation and stopping-rules for software testing, based on repeated appearances of bugs, IEEE Transactions on Reliability 44,2 (1995), pp. 315--321.
• 7: Thomas Thelin, Team-Based Fault Content Estimation in the Software Inspection Process, Proceedings of the 26th International Conference on Software Engineering, p.263-272, May 23-28, 2004
• 8: Tron Foss , Erik Stensrud , Barbara Kitchenham , Ingunn Myrtveit, A Simulation Study of the Model Evaluation Criterion MMRE, IEEE Transactions on Software Engineering, v.29 n.11, p.985-995, November 2003 [doi>10.1109/TSE.2003.1245300]
• 9: Amr Kamel , Paul G. Sorenson, The Application of Capture-Recapture Log-Linear Models To Software Inspections Data, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.213, September 30-October 01, 2003
• 10: Lionel C. Briand , Khaled El Emam , Bernd G. Freimut , Oliver Laitenberger, A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content, IEEE Transactions on Software Engineering, v.26 n.6, p.518-540, June 2000 [doi>10.1109/32.852741]
• 11: Håkan Petersson , Claes Wohlin, An Empirical Study of Experience-Based Software Defect Content Estimation Methods, Proceedings of the 10th International Symposium on Software Reliability Engineering, p.126, November 01-04, 1999
• 12: Ching-Pao Chang , Jia-Lyn Lv , Chih-Ping Chu, A Defect Estimation Approach for Sequential Inspection Using a Modified Capture-Recapture Model, Proceedings of the 29th Annual International Computer Software and Applications Conference (COMPSAC'05) Volume 1, p.41-46, July 26-28, 2005
• 13: Thomas Thelin and Per Runeson, Confidence intervals for capture-recapture estimations in software inspections, Information and Software Technology 44,12 (Sept. 2002) pp. 683--702.
• 14: Lionel C. Briand , John W. Daly , Jürgen K. Wüst, A Unified Framework for Coupling Measurement in Object-Oriented Systems, IEEE Transactions on Software Engineering, v.25 n.1, p.91-121, January 1999 [doi>10.1109/32.748920]
• 15: Norman E. Fenton , Martin Neil, Software metrics: roadmap, Proceedings of the Conference on The Future of Software Engineering, p.357-370, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336588]
• 16: Dimitri van Heesch, Doxygen, http://www.stack.nl/~dimitri/doxygen/ 2008-02-29.
• 17: Dokbook, http://www.docbook.org/ 2008-02-29.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Statistical methods

General Terms:
Measurement, Reliability

Keywords:
capture-recapture, failures, faults, prediction, unit test