Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc.

ABSTRACT

Quantitatively-based risk management can reduce the risks associated with field defects for both software producers and software consumers. In this paper, we report experiences and results from initiating risk-management activities at a large systems development organization. The initiated activities aim to improve product testing (system/integration testing), to improve maintenance resource allocation, and to plan for future process improvements. The experiences we report address practical issues not commonly addressed in research studies: how to select an appropriate modeling method for product testing prioritization and process improvement planning, how to evaluate accuracy of predictions across multiple releases in time, and how to conduct analysis with incomplete information. In addition, we report initial empirical results for two systems with 13 and 15 releases. We present prioritization of configurations to guide product testing, field defect predictions within the first year of deployment to aid maintenance resource allocation, and important predictors across both systems to guide process improvement planning. Our results and experiences are steps towards quantitatively-based risk management.

REFERENCES

• 1: Norman Fenton , Shari Lawrence Pfleeger, Software metrics (2nd ed.): a rigorous and practical approach, PWS Publishing Co., Boston, MA, 1997
• 2: Norman E. Fenton , Martin Neil, Software metrics: roadmap, Proceedings of the Conference on The Future of Software Engineering, p.357-370, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336588]
• 3: Donald E. Harter , Mayuram S. Krishnan , Sandra A. Slaughter, Effects of Process Maturity on Quality, Cycle Time, and Effort in Software Product Development, Management Science, v.46 n.4, p.451-466, April 2000 [doi>10.1287/mnsc.46.4.451.12056]
• 4: Myles Hollander and Douglas A. Wolfe. Nonparametric statistical inference. New York, NY: Wiley & Sons, 1973.
• 5: Wendell D. Jones , John P. Hudepohl , Taghi M. Khoshgoftaar , Edward B. Allen, Application of a Usage Profile in Software Quality Models, Proceedings of the Third European Conference on Software Maintenance and Reengineering, p.148, March 03-05, 1999
• 6: IEEE standard for a software quality metrics methodology. IEEE Std 1061-1998, Dec 1998.
• 7: Garrison Kenny. Estimating Defects in Commercial Software during Operational Use. IEEE Tr. on Reliability, vol. 42 no. 1, Mar 1993, pp. 107--115.
• 8: Taghi M. Khoshgoftaar , Edward B. Allen, Predicting Fault-Prone Software Modules in Embedded Systems with Classification Trees, The 4th IEEE International Symposium on High-Assurance Systems Engineering, p.105, November 17-19, 1999
• 9: Taghi M. Khoshgoftaar , Edward B. Allen , Jianyu Deng, Controlling Overfitting in Software Quality Models: Experiments with Regression Trees and Classification, Proceedings of the 7th International Symposium on Software Metrics, p.190, April 04-06, 2001
• 10: Taghi M. Khoshgoftaar and Edward B. Allen and John P. Hudepohl and Stephen J. Aud. Application of Neural Networks to Software Quality Modeling of a Very Large Telecommunications System. IEEE Tr. on Neural Networks, vol. 8 no. 4, Jul 1997, pp. 902--909.
• 11: Taghi Khoshgoftaar, Bibhuti Bhattacharyya, and Gary Richardson. Predicting Software Errors, During Development, Using Nonlinear Regression Models: A Comparative Study. IEEE Tr. On Reliability, vol. 41 no. 3, Sep 1992, pp. 390--395.
• 12: Taghi Khoshgoftaar and John Munson. Predicting Software Development Errors using Software Complexity Metrics. IEEE J. Selected Areas in Communications, vol. 8 no. 2, Feb 1990, pp. 253--261.
• 13: Taghi M. Khoshgoftaar , John C. Munson , David L. Lanning, A Comparative Study of Predictive Models for Program Changes During System Testing and Maintenance, Proceedings of the Conference on Software Maintenance, p.72-79, September 01, 1993
• 14: Taghi Khoshgoftaar, Abhijit Pandya, and David Lanning. Application of Neural Networks for Predicting Program Fault. Annals of Software Engineering, vol. 1, 1995, pp. 141--154.
• 15: Taghi Khoshgoftaar, Abhijit Pandya, and Hamant More. A Neural Networks Approach for Predicting Software Development Faults. Proc. ISSRE, Oct 1992, pp. 83--89.
• 16: Taghi M. Khoshgoftaar , Naeem Seliya, Tree-Based Software Quality Estimation Models For Fault Prediction, Proceedings of the 8th International Symposium on Software Metrics, p.203, June 04-07, 2002
• 17: Taghi M. Khoshgoftaar , Vishal Thaker , Edward B. Allen, Modeling Fault-Prone Modules of Subsystems, Proceedings of the 11th International Symposium on Software Reliability Engineering (ISSRE'00), p.259, October 08-11, 2000
• 18: Paul Luo Li , Mary Shaw , Jim Herbsleb , Bonnie Ray , P. Santhanam, Empirical evaluation of defect projection models for widely-deployed production software systems, ACM SIGSOFT Software Engineering Notes, v.29 n.6, November 2004
• 19: Paul Luo Li, Mary Shaw, Jim Herbsleb, Bonnie Ray, and P. Santhanam. An Empirical Comparison of Field Defect Modeling Methods, CMU Tech Report CMU-ISRI-06-102, 2006
• 20: Paul Luo Li , Jim Herbsleb , Mary Shaw, Finding Predictors of Field Defects for Open Source Software Systems in Commonly Available Data Sources: A Case Study of OpenBSD, Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS'05), p.32, September 19-22, 2005 [doi>10.1109/METRICS.2005.26]
• 21: Paul Luo Li, Jim Herbsleb, and Mary Shaw. ForecastingField Defects Using a Combined Time-based and Metrics-based Approach: a Case Study of OpenBSD. CMUTechReport, CMU-ISRI-05-125, 2005.
• 22: Zhaohui Liu, Nalini Ravishanker, and Bonnie Ray. Modeling Dynamic Reliability Growth Using Bayesian Methods. Reliability Review, vol. 23 no. 1, Mar 2003, pp. 5--9.
• 23: Michael R. Lyu, Handbook of software reliability and system reliability, McGraw-Hill, Inc., Hightstown, NJ, 1996
• 24: Microsoft Visual SourceSafe. msdn.microsoft.com/ssafe/
• 25: Audris Mockus , David M. Weiss , Ping Zhang, Understanding and predicting effort in software projects, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 26: Audris Mockus , Ping Zhang , Paul Luo Li, Predictors of customer perceived software quality, Proceedings of the 27th international conference on Software engineering, p.225-233, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062506]
• 27: J. C. Munson , T. M. Khoshgoftaar, The dimensionality of program complexity, Proceedings of the 11th international conference on Software engineering, p.245-253, May 1989, Pittsburgh, Pennsylvania, United States [doi>10.1145/74587.74620]
• 28: John Musa and Anthony Iannino and Kazuhira Okumoto. Software Reliability. McGraw-Hill, 1990.
• 29: National Institute of Standards and Technology. The economic impacts of inadequate infrastructure for software testing. Planning Report 02-3, Jun 2002
• 30: Magnus C. Ohlsson , Per Runeson, Experience from Replicating Empirical Studies on Prediction Models, Proceedings of the 8th International Symposium on Software Metrics, p.217, June 04-07, 2002
• 31: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Where the bugs are, ACM SIGSOFT Software Engineering Notes, v.29 n.4, July 2004
• 32: The R project for statistical computing. www.r-project.org
• 33: R. W. Selby , A. A. Porter, Learning from Examples: Generation and Evaluation of Decision Trees for Software Resource Analysis, IEEE Transactions on Software Engineering, v.14 n.12, p.1743-1757, December 1988 [doi>10.1109/32.9061]
• 34: Norman F. Schneidewind, Body of Knowledge for Software Quality Measurement, Computer, v.35 n.2, p.77-83, February 2002 [doi>10.1109/2.982919]
• 35: Martin Shepperd, Evaluating Software Project Prediction Systems, Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS'05), p.2, September 19-22, 2005 [doi>10.1109/METRICS.2005.22]
• 36: Tracker. www.serena.com/Products/professional/tracker/
• 37: Understand for C++. www.scitools.com/metrics.txt
• 38: Sanford Weisberg. Applied Linear Regression, 2nd Edition. New York, NY: Wiley and Son, 1985.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Complexity measures

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Product metrics; Process metrics
      D.2.9 Management
          Subjects: Cost estimation

General Terms:
Economics, Management, Measurement, Reliability

Keywords:
deployment and usage metrics, software and hardware configuration metrics, software reliability modeling, system test prioritization