How to measure success of fault prediction models

ABSTRACT

Many fault prediction models have been proposed in the software engineering literature, and their success evaluated according to various metrics that are widely used in the statistics community. To be able to make meaningful comparisons among the proposed models, it is important that the metrics assess meaningful properties of the predictions. We examine several of the more common metrics, discuss the advantages and disadvantages of each, and illustrate their application to predictions made on a large industrial system. We conclude that the most useful metrics are the percentage of faults that occur in the predicted most fault-prone files, and the Type II misclassification rate.

REFERENCES

• 1: Erik Arisholm , Lionel C. Briand, Predicting fault-prone components in a java legacy system, Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, September 21-22, 2006, Rio de Janeiro, Brazil [doi>10.1145/1159733.1159738]
• 2: Robert M. Bell , Thomas J. Ostrand , Elaine J. Weyuker, Looking for bugs in all the right places, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146246]
• 3: Giovanni Denaro , Mauro Pezzè, An empirical evaluation of fault-proneness models, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581371]
• 4: Taghi M. Khoshgoftaar , Edward B. Allen , Kalai S. Kalaichelvan , Nishith Goel, Early Quality Prediction: A Case Study in Telecommunications, IEEE Software, v.13 n.1, p.65-71, January 1996 [doi>10.1109/52.476287]
• 5: T. M. Khoshgoftaar, E. B. Allen, J. Deng. Using Regression Trees to Classify Fault-Prone Software Modules. IEEE Trans. on Reliability, Vol 51, No. 4, Dec 2002, pp. 455--462.
• 6: P. McCullagh and J. A. Nelder. Generalized Linear Models, Second Edition Chapman and Hall, London, 1989.
• 7: A. Mockus and D. M. Weiss. Predicting Risk of Software Changes. Bell Labs Technical Journal, April-June 2000, pp. 169--180.
• 8: Thomas J. Ostrand , Elaine J. Weyuker, The distribution of faults in a large industrial software system, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 9: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 10: SAS Institute Inc. SAS/STAT User's Guide, Version 8, SAS Institute, Cary, NC, 1999.
• 11: Adrian Schröter , Thomas Zimmermann , Andreas Zeller, Predicting component failures at design time, Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, September 21-22, 2006, Rio de Janeiro, Brazil [doi>10.1145/1159733.1159739]
• 12: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Automating algorithms for the identification of fault-prone files, Proceedings of the 2007 international symposium on Software testing and analysis, July 09-12, 2007, London, United Kingdom [doi>10.1145/1273463.1273493]