Improving defect prediction using temporal features and non linear models

ABSTRACT

Predicting the defects in the next release of a large software system is a very valuable asset for the project manger to plan her resources. In this paper we argue that temporal features (or aspects) of the data are central to prediction performance. We also argue that the use of non-linear models, as opposed to traditional regression, is necessary to uncover some of the hidden interrelationships between the features and the defects and maintain the accuracy of the prediction in some cases.

Using data obtained from the CVS and Bugzilla repositories of the Eclipse project, we extract a number of temporal features, such as the number of revisions and number of reported issues within the last three months. We then use these data to predict both the location of defects (i.e., the classes in which defects will occur) as well as the number of reported bugs in the next month of the project. To that end we use standard tree-based induction algorithms in comparison with the traditional regression.

Our non-linear models uncover the hidden relationships between features and defects, and present them in easy to understand form. Results also show that using the temporal features our prediction model can predict whether a source file will have a defect with an accuracy of 99% (area under ROC curve 0.9251) and the number of defects with a mean absolute error of 0.019 (Spearman's correlation of 0.96).

REFERENCES

• 1: Mina Askari , Ric Holt, Information theoretic evaluation of change prediction models for large-scale software, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138013]
• 2: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 3: Ahmed E. Hassan , Richard C. Holt, The Top Ten List: Dynamic Fault Prediction, Proceedings of the 21st IEEE International Conference on Software Maintenance, p.263-272, September 25-30, 2005 [doi>10.1109/ICSM.2005.91]
• 4: Hemant Joshi , Chuanlei Zhang , S. Ramaswamy , Coskun Bayrak, Local and Global Recency Weighting Approach to Bug Prediction, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.33, May 20-26, 2007 [doi>10.1109/MSR.2007.17]
• 5: T. M. Khoshgoftaar , E. B. Allen , N. Goel , A. Nandi , J. McMullan, Detection of software modules with high debug code churn in a very large legacy system, Proceedings of the The Seventh International Symposium on Software Reliability Engineering (ISSRE '96), p.364, October 30-November 02, 1996
• 6: C. Kiefer, A. Bernstein, and J. Tappolet. Analyzing software with isparql. In Proceedings of the 3rd International Workshop on Semantic Web Enabled Software Engineering (SWESE 2007). Springer, June 2007. to appear.
• 7: Patrick Knab , Martin Pinzger , Abraham Bernstein, Predicting defect densities in source code files with decision tree learners, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138012]
• 8: Ron Kohavi , George H. John, Wrappers for feature subset selection, Artificial Intelligence, v.97 n.1-2, p.273-324, Dec. 1997 [doi>10.1016/S0004-3702(97)00043-X]
• 9: Audris Mockus , Lawrence G. Votta, Identifying Reasons for Software Changes Using Historic Databases, Proceedings of the International Conference on Software Maintenance (ICSM'00), p.120, October 11-14, 2000
• 10: Nachiappan Nagappan , Thomas Ball, Static analysis tools as early indicators of pre-release defect density, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062558]
• 11: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 12: Foster Provost , Tom Fawcett, Robust Classification for Imprecise Environments, Machine Learning, v.42 n.3, p.203-231, March 2001 [doi>10.1023/A:1007601015854]
• 13: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 14: R. J. Quinlan. Learning with continuous classes. In 5th Australian Joint Conference on Artificial Intelligence, pages 343--348, Singapore, 1992.
• 15: Adrian Schroter, Predicting Defects and Changes with Import Relations, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.31, May 20-26, 2007 [doi>10.1109/MSR.2007.24]
• 16: Jacek Śliwerski , Thomas Zimmermann , Andreas Zeller, HATARI: raising risk awareness, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 17: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 18: Thomas Zimmermann , Rahul Premraj , Andreas Zeller, Predicting Defects for Eclipse, Proceedings of the 29th International Conference on Software Engineering Workshops, p.76, May 20-26, 2007 [doi>10.1109/ICSEW.2007.166]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics

Keywords:
decision tree learner, defect prediction, mining software repository