Information theoretic evaluation of change prediction models for large-scale software

ABSTRACT

In this paper, we analyze the data extracted from several open source software repositories. We observe that the change data follows a Zipf distribution. Based on the extracted data, we then develop three probabilistic models to predict which files will have changes or bugs. The first model is Maximum Likelihood Estimation (MLE), which simply counts the number of events, i.e., changes or bugs, that happen to each file and normalizes the counts to compute a probability distribution. The second model is Reflexive Exponential Decay (RED) in which we postulate that the predictive rate of modification in a file is incremented by any modification to that file and decays exponentially. The third model is called RED-Co-Change. With each modification to a given file, the RED-Co-Change model not only increments its predictive rate, but also increments the rate for other files that are related to the given file through previous co-changes. We then present an information-theoretic approach to evaluate the performance of different prediction models. In this approach, the closeness of model distribution to the actual unknown probability distribution of the system is measured using cross entropy. We evaluate our prediction models empirically using the proposed information-theoretic approach for six large open source systems. Based on this evaluation, we observe that of our three prediction models, the RED-Co-Change model predicts the distribution that is closest to the actual distribution for all the studied systems.

REFERENCES

• 1: Allen, J. F. Using Entropy for Evaluating and Comparing Probability Distributions, available at: http://www.cs.rochester.edu/u/james/CSC248/Lec6.pdf
• 2: Victor R. Basili , Barry T. Perricone, Software errors and complexity: an empirical investigation0, Communications of the ACM, v.27 n.1, p.42-52, Jan. 1984 [doi>10.1145/69605.2085]
• 3: Stephen G. Eick , Todd L. Graves , Alan F. Karr , J. S. Marron , Audris Mockus, Does Code Decay? Assessing the Evidence from Change Management Data, IEEE Transactions on Software Engineering, v.27 n.1, p.1-12, January 2001 [doi>10.1109/32.895984]
• 4: S. G. Eick , T. L. Graves , A. F. Karr , A. Mockus , P. Schuster, Visualizing Software Changes, IEEE Transactions on Software Engineering, v.28 n.4, p.396-412, April 2002 [doi>10.1109/TSE.2002.995435]
• 5: Harald Gall , Karin Hajek , Mehdi Jazayeri, Detection of Logical Coupling Based on Product Release History, Proceedings of the International Conference on Software Maintenance, p.190, March 16-19, 1998
• 6: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 7: Ahmed E. Hassan, Mining software repositories to assist developers and support managers, University of Waterloo, Waterloo, Ont., Canada, 2005
• 8: Ahmed E. Hassan , Richard C. Holt, The Top Ten List: Dynamic Fault Prediction, Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05), p.263-272, September 25-30, 2005 [doi>10.1109/ICSM.2005.91]
• 9: Taghi M. Khoshgoftaar , Edward B. Allen , Robert Halstead , Gary P. Trio , Ronald M. Flass, Using Process History to Predict Software Quality, Computer, v.31 n.4, p.66-72, April 1998 [doi>10.1109/2.666844]
• 10: Khoshgoftaar, T. M., Allen, E. B., Jones, W. D., and Hudepohl, J. P. Data Mining for Predictors of Software Quality. International Journal of Software Engineering and Knowledge Engineering, 9(5), 1999.
• 11: Christopher D. Manning , Hinrich Schütze, Foundations of statistical natural language processing, MIT Press, Cambridge, MA, 1999
• 12: Audris Mockus , Lawrence G. Votta, Identifying Reasons for Software Changes Using Historic Databases, Proceedings of the International Conference on Software Maintenance (ICSM'00), p.120, October 11-14, 2000
• 13: Audris Mockus , David M. Weiss , Ping Zhang, Understanding and predicting effort in software projects, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 14: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 15: Pareto Law: http://www.it-cortex.com/Pareto_law.htm
• 16: Perry, D. E. and Evangelist, W. M. An Empirical Study of Software Interface Faults - An Update. In Proceedings of the 20th Annual Hawaii International Conference on Systems Sciences, pages 113--136, Hawaii, USA, January 1987.
• 17: Dewayne E. Perry , Carol S. Stieg, Software Faults in Evolving a Large, Real-Time System: a Case Study, Proceedings of the 4th European Software Engineering Conference on Software Engineering, p.48-67, September 13-17, 1993
• 18: Reliability Analysis Center, Introduction to Software Reliability: A state of the Art Review. Reliability Analysis Center (RAC), 1996. http://rome.iitri.com/RAC/
• 19: Thomas Zimmermann , Peter Weissgerber , Stephan Diehl , Andreas Zeller, Mining Version Histories to Guide Software Changes, IEEE Transactions on Software Engineering, v.31 n.6, p.429-445, June 2005 [doi>10.1109/TSE.2005.72]
• 20: Zipf, G. K. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Version control

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures; Process metrics

General Terms:
Performance, Reliability, Theory

Keywords:
evaluation approach, information theory, prediction models