Looking for bugs in all the right places

ABSTRACT

We continue investigating the use of a negative binomial regression model to predict which files in a large industrial software system are most likely to contain many faults in the next release. A new empirical study is described whose subject is an automated voice response system. Not only is this system's functionality substantially different from that of the earlier systems we studied (an inventory system and a service provisioning system), it also uses a significantly different software development process. Instead of having regularly scheduled releases as both of the earlier systems did, this system has what are referred to as "continuous releases." We explore the use of three versions of the negative binomial regression model, as well as a simple lines-of-code based model, to make predictions for this system and discuss the differences observed from the earlier studies. Despite the different development process, the best version of the prediction model was able to identify, over the lifetime of the project, 20% of the system's files that contained, on average, nearly three quarters of the faults that were detected in the system's next releases.

REFERENCES

• 1: E.N. Adams. Optimizing Preventive Service of Software Products. IBM J. Res. Develop., Vol 28, No 1, Jan 1984, pp. 2--14.
• 2: Victor R. Basili , Barry T. Perricone, Software errors and complexity: an empirical investigation0, Communications of the ACM, v.27 n.1, p.42-52, Jan. 1984 [doi>10.1145/69605.2085]
• 3: Giovanni Denaro , Mauro Pezzè, An empirical evaluation of fault-proneness models, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581371]
• 4: Stephen G. Eick , Todd L. Graves , Alan F. Karr , J. S. Marron , Audris Mockus, Does Code Decay? Assessing the Evidence from Change Management Data, IEEE Transactions on Software Engineering, v.27 n.1, p.1-12, January 2001 [doi>10.1109/32.895984]
• 5: Norman E. Fenton , Niclas Ohlsson, Quantitative Analysis of Faults and Failures in a Complex Software System, IEEE Transactions on Software Engineering, v.26 n.8, p.797-814, August 2000 [doi>10.1109/32.879815]
• 6: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 7: Lan Guo , Yan Ma , Bojan Cukic , Harshinder Singh, Robust Prediction of Fault-Proneness by Random Forests, Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE'04), p.417-428, November 02-05, 2004 [doi>10.1109/ISSRE.2004.35]
• 8: Les Hatton, Reexamining the Fault Density-Component Size Connection, IEEE Software, v.14 n.2, p.89-97, March 1997 [doi>10.1109/52.582978]
• 9: Taghi M. Khoshgoftaar , Edward B. Allen , Kalai S. Kalaichelvan , Nishith Goel, Early Quality Prediction: A Case Study in Telecommunications, IEEE Software, v.13 n.1, p.65-71, January 1996 [doi>10.1109/52.476287]
• 10: T.M. Khoshgoftaar, E.B. Allen, J. Deng. Using Regression Trees to Classify Fault-Prone Software Modules. IEEE Trans. on Reliability, Vol 51, No. 4, Dec 2002, pp. 455--462.
• 11: P. McCullagh and J.A. Nelder. Generalized Linear Models, Second Edition, Chapman and Hall, London, 1989.
• 12: A. Mockus and D.M. Weiss. Predicting Risk of Software Changes. Bell Labs Technical Journal, April-June 2000, pp. 169--180.
• 13: K H. Moller and D.J. Paulish. An Empirical Investigation of Software Fault Distribution. Proc. IEEE First International Software Metrics Symposium, Baltimore, Md., May 21-22, 1993, pp. 82--90.
• 14: John C. Munson , Taghi M. Khoshgoftaar, The Detection of Fault-Prone Programs, IEEE Transactions on Software Engineering, v.18 n.5, p.423-433, May 1992 [doi>10.1109/32.135775]
• 15: Niclas Ohlsson , Hans Alberg, Predicting Fault-Prone Software Modules in Telephone Switches, IEEE Transactions on Software Engineering, v.22 n.12, p.886-894, December 1996 [doi>10.1109/32.553637]
• 16: Thomas J. Ostrand , Elaine J. Weyuker, The distribution of faults in a large industrial software system, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 17: T. Ostrand, E.J. Weyuker, and R.M. Bell. Using Static Analysis to Determine Where to Focus Dynamic Testing Effort. Proc. IEE/Workshop Dynamic Analysis (WODA 04), May 2004.
• 18: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Where the bugs are, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 19: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 20: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell , Rachel C. W. Ostrand, A Different View of Fault Prediction, Proceedings of the 29th Annual International Computer Software and Applications Conference (COMPSAC'05) Volume 2, p.3-4, July 26-28, 2005
• 21: Maurizio Pighin , Anna Marzona, An Empirical Analysis of Fault Persistence Through Software Releases, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.206, September 30-October 01, 2003
• 22: SAS Institute Inc. SAS/STAT 9.1 User's Guide, SAS Institute, Cary, NC, 2004.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Debugging aids

General Terms:
Experimentation

Keywords:
empirical study, fault-prone, prediction, regression model, software faults, software testing