Where the bugs are

ABSTRACT

The ability to predict which files in a large software system are most likely to contain the largest numbers of faults in the next release can be a very valuable asset. To accomplish this, a negative binomial regression model using information from previous releases has been developed and used to predict the numbers of faults for a large industrial inventory system. The files of each release were sorted in descending order based on the predicted number of faults and then the first 20% of the files were selected. This was done for each of fifteen consecutive releases, representing more than four years of field usage. The predictions were extremely accurate, correctly selecting files that contained between 71% and 92% of the faults, with the overall average being 83%. In addition, the same model was used on data for the same system's releases, but with all fault data prior to integration testing removed. The prediction was again very accurate, ranging from 71% to 93%, with the average being 84%. Predictions were made for a second system, and again the first 20% of files accounted for 83% of the identified faults. Finally, a highly simplified predictor was considered which correctly predicted 73% and 74% of the faults for the two systems.

REFERENCES

• 1: E.N. Adams. Optimizing Preventive Service of Software Products. IBM J. Res. Develop., Vol 28, No 1, Jan 1984, pp.2--14.
• 2: Victor R. Basili , Barry T. Perricone, Software errors and complexity: an empirical investigation0, Communications of the ACM, v.27 n.1, p.42-52, Jan. 1984 [doi>10.1145/69605.2085]
• 3: Norman E. Fenton , Niclas Ohlsson, Quantitative Analysis of Faults and Failures in a Complex Software System, IEEE Transactions on Software Engineering, v.26 n.8, p.797-814, August 2000 [doi>10.1109/32.879815]
• 4: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 5: Les Hatton, Reexamining the Fault Density-Component Size Connection, IEEE Software, v.14 n.2, p.89-97, March 1997 [doi>10.1109/52.582978]
• 6: Taghi M. Khoshgoftaar , Edward B. Allen , Kalai S. Kalaichelvan , Nishith Goel, Early Quality Prediction: A Case Study in Telecommunications, IEEE Software, v.13 n.1, p.65-71, January 1996 [doi>10.1109/52.476287]
• 7: T.J. McCabe. A Complexity Measure. IEEE Trans. on Software Engineering, Vol 2, 1976, pp.308--320.
• 8: P. McCullagh and J.A. Nelder. Generalized Linear Models, Second Edition, Chapman and Hall, London, 1989.
• 9: K-H. Moller and D.J. Paulish. An Empirical Investigation of Software Fault Distribution. Proc. IEEE First International Software Metrics Symposium, Baltimore, Md., May 21-22, 1993, pp.82--90.
• 10: John C. Munson , Taghi M. Khoshgoftaar, The Detection of Fault-Prone Programs, IEEE Transactions on Software Engineering, v.18 n.5, p.423-433, May 1992 [doi>10.1109/32.135775]
• 11: Thomas J. Ostrand , Elaine J. Weyuker, The distribution of faults in a large industrial software system, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 12: T. Ostrand, E.J. Weyuker, and R. Bell. Using Static Analysis to Determine Where to Focus Dynamic Testing Effort. Proc. IEE/Workshop on Dynamic Analysis (WODA04), Edinburgh, May 2004.
• 13: Maurizio Pighin , Anna Marzona, An Empirical Analysis of Fault Persistence Through Software Releases, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.206, September 30-October 01, 2003
• 14: SAS Institute Inc. SAS/STAT User's Guide, Version 8, SAS Institute, Cary, NC, 1999.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Debugging aids

Keywords:
empirical study, fault-prone, prediction, regression model, software faults, software testing