Quantitative analysis of faults and failures with multiple releases of softpm

ABSTRACT

To date, very little published empirical data has reported on the quality and reliability aspects of commercial software systems. In this paper, we present quantitative empirical study results on faults and failures with four releases of SoftPM [29, 31], one of the most widely adopted software project management tools in China. Our approach verifies Fenton's Hypotheses in [11] and explores the relationship between pre-release faults and post-release failures in four releases of SoftPM. We also present the distribution of Mean Time to Remove Faults (MTRF), the review and testing efficiency, and the fault and failure data for benchmarking. Our study shows that a few modules that are not fault-prone contain most failures in the field and most faults detected at an early stage cost less time to be fixed and removed. Our results have been validated and evaluated at four subsequent major releases of SoftPM at the laboratory for Internet Software Technologies of the Institute of Software, Chinese Academy of Sciences (iTechs).

REFERENCES

• 1: http://www.scitools.com.
• 2: F. AKIYAMA. An example of software system debugging. In Proceedings of the International Federation of Information Processing Societies Congress, pages 353--359. North-Holland Publishing Company, 1971.
• 3: Sousuke Amasaki , Yasunari Takagi , Osamu Mizuno , Tohru Kikuno, A Bayesian Belief Network for Assessing the Likelihood of Fault Content, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.215, November 17-21, 2003
• 4: Victor R. Basili , Lionel C. Briand , Walcélio L. Melo, A Validation of Object-Oriented Design Metrics as Quality Indicators, IEEE Transactions on Software Engineering, v.22 n.10, p.751-761, October 1996 [doi>10.1109/32.544352]
• 5: S. Biyani , P. Santhanam, Exploring Defect Data from Development and Customer Usage on Software Modules over Multiple Releases, Proceedings of the The Ninth International Symposium on Software Reliability Engineering, p.316, November 04-07, 1998
• 6: Barry Boehm , Victor R. Basili, Software Defect Reduction Top 10 List, Computer, v.34 n.1, p.135-137, January 2001 [doi>10.1109/2.962984]
• 7: B. T. Compton , C. Withrow, Prediction and control of ADA software defects, Journal of Systems and Software, v.12 n.3, p.199-207, Jul. 1990 [doi>10.1016/0164-1212(90)90040-S]
• 8: Barry W. Boehm , Clark , Horowitz , Brown , Reifer , Chulani , Ray Madachy , Bert Steece, Software Cost Estimation with Cocomo II with Cdrom, Prentice Hall PTR, Upper Saddle River, NJ, 2000
• 9: Chin-Feng Fan , Yuan-Chang Yu, BBN-based software project risk management, Journal of Systems and Software, v.73 n.2, p.193-203, October 2004 [doi>10.1016/j.jss.2003.12.032]
• 10: N. Fenton and W. Wang. Risk and confidence analysis for fuzzy multicriteria decision making. Knowledge-Based Systems, 19(6):430--437, October 2006.
• 11: Norman E. Fenton , Niclas Ohlsson, Quantitative Analysis of Faults and Failures in a Complex Software System, IEEE Transactions on Software Engineering, v.26 n.8, p.797-814, August 2000 [doi>10.1109/32.879815]
• 12: Michael Fischer , Martin Pinzger , Harald Gall, Analyzing and Relating Bug Report Data for Feature Tracking, Proceedings of the 10th Working Conference on Reverse Engineering, p.90, November 13-17, 2003
• 13: L. Gou, Q. Wang. et al. Quantitatively managing defects for iterative projects: An industrial experience report in China. In Q. Wang, D. Pfahl, and D. M.Raffo, editors, Making Globally Distributed Software Development a Success Story, volume 5007 of LNCS, pages 369--380. Springer, 2008.
• 14: Les Hatton, Reexamining the Fault Density-Component Size Connection, IEEE Software, v.14 n.2, p.89-97, March 1997 [doi>10.1109/52.582978]
• 15: S. I., A. L., D. P., and S. E. On the use of Bayesian Belief Networks for the prediction of software productivity. Information and Software Technology, 45(1):51--60, January 2003.
• 16: Taghi M. Khoshgoftaar , Edward B. Allen , Robert Halstead , Gary P. Trio , Ronald M. Flass, Using Process History to Predict Software Quality, Computer, v.31 n.4, p.66-72, April 1998 [doi>10.1109/2.666844]
• 17: Taghi M. Khoshgoftaar , Edward B. Allen , Wendell D. Jones , John P. Hudepohl, Data Mining of Software Development Databases, Software Quality Control, v.9 n.3, p.161-176, November 2001 [doi>10.1023/A:1013349419545]
• 18: Taghi M. Khoshgoftaar , Erik Geleyn , Laurent Nguyen , Lofton Bullard, Cost-Sensitive Boosting In Software Quality Modeling, Proceedings of the 7th IEEE International Symposium on High Assurance Systems Engineering (HASE'02), p.51, October 23-25, 2002
• 19: Taghi M. Khoshgoftaar , Naeem Seliya, Fault Prediction Modeling for Software Quality Estimation: Comparing Commonly Used Techniques, Empirical Software Engineering, v.8 n.3, p.255-283, September 2003 [doi>10.1023/A:1024424811345]
• 20: Taghi M. Khoshgoftaar , Naeem Seliya, Comparative Assessment of Software Quality Classification Techniques: An Empirical Case Study, Empirical Software Engineering, v.9 n.3, p.229-257, September 2004 [doi>10.1023/B:EMSE.0000027781.18360.9b]
• 21: A. Gunes Koru , Hongfang Liu, Building Defect Prediction Models in Practice, IEEE Software, v.22 n.6, p.23-29, November 2005 [doi>10.1109/MS.2005.149]
• 22: A. Gunes Koru , Jeff (Jianhui) Tian, Comparing High-Change Modules and Modules with the Highest Measurement Values in Two Large-Scale Open-Source Products, IEEE Transactions on Software Engineering, v.31 n.8, p.625-642, August 2005 [doi>10.1109/TSE.2005.89]
• 23: Paul Luo Li , Jim Herbsleb , Mary Shaw, Forecasting Field Defect Rates Using a Combined Time-Based and Metrics-Based Approach: A Case Study of OpenBSD, Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering, p.193-202, November 08-11, 2005 [doi>10.1109/ISSRE.2005.19]
• 24: Paul Luo Li , James Herbsleb , Mary Shaw , Brian Robinson, Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc., Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134343]
• 25: Paul Luo Li , Mary Shaw , Jim Herbsleb , Bonnie Ray , P. Santhanam, Empirical evaluation of defect projection models for widely-deployed production software systems, ACM SIGSOFT Software Engineering Notes, v.29 n.6, November 2004
• 26: Nachiappan Nagappan , Thomas Ball , Andreas Zeller, Mining metrics to predict component failures, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134349]
• 27: Niclas Ohlsson , Hans Alberg, Predicting Fault-Prone Software Modules in Telephone Switches, IEEE Transactions on Software Engineering, v.22 n.12, p.886-894, December 1996 [doi>10.1109/32.553637]
• 28: L. H. Putnam, A General Empirical Solution to the Macro Software Sizing and Estimating Problem, IEEE Transactions on Software Engineering, v.4 n.4, p.345-361, July 1978 [doi>10.1109/TSE.1978.231521]
• 29: Q. Wang, L. Gou. et al. An empirical study on establishing quantitative management model for testing process. In Q. Wang, D. Pfahl, and D. M.Raffo, editors, Software Process Dynamics and Agility, volume 4470 of LNCS, pages 233--245. Springer, 2007.
• 30: Qing Wang , Lang Gou , Nan Jiang , Meiru Che , Ronghui Zhang , Yun Yang , Mingshu Li, Estimating fixing effort and schedule based on defect injection distribution, Software Process: Improvement and Practice, v.13 n.1, p.35-50, January 2008 [doi>10.1002/spip.v13:1]
• 31: Q. Wang and M. Li. Software process management: Practices in China. In Unifying the Software Process Spectrum, pages 317--331. Springer Berlin / Heidelberg, December 2006.
• 32: M. Xie. Software Reliability Modelling. Word Scientific Publishing Co. Pte. Ltd., Singapore, 1991.
• 33: S. Zhang, Y. Wang, Y. Yang, and J. Xiao. Capability assessment of individual software development processes using software repositories and DEA. In Q. Wang, D. Pfahl, and D. M. Raffo, editors, Making Globally Distributed Software Development a Success Story, volume 5007 of LNCS, pages 147--159. Springer, 2008.
• 34: Jiang Zheng , Laurie Williams , Nachiappan Nagappan , Will Snipes , John P. Hudepohl , Mladen A. Vouk, On the Value of Static Analysis for Fault Detection in Software, IEEE Transactions on Software Engineering, v.32 n.4, p.240-253, April 2006 [doi>10.1109/TSE.2006.38]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Complexity measures; Product metrics

General Terms:
Experimentation, Management, Measurement

Keywords:
empirical studies, metrics, software faults and failures