Deriving models of software fault-proneness

ABSTRACT

The effectiveness of the software testing process is a key issue for meeting the increasing demand of quality without augmenting the overall costs of software development. The estimation of software fault-proneness is important for assessing costs and quality and thus better planning and tuning the testing process. Unfortunately, no general techniques are available for estimating software fault-proneness and the distribution of faults to identify the correct level of test for the required quality. Although software complexity and testing thoroughness are intuitively related to the costs of quality assurance and the quality of the final product, single software metrics and coverage criteria provide limited help in planning the testing process and assuring the required quality.By using logistic regression, this paper shows how models can be built that relate software measures and software fault-proneness for classes of homogeneous software products. It also proposes the use of cross-validation for selecting valid models even for small data sets.The early results show that it is possible to build statistical models based on historical data for estimating fault-proneness of software modules before testing, and thus better planning and monitoring the testing activities.

REFERENCES

• 1: V. Basili and D. Hutchens. An empirical study of a syntactic complexity family. IEEE Transactions on Software Engineering, 9(6):664-672, November 1983. Special Section on Software Metrics.
• 2: Lionel C. Briand , Victor R. Basili , William M. Thomas, A Pattern Recognition Approach for Software Engineering Data Analysis, IEEE Transactions on Software Engineering, v.18 n.11, p.931-942, November 1992 [doi>10.1109/32.177363]
• 3: Lionel C. Briand , Sandro Morasca , Victor R. Basili, Defining and Validating Measures for Object-Based High-Level Design, IEEE Transactions on Software Engineering, v.25 n.5, p.722-743, September 1999 [doi>10.1109/32.815329]
• 4: Norman E. Fenton , Martin Neil, A Critique of Software Defect Prediction Models, IEEE Transactions on Software Engineering, v.25 n.5, p.675-689, September 1999 [doi>10.1109/32.815326]
• 5: Phyllis G. Frankl , Oleg Iakounenko, Further empirical studies of test effectiveness, ACM SIGSOFT Software Engineering Notes, v.23 n.6, p.153-162, Nov. 1998
• 6: P. G. Frankl , E. J. Weyuker, Provable Improvements on Branch Testing, IEEE Transactions on Software Engineering, v.19 n.10, p.962-975, October 1993 [doi>10.1109/32.245738]
• 7: Geoffrey K. Gill , Chris F. Kemerer, Cyclomatic Complexity Density and Software Maintenance Productivity, IEEE Transactions on Software Engineering, v.17 n.12, p.1284-1288, December 1991 [doi>10.1109/32.106988]
• 8: Maurice H. Halstead, Elements of Software Science (Operating and programming systems series), Elsevier Science Inc., New York, NY, 1977
• 9: D. Hosmer and S. Lemeshow. Applied Logistic Regression. Wiley-Interscience, 1989.
• 10: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 11: Taghi M. Khoshgoftaar , Edward B. Allen , Robert Halstead , Gary P. Trio , Ronald M. Flass, Using Process History to Predict Software Quality, Computer, v.31 n.4, p.66-72, April 1998 [doi>10.1109/2.666844]
• 12: Taghi M. Khoshgoftaar , Edward B. Allen , Kalai S. Kalaichelvan , Nishith Goel, Early Quality Prediction: A Case Study in Telecommunications, IEEE Software, v.13 n.1, p.65-71, January 1996 [doi>10.1109/52.476287]
• 13: T. Khoshgoftaar, D. Lanning, and A. Pandya. A comparative-study of pattern-recognition techniques for quality evaluation of telecommunications software. IEEE Journal On Selected Areas In Communications, 12(2):279-291, 1994.
• 14: Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test application frequency, Proceedings of the 22nd international conference on Software engineering, p.126-135, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337196]
• 15: M. M. Lehman , D. E. Perry , J. F. Ramil, Implications of Evolution Metrics on Software Maintenance, Proceedings of the International Conference on Software Maintenance, p.208, March 16-19, 1998
• 16: H. F. Li , W. K. Cheung, An empirical study of software metrics, IEEE Transactions on Software Engineering, v.13 n.6, p.697-708, June 1987 [doi>10.1109/TSE.1987.233475]
• 17: T. McCabe. A complexity measure. IEEE Transactions on Software Engineering, 2(4):308-320, December 1976.
• 18: P. McCullagh and J. A. Nelder. Generalized Linear Models. Chapman and Hall, London, second edition, 1989.
• 19: Keith W. Miller , Larry J. Morell , Robert E. Noonan , Stephen K. Park , David M. Nicol , Branson W. Murrill , Jeffrey M. Voas, Estimating the Probability of Failure When Testing Reveals No Failures, IEEE Transactions on Software Engineering, v.18 n.1, p.33-43, January 1992 [doi>10.1109/32.120314]
• 20: Sandro Morasca , Günther Ruhe, A hybrid approach to analyze empirical software engineering data and its application to predict module fault-proneness in maintenance, Journal of Systems and Software, v.53 n.3, p.225-237, Sep 15, 2000 [doi>10.1016/S0164-1212(00)00014-5]
• 21: John C. Munson , Taghi M. Khoshgoftaar, The Detection of Fault-Prone Programs, IEEE Transactions on Software Engineering, v.18 n.5, p.423-433, May 1992 [doi>10.1109/32.135775]
• 22: Niclas Ohlsson , Hans Alberg, Predicting Fault-Prone Software Modules in Telephone Switches, IEEE Transactions on Software Engineering, v.22 n.12, p.886-894, December 1996 [doi>10.1109/32.553637]
• 23: A. Pasquini, A. Crespo, and P. Matrella. Sensitivity of reliability-growth models to operational profile errors vs. testing accuracy {software testing}. IEEE Transaction on Reliability, 45(4):531-540, December 1996.
• 24: Adam A. Porter , Richard W. Selby, Empirically Guided Software Development Using Metric-Based Classification Trees, IEEE Software, v.7 n.2, p.46-54, March 1990 [doi>10.1109/52.50773]
• 25: J. R. Quinlan, Induction of Decision Trees, Machine Learning, v.1 n.1, p.81-106 [doi>10.1023/A:1022643204877]
• 26: Richard W. Selby , Victor R. Basili, Analyzing Error-Prone System Structure, IEEE Transactions on Software Engineering, v.17 n.2, p.141-152, February 1991 [doi>10.1109/32.67595]
• 27: R. W. Selby , A. A. Porter, Learning from Examples: Generation and Evaluation of Decision Trees for Software Resource Analysis, IEEE Transactions on Software Engineering, v.14 n.12, p.1743-1757, December 1988 [doi>10.1109/32.9061]
• 28: Jeffrey M. Voas , Keith W. Miller, Semantic metrics for software testability, Journal of Systems and Software, v.20 n.3, p.207-216, March 1993 [doi>10.1016/0164-1212(93)90064-5]
• 29: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 30: M. Woodward, M. Hennell, and D. Hedley. A measure of control flow complexity in program text. IEEE Transactions on Software Engineering, 5(1):45-50, January 1979.

INDEX TERMS

Keywords:
cross-validation, fault-proneness models, logistic regression, software faultiness, software metrics, software testing process