Reducing biases in individual software effort estimations: a combining approach

ABSTRACT

Software effort estimation techniques abound, each with its own set of advantages and disadvantages, and no one proves to be the single best answer. Combining estimating is an appealing approach. Avoiding the difficult problem of choosing the single "best" technique, it solves the problem by asking which techniques would help to improve accuracy, assuming that each has something to contribute. In this paper, we firstly introduce the systematic "external" combining idea into the field of software effort estimation, and estimate software effort using Optimal Linear Combining (OLC) method with an experimental study based on a real-life data set. The result indicates that combining different techniques can significantly improve the accuracy and consistency of software effort estimation by making full use of information provided by all components, even the much "worse" one.

REFERENCES

• 1: Barry W. Boehm, Software Engineering Economics, Prentice Hall PTR, Upper Saddle River, NJ, 1981
• 2: Barry W. Boehm , Clark , Horowitz , Brown , Reifer , Chulani , Ray Madachy , Bert Steece, Software Cost Estimation with Cocomo II with Cdrom, Prentice Hall PTR, Upper Saddle River, NJ, 2000
• 3: Lionel C. Briand , Khaled El Emam , Frank Bomarius, COBRA: a hybrid method for software cost estimation, benchmarking, and risk assessment, Proceedings of the 20th international conference on Software engineering, p.390-399, April 19-25, 1998, Kyoto, Japan
• 4: Barry Boehm , Chris Abts , Sunita Chulani, Software development cost estimation approaches – A survey, Annals of Software Engineering, v.10 n.1-4, p.177-205, 2000 [doi>10.1023/A:1018991717352]
• 5: C. Briand, L., Resource Estimation in Software Engineering, in International Software Engineering Research Network. 2000.
• 6: Magne Jorgensen , Martin Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, v.33 n.1, p.33-53, January 2007 [doi>10.1109/TSE.2007.3]
• 7: Lionel C. Briand , Tristen Langley , Isabella Wieczorek, A replicated assessment and comparison of common software cost modeling techniques, Proceedings of the 22nd international conference on Software engineering, p.377-386, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337223]
• 8: Andrew R. Gray , Stephen G. Macdonell, Software Metrics Data Analysis—Exploring the RelativePerformance of Some Commonly Used Modeling Techniques, Empirical Software Engineering, v.4 n.4, p.297-316, December 1999 [doi>10.1023/A:1009849100780]
• 9: Stephen G. MacDonell , Martin J. Shepperd, Combining techniques to optimize effort predictions in software project management, Journal of Systems and Software, v.66 n.2, p.91-98, 15 May 2003 [doi>10.1016/S0164-1212(02)00067-5]
• 10: Ross Jeffery , Melanie Ruhe , Isabella Wieczorek, Using Public Domain Metrics To Estimate Software Development Effort, Proceedings of the 7th International Symposium on Software Metrics, p.16, April 04-06, 2001
• 11: Idri, A., A. Abran, and T. M. Khoshgoftaar, Computational Intelligence in Empirical Software Engineering http://www.lrgl.uqam.ca/publications/pdf/807.pdf, in ENSIAS. 2003.
• 12: Barbara A. Kitchenham, Software Metrics: Measurement for Software Process Improvement, Blackwell Publishers, Inc., Cambridge, MA, 1996
• 13: Sherif Hashem, Optimal linear combinations of neural networks, Neural Networks, v.10 n.4, p.599-614, June 1997 [doi>10.1016/S0893-6080(96)00098-6]
• 14: Armstrong, J. S., ed. Principles of Forecasting: A Handbook for Researchers and Practitioners. Chapter 13 Combining Forecasts. 1 ed. 2001, Springer.
• 15: J. M. Bates and C. W. J. Granger, The Combination of Forecasts. Operational Research Quarterly, 1969. 20(4): p. 451--468.
• 16: Clemen, R. T., Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 1989. 5: p. 559--583.
• 17: Menezes, L. M. d., Review of Guidelines for the Use of Combined Forecasts. European Journal of Operation Research, 2000. 120(1): p. 190--204
• 18: E. Rapach, D. and J. K. Strauss, Forecasting U.S. Employment Growth Using Forecasting Combining Methods. Journal of Forecasting 2008. 27(1):p. 75--93.
• 19: Stock, J. H. and M. W. Watson, Forecasting Inflation. Journal of Monetary Economics, 1999. 44: p. 293--335.
• 20: Granger, C. W. J. and R. Ramanathan, Improved Methods of Combining Forecasts. Journal of Forecasting, 1984. 3(2): p. 197--204.
• 21: Diebold, F. X. and P. Pauly, Structural Change and the Combination of Forecasts. Journal of Forecasting, 1987. 6: p. 21--40.
• 22: Stock, J. H. and M. W. Watson, Combination Forecasts of Output Growth in a Seven-Country Data Set. Journal of Forecasting, 2004. 23: p. 405--430.
• 23: Clemen, R. T. and R. L. Winkler, Combining Economic Forecasts. Journal of Business and Economic Statistics, 1986. 4: p. 39--46.
• 24: Aiolfi, M. and A. Timmermann, Persistence in Forecasting Performance and Conditional Combination Strategies. Journal of Econometrics, 2004. 135(1-2): p.31--53.
• 25: Swanson, N. R. and T. Zeng, Choosing Among Competing Econometric Forecasts: Regression-Based Forecast Combination Using Model Selection. Journal of Forecasting, 2001. 20: p. 425--440.
• 26: Chan, Y. L., J. H. Stock, and M. W. Watson, A Dynamic Factor Model Framework for Forecast Combination. Spanish Economic Review, 1999. 1: p. 21--121.
• 27: Garratt, A., et al., Forecast Uncertainties in Macroeconomic Modeling: An Application to the U.K. Economy. Journal of the American Statistical Association, 2003. 98: p. 829--838.
• 28: Yang, Y., Combining Forecasting Procedures: Some Theoretical Results. Econometric Theory, 2004. 20: p. 176--222.
• 29: D. W. Bunn, Forecasting with more than one model. Journal of Forecasting, 1989. 8: p. 161--166.
• 30: Sherif Hashem, Optimal linear combinations of neural networks, Purdue University, West Lafayette, IN, 1993
• 31: Sunita Chulani , Barry Boehm , Bert Steece, Bayesian Analysis of Empirical Software Engineering Cost Models, IEEE Transactions on Software Engineering, v.25 n.4, p.573-583, July 1999 [doi>10.1109/32.799958]
• 32: Shepperd, M., Bournemouth University Automated Project Cost Estimation: Using Analogies The ANGEL Project.http://dec.bournemouth.ac.uk/ESERG/ANGEL. 1996.
• 33: Nikolaos Mittas , Marinos Athanasiades , Lefteris Angelis, Improving analogy-based software cost estimation by a resampling method, Information and Software Technology, v.50 n.3, p.221-230, February, 2008 [doi>10.1016/j.infsof.2007.04.001]
• 34: Magne Jorgensen, Practical Guidelines for Expert-Judgment-Based Software Effort Estimation, IEEE Software, v.22 n.3, p.57-63, May 2005 [doi>10.1109/MS.2005.73]
• 35: Barbara Kitchenham , Shari Lawrence Pfleeger , Beth McColl , Suzanne Eagan, An empirical study of maintenance and development estimation accuracy, Journal of Systems and Software, v.64 n.1, p.57-77, 15 October 2002 [doi>10.1016/S0164-1212(02)00021-3]
• 36: A. Rice, J., Mathematical Statistics and Data Analysis, Second Edition. Chapter 14 Linear Least Squares. 2003: Duxbury Press.
• 37: Armstrong, J. S., ed. Principles of Forecasting: A Handbook for Researchers and Practitioners. Chapter 14 Evaluation Methods. 1 ed. 2001, Springer.
• 38: H. Witten, I. and E. Frank, DATA MINING: Practical Machine Learning Tools and Techniques. Chapter 5 Credibility: Evaluating What's Been Learned. Second ed. 2005: Morgan Kaufmann.
• 39: A. J. C., S., On Combining Artificial Neural Nets. Connection Science, 1996. 8: p. 299--314.
• 40: Krogh, A. and J. Vedelsby, Statistical mechanics of ensemble learning. Physical Review E, 1997. 55(1): p. 811--825.
• 41: A. Rice, J., Mathematical Statistics and Data Analysis, Second Edition. Chapter 4.3 Covariance and Correlation. Second ed. 2003: Duxbury Press.
• 42: Chris F Kemerer, An empirical validation of software cost estimation models, Communications of the ACM, v.30 n.5, p.416-429, May 1987 [doi>10.1145/22899.22906]
• 43: Guerard, J. B. J. and R. T. Clemen, Colinearity and the use of latent root regression for combining GDP forecasts. Journal of Forecasting, 1989. 8: p. 231--238.
• 44: Gujarati, Basic Econometrics. Chapter 10. 4 ed. 2004: The McGraw-Hill Companies.
• 45: Gujarati, Basic Econometrics. Chapter 7. 4 ed. 2004: The McGraw-Hill Companies.
• 46: Motulsky, H., Analyzing Data with GraphPad Prism: A companion to GraphPad Prism version 3 http://www.graphpad.com. 1999: GraphPad Software, Inc.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Cost estimation

General Terms:
Algorithms, Economics, Experimentation, Management, Measurement

Keywords:
combining, olc, software effort estimation