Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Software cost model research results depend on model accuracy criteria such as MMRE and PRED. Despite criticism, MMRE has emerged as the de facto standard criterion. Many alternatives have been proposed and studied, surprisingly however PRED, the second most popular criterion, has not been extensively studied. This work attempts to fill this gap in the literature and expand the understanding and use of evaluation criterion in general. The majority of this work is empirically based, applying MMRE and PRED to a number of COCOMO model variations with respect to a simulated data set and four publicly available cost estimation data sets. We replicate a number of results based on MMRE and extend them to PRED. We study qualities of MMRE and PRED as sample estimator statistics for parameters of a cost model error distribution. Standard error is used to ensure greater confidence in replicated and new results based on sample data.

REFERENCES

• 1: Kjetil Molkken , Magne Jrgensen, A Review of Surveys on Software Effort Estimation, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.223, September 30-October 01, 2003
• 2: Ingunn Myrtveit , Erik Stensrud , Martin Shepperd, Reliability and Validity in Comparative Studies of Software Prediction Models, IEEE Transactions on Software Engineering, v.31 n.5, p.380-391, May 2005 [doi>10.1109/TSE.2005.58]
• 3: Tim Menzies , Zhihao Chen , Jairus Hihn , Karen Lum, Selecting Best Practices for Effort Estimation, IEEE Transactions on Software Engineering, v.32 n.11, p.883-895, November 2006 [doi>10.1109/TSE.2006.114]
• 4: Martin Shepperd, Evaluating Software Project Prediction Systems, Proceedings of the 11th IEEE International Software Metrics Symposium, p.2, September 19-22, 2005 [doi>10.1109/METRICS.2005.22]
• 5: Tim Menzies , Dan Port , Zhihao Chen , Jairus Hihn , Sherry Stukes, Validation methods for calibrating software effort models, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062559]
• 6: Isabella Wieczorek , Melanie Ruhe, How Valuable is company-specific Data Compared to multi-company Data for Software Cost Estimation?, Proceedings of the 8th International Symposium on Software Metrics, p.237, June 04-07, 2002
• 7: Ch. Mooney, Robert Duval, "Bootstrapping: A Nonparametric Approach to Statistical Inference", Sage Publications; 1. edition (1993)
• 8: B. Efron (1979). "Bootstrap methods: Another look at the jackknife", The Annals of Statistics, 7, 1--26
• 9: Tron Foss , Erik Stensrud , Barbara Kitchenham , Ingunn Myrtveit, A Simulation Study of the Model Evaluation Criterion MMRE, IEEE Transactions on Software Engineering, v.29 n.11, p.985-995, November 2003 [doi>10.1109/TSE.2003.1245300]
• 10: Martin Shepperd , Gada Kadoda, Using Simulation to Evaluate Prediction Techniques, Proceedings of the 7th International Symposium on Software Metrics, p.349, April 04-06, 2001
• 11: Barry W. Boehm, Software Engineering Economics, Prentice Hall PTR, Upper Saddle River, NJ, 1981
• 12: J. M. Desharnais, "Analyse statistique de la productivitie des projets informatique a partie de la technique des point des fonction," masters thesis, Univ. of Montreal, 1989.
• 13: Magne Jørgensen, How much does a vacation cost?: or what is a software cost estimate?, ACM SIGSOFT Software Engineering Notes, v.28 n.6, November 2003 [doi>10.1145/966221.966238]
• 14: S. D. Conte , H. E. Dunsmore , Y. E. Shen, Software engineering metrics and models, Benjamin-Cummings Publishing Co., Inc., Redwood City, CA, 1986
• 15: Magne Jørgensen, Experience With the Accuracy of Software Maintenance Task Effort Prediction Models, IEEE Transactions on Software Engineering, v.21 n.8, p.674-681, August 1995 [doi>10.1109/32.403791]
• 16: G. Boetticher, T. Menzies, and T. Ostrand. The PROMISE Repository of Empirical Software Engineering Data, 2007. http://promisedata.org/repository.
• 17: B. Kitchenham, L. Pickard, S. MacDonell, M. Sheppard, "What accuracy statistics really measure", Proceedings of the IEEE, Vol. 148, No. 3, 2001
• 18: Magne Jorgensen , Martin Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, v.33 n.1, p.33-53, January 2007 [doi>10.1109/TSE.2007.3]
• 19: Martin Shepperd , Chris Schofield, Estimating Software Project Effort Using Analogies, IEEE Transactions on Software Engineering, v.23 n.11, p.736-743, November 1997 [doi>10.1109/32.637387]
• 20: G. Hood, http://www.cse.csiro.au/poptools, 01/19/2008
• 21: http://en.wikipedia.org/wiki/Vysochanskii-Petunin_inequality, 01/19/2008
• 22: R. Larsen, M. Marx, "An Introduction to Mathematical Statistics and its Applications", Second Edition, Prentice Hall, 1986
• 23: Lionel C. Briand , Tristen Langley , Isabella Wieczorek, A replicated assessment and comparison of common software cost modeling techniques, Proceedings of the 22nd international conference on Software engineering, p.377-386, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337223]
• 24: K. Lum, J. Hihn, T. Menzies, "Studies in Software Cost Model Behavior: Do We Really Understand Cost Model Performance?", Proceedings of the ISPA International Conference 2006, Seattle, WA
• 25: COCOMO81 dataset, http://promisedata.org/repository/#coc81, 12/29/2007
• 26: COCOMONASA dataset, http://promisedata.org/repository/#cocomonasa_v1, 01/19/2008
• 27: NASA93 dataset, http://promisedata.org/repository/#nasa93, 12/29/2007
• 28: Desharnis dataset, http://promisedata.org/repository/#desharnais, 12/29/2007
• 29: L. C. Briand, I. Wieczorek, "Resource Estimation in Software Engineering, Encyclopedia of Software Engineering", Pp. 1160--1196, Wiley-Interscience Publishing, 2001
• 30: BESTweb - Better Estimation of Software Tasks, http://www.simula.no/~simula/se/bestweb/, 2007
• 31: Lionel C. Briand , Khaled El Emam , Dagmar Surmann , Isabella Wieczorek , Katrina D. Maxwell, An assessment and comparison of common software cost estimation modeling techniques, Proceedings of the 21st international conference on Software engineering, p.313-322, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302647]
• 32: Barbara A. Kitchenham , Emilia Mendes , Guilherme H. Travassos, Cross versus Within-Company Cost Estimation Studies: A Systematic Review, IEEE Transactions on Software Engineering, v.33 n.5, p.316-329, May 2007 [doi>10.1109/TSE.2007.1001]
• 33: Carolyn Mair , Martin Shepperd , Magne Jørgensen, An analysis of data sets used to train and validate cost prediction systems, Proceedings of the 2005 workshop on Predictor models in software engineering, p.1-6, May 15-15, 2005, St. Louis, Missouri
• 34: Land, C. E. (1971), "Confidence intervals for linear functions of the normal mean and variance", Annals of Mathematical Statistics, 42, 1187--1205

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Cost estimation

General Terms:
Management, Measurement, Performance, Reliability, Standardization, Theory, Verification

Keywords:
bootstrapping, calibration, confidence, confidence interval, cost estimation, cost model, mmre, model selection, parameters, pred, standard error