Marcel Korte
ABSTRACT
Bootstrapping is used to approximate the standard error and 95% confidence intervals of MMRE and PRED for a number of COCOMO I model variations applied to four PROMISE data sets. This is used to illustrate a lack of confidence in numerous published cost estimation research results based on MMRE and PRED comparisons such as model selection. We show that many such results are of questionable significance due to large possible variations resulting from population sampling error and suggest that a number of inconsistent and contradictory results may be explained by this. By using more standard statistical approaches that account for standard error, we may reduce the incidence of this and obtain greater confidence cost estimation in research results.
- K. Moløkken, M. Jørgensen, "A Review of Surveys on Software Effort Estimation," International Symposium on Empirical Software Engineering, Rome, Italy, 2003 Google Scholar
Digital Library
- I. Myrtweit, E. Stensrud, M. Shepperd, "Reliability and Validity in Comparative Studies of Software Prediction Models, IEEE Transactions on Software Engineering, Vol. 31, No. 5, 2005 Google ScholarDigital Library
- T. Menzies, Z. Chen, J. Hihn, K. Lum, "Selecting Best Practices for Effort Estimation, IEEE Transactions on Software Engineering, Vol. 32, No. 11, 2006 Google ScholarDigital Library
- M. Shepperd "Evaluating Software Project Prediction Systems," 11th IEEE International Software Metrics Symposium, Como, Italy, 2005 Google ScholarDigital Library
- T. Menzies, D. Port, Z. Chen, J. Hihn, S. Stukes, "Validation Methods for Calibrating Software Effort Models, Proceedings of the 27th international conference on Software engineering, 2005 Google ScholarDigital Library
- I. Wieczorek, M. Ruhe, "How valuable is company-specific data compared to multi-company data for software cost estimation?, Proceeding for the Eights IEEE Symposium on Software Metrics (METRICS 02), 2002 Google ScholarDigital Library
- Ch. Mooney, Robert Duval, "Bootstrapping: A Nonparametric Approach to Statistical Inference, Sage Publications; 1. edition (1993)Google Scholar
- B. Efron (1979). "Bootstrap methods: Another look at the jackknife," The Annals of Statistics, 7, 1--26Google Scholar
- T. Foss, E. Stensrud, B. Kitchenham, I. Myrtveit, "A simulation study of the Model Evaluation Criterium MMRE, IEEE Transactions on Software Engineering, Vol. 20, No. 11, 2003 Google ScholarDigital Library
- M. Shepperd, G. Kadoda, "using Simulation to Evaluate Prediction Techniques," Proc. Fifth Int'l Software Metrics Symp., 2001 Google ScholarDigital Library
- B. Boehm. Software Engineering Economics. Prentice Hall, 1981 Google ScholarDigital Library
- J.M. Desharnais, "Analyse statistique de la productivitie des projets informatique a partie de la technique des point des fonction," masters thesis, Univ. of Montreal, 1989.Google Scholar
- M. Jørgensen, "How Much Does a Vacation Cost? or What is a Software Cost Estimate?," ACM SIGSOFT Software Engineering Notes, P. 5, Vol. 28, No. 6, 2003 Google ScholarDigital Library
- S. D. Conte, H. E. Dunsmore, and V. Y. Shen, "Software engineering metrics and models, Benjamin--Cummings Publishing, 1986 Google ScholarDigital Library
- M. Jørgensen, "Experience with the accuracy of software maintenance task effort prediction models," IEEE Transactions on Software Engineering, Vol. 21, No. 8, 1995 Google ScholarDigital Library
- G. Boetticher, T. Menzies, and T. Ostrand. The PROMISE Repository of Empirical Software Engineering Data, 2007. http://promisedata.org/repository.Google Scholar
- B. Kitchenham, L. Pickard, S. MacDonell, M. Sheppard, "What accuracy statistics really measure, Proceedings of the IEEE, Vol. 148, No. 3, 2001Google Scholar
- M. Jørgensen, M. Shepperd, :"A Systematic Review of Software Development Cost Estimation Studies," IEEE Trans. Software Eng., Vol. 33, No. 1, 2007 Google ScholarDigital Library
- M. Shepperd, C. Schofield, "Estimating Software Effort using Anologies," IEEE Transactions on Software Engineering, 1997 Google ScholarDigital Library
- G. Hood, http://www.cse.csiro.au/poptools, 01/19/2008Google Scholar
- http://en.wikipedia.org/wiki/Vysochanskii-Petunin_inequality, 01/19/2008Google Scholar
- R. Larsen, M. Marx, "An Introduction to Mathematical Statistics and its Applications, Second Edition, Prentice Hall, 1986Google Scholar
- L. Briand, T. Langley, and I. Wieczorek. "A replicated assessment and comparison of common software cost modeling techniques, in Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland, 2000, pp. 377--386. Google ScholarDigital Library
- K. Lum, J. Hihn, T. Menzies, "Studies in Software Cost Model Behavior:Do We Really Understand Cost Model Performance?, Proceedings of the ISPA International Conference 2006, Seattle, WAGoogle Scholar
- COCOMO81 dataset, http://promisedata.org/repository/#coc81, 12/29/2007Google Scholar
- COCOMONASA dataset, http://promisedata.org/repository/#cocomonasa_v1, 01/19/2008Google Scholar
- NASA93 dataset, http://promisedata.org/repository/#nasa93, 12/29/2007Google Scholar
- Desharnis dataset, http://promisedata.org/repository/#desharnais, 12/29/2007Google Scholar
Index Terms
- Confidence in software cost estimation results based on MMRE and PRED
Recommendations
Comparative studies of the model evaluation criterions mmre and pred in software cost estimation research
ESEM '08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurementSoftware cost model research results depend on model accuracy criteria such as MMRE and PRED. Despite criticism, MMRE has emerged as the de facto standard criterion. Many alternatives have been proposed and studied, surprisingly however PRED, the second ...
A novel fuzzy based approach for effort estimation in software development
Accurate and credible software effort estimation is always a challenge for academic research and software industry. In the beginning, estimation was carried out using only human expertise or algorithmic models, but more recently, interest has turned to ...
Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation
AbstractSoftware development effort estimation (SDEE) plays a primary role in software project management. But choosing the appropriate SDEE technique remains elusive for many project managers and researchers. Moreover, the choice of a reliable estimation ...
Comments