Sira Vegas
ABSTRACT
Experimentation is a key issue in science and engineering. But it is one of software engineering's stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some experimentation issues being conceived of differently than in other disciplines. The aim of this technical briefing is to help participants to avoid common pitfalls when analyzing the results of software engineering experiments. The technical briefing is not intended as a data analysis course, because there is already plenty of literature on this subject. It reviews several key issues that we have identified in published software engineering experiments, and addresses them based on the knowledge acquired after 19 years running experiments.
- H. Pashler, E.J. Wagenmakers. Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science, 7(6):528--530, 2012.Google Scholar
Cross Ref
- M. Shepperd, D. Bowes, T. Hall. Researcher Bias: The Use of Machine Learning in Software Defect Prediction. IEEE Transactions on Software Engineering, 40(6):603--616, 2014.Google ScholarCross Ref
- D.I.K. Sjøberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.K. Liborg, A.C. Rekdal. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733--753, 2005 Google ScholarDigital Library
Index Terms
- Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask
Recommendations
Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask
ICSE-C '17: Proceedings of the 39th International Conference on Software Engineering CompanionExperimentation is a key issue in science and engineering. But it is one of software engineering's stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some ...
Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask
ICSE '16: Proceedings of the 38th International Conference on Software Engineering CompanionExperimentation is a key issue in science and engineering. But it is one of software engineering's stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some ...
Incorrect results in software engineering experiments
Publication and researcher bias is common in software engineering experiments.Our model shows how these biases lead to a high proportion of incorrect results.Increased statistical power is a key factor to improve the trustworthiness. ContextThe ...
Comments