Investigations about replication of empirical studies in software engineering: preliminary findings from a mapping study

ABSTRACT

Context. A recent mapping study intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: the empirical studies reporting actually replications and a second group of studies that were concerned with concepts, classifications, guidelines, and other themes about replication. Objective. The purpose of this research is to analyze and discuss the content of the second set of studies about replications published between 1996 and 2013. Method. This is a mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996-2012 complemented by a non-systematic search that collected some articles published in 2013. Results. We analyzed 36 papers reporting empirical and non-empirical studies about replications in SE published in the last 17 years. These papers explored different topics related to concepts and classifications, presented guidelines, and discussed theoretical issues that are relevant for our understanding of replication in our field.

References

1. Popper, K. 1959. The Logic of Scientific Discovery. Hutchinson & Co. p. 45.
2. Basili, V., Shull, F. and Lanubile, F., 1999. Building knowledge through families of experiments, IEEE Transactions on Software Engineering 25 (1999) 456--473.
3. Gómez, O. S., Juristo, N. and Vegas, S. 2010. "Replications Types in Experimental Disciplines" In: Proceedings of the ACM/IEEE 4th International Symposium on Empirical Software Engineering and Measurement, September 16--17, Bolzano-Bozen, Italy. pp. 1--10.
4. Shull, F., Carver, J., Vegas, S. and Juristo, N. 2008. The role of replications in empirical Software Engineering. Empirical Software Engineering, 13(2):211--218, 2008. DOI= 10.1007/s10664-008-9060-1.
5. Juristo, N and Vegas S. 2011. The role of non-exact replications in Software Engineering experiments. Empirical Software Engineering. 16, 295--324. DOI= 10.1007/s10664-010-9141-9
6. Carver, J. Towards Reporting Guidelines for Experimental Replications: A Proposal. 2010. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa.
7. De Magalhaes, C. V. C and Da Silva, F. Q. B. 2013. Towards a Taxonomy of Replications in Empirical Software Engineering Research: A Research Proposal. 2013. In: Proceedings of the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER). DOI= 10.1109/RESER.2013.10.
8. Easterbrook, S. M., Singer, J., M.-A. Storey, S. M. and Damian, D. 2007. Selecting empirical methods for Software Engineering research. Guide to Advanced Empirical Software Engineering, pages 285--311. Springer.
9. Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonça, M. and Fabbri, S. 2002. Replicating Software Engineering experiments: Addressing the tacit knowledge problem. In: Proceedings of Int. Symp. on Empirical Softw. Eng., Washington, DC, USA, IEEE Computer Society
10. Lung, J., Aranda J., Easterbrook, S. and Wilson, G. 2008. On the difficulty of replicating human subjects studies in Software Engineering. In Proceedings of the 13th international conference on Software Engineering - ICSE '08, New York, USA: ACM Press, p. 191.
11. da Silva, F. Q. B., Suassuna, M., França, A. C. C., Grubb, A. M., Gouveia, T. B., Monteiro, C. V. F., dos Santos, I. E. 2012. Replication of empirical studies in Software Engineering research: a systematic mapping study. Empirical Software Engineering (Dordrecht. Online). DOI= 10.1007/s10664-012-9227-7.
12. Arksey, H. and O'Malley, L. 2005. Scoping studies: towards a methodological framework. Int J Social Research Methodology. 8(1):19--32.
13. da Silva, F. Q. B., Santos, A. L. M., Soares, S. C. B., França, A. C. C., Monteiro, C. V. F. and Maciel, F. F. 2011. Six years of systematic literature reviews in Software Engineering: An updated tertiary study. Information and Software Technology 53, 899--913.
14. Petticrew, M. and Roberts, H. 2006. Systematic Reviews in the Social Sciences. Blackwell Publishing. 336
15. Kitchenham, B. and Charters, S. Guidelines for performing systematic literature reviews in Software Engineering. 2007. Technical Report EBSE-2007-01, School of Computer Science and Mathematics, Keele University.
16. Almqvist, J. F. P. 2006. Replication of controlled experiments in empirical Software Engineering -- a survey. 2006. Master's thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden.
17. Krein, J. L. and Knutson, C. D. 2010. A Case for Replication: Synthesizing Research Methodologies in Software Engineering. In: 1st International Workshop on Replication in Empirical Software Engineering Research, RESER 2010
18. {ABO001} Mendonça, M., Maldonado, J., Oliveira, M., Carver, J., Fabbri, S., Shull, F., Travassos, G., Höhn, E., and Basili, V. 2008. A Framework for Software Engineering Experimental Replications. In Proceedings of the 13th International Conference on Engineering Complex Computer Systems (ICECCS 2008). March 31 - April 4, 2008. Belfast, Northern Ireland. p. 203--212.
19. {ABO002} Cater-Steel, A., Toleman, M. and Rout, T. 2005. Addressing the challenges of replications of surveys in Software Engineering research. In: International Symposium on Empirical Software Engineering, 17--18 Nov 2005, Noosa Heads, Australia.
20. {ABO003} Vegas, S., Juristo N., Moreno, A. M., Solari, M. and Letelier, P. 2006. Analysis of the influence of communication between researchers on experiment replication. In Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering: 28--37. New York, NY, USA. DOI= 10.1145/1159733.1159741
21. {ABO004} Basili, V., Shull, F. and Lanubile, F., 1999. Building knowledge through families of experiments, IEEE Transactions on Software Engineering. 25 (1999) 456--473.
22. {ABO005} Kitchenham, B., Al-Khilidar, H., Ali Babar, M., Berry, M., Cox, C., Keung, J., Kurniawati, F., Staples, M., Zhang, H. and Zhu, L. 2006. Evaluating Guidelines for Empirical Software Engineering Studies; In Proc. of ACM/IEEE Intern. Symposium on Software Engineering, Rio de Janerio, Brazil, Sep 2006, IEEE CS.
23. {ABO006} Ohlsson M. C. and Runeson P. 2002. "Experience from replicating empirical studies on prediction models", In: Proceedings of the 8th International Software Metrics Symposium, Ottawa, Ontario, Canada, IEEE Computer Society, June 2002, pp. 217--226.
24. {ABO007} Shull, F., Mendonça, M. G., Basili, V., Carver, J., Maldonado, J. C., Fabbri, S., Travassos, G. H., and Ferreira, M. C. 2004. Knowledge-Sharing Issues in Experimental Software Engineering. Empirical Software Engineering, 9(1--2):111--137, March 2004.
25. {ABO008} Robles, G. 2010. Replicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories proceedings. In: 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). Cape Town, South Africa: IEEE; p. 171--80
26. {ABO009} Mende, T. 2010. Replication of Defect Prediction Studies: Problems, Pitfalls and Recommendations. In: Proceedings of the International Conference on Predictive Models in Software Engineering, ACM Press; p. 1--10.
27. {ABO010} Gomez, O. S., Juristo, N. and Vegas, S. 2010. Replications types in experimental disciplines. In: Proceedings of International Symposium on Empirical Software Engineering and Measurement. DOI= 10.1145/1852786.1852790.
28. {ABO011} Juristo, N. and Vegas, S. 2011. The role of non-exact replications in Software Engineering experiments. Empirical Software Engineering 16(3): 295--324. DOI= 10.1007/s10664-010-9141-9.
29. {ABO012} Shull F., Carver, J. C., Vegas, S. and Juristo, N. 2008. The role of replications in Empirical Software Engineering. Empirical Software Engineering 13(2): 211--218, DOI= 10.1007/s10664-008-9060-1.
30. {ABO013} Kitchenham, B. 2008. The role of replications in empirical Software Engineering - a word of warning. Empirical Software Engineering 13(2): 219--221. DOI= 10.1007/s10664-008-9061-0.
31. {ABO014} Juristo, N. J. and Vegas, S. 2010. Using differences among replications of Software Engineering experiments to gain knowledge. In: Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement, Cape Town, South Africa. DOI= 10.1109/ESEM.2009.5314236.
32. {ABO015} Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonça, M., and Fabbri, S. 2002. Replicating Software Engineering Experiments: Addressing the Tacit Knowledge Problem. In: Proceedings of the 2002 International Symposium on Empirical Software Engineering, p. 7--16. Nara, Japan, October 3-4 2002.
33. {ABO016} Shull, F., Carver, J., Travassos, G., Maldonado, J., Conradi, R., and Basili, V. 2003. Replicated Studies: Building a Body of Knowledge about Software Reading Techniques. Lecture Notes on Empirical Software Engineering. World Scientific.
34. {ABO017} Miller, J. 2005. Replicating Software Engineering Experiments: A Poisoned Chalice or the Holy Grail, Information and Software Technology, vol. 47, no. 4, pp. 233--244.
35. {ABO018} Carver, J. 2010. Towards Reporting Guidelines for Experimental Replications: A Proposal. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
36. {ABO019} Dieste, O., Fernandez, E., Garcia, R., and Juristo, N. 2010. Hidden evidence behind useless replications. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
37. {ABO020} Ferrari, R., Sudmann, O., Henke, C., Geisler, J., Schäfer, W. and Madhavji, N. H. Transitioning from lab studies to large-scale studies: Emerging results from a literal replication. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
38. {ABO021} Gomez O. S., Juristo, N., Vegas, S. 2010. Replication, reproduction and re-analysis: three ways for verifying experimental findings. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
39. {ABO022} Jonathan L. Krein and Charles D. Knutson. 2010. A Case for Replication: Synthesizing Research Methodologies in Software Engineering. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
40. {ABO023} Mäntylä, M. V., Lassenius, C. and Vanhanen, J., 2010. Rethinking Replication in Software Engineering: Can We See the Forest for the Trees? In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
41. {ABO024} A. Mockus, B. Anda and D. I. K. Sjøberg. 2010. Experiences from Replicating a Case Study to Investigate Reproducibility of Software Development. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
42. {ABO025} Robles, G. and German, D. M. 2010. Beyond Replication: An example of the potential benefits of replicability in the Mining of Software Repositories Community. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.
43. {ABO026} Juristo, N., Vegas, S. 2011. The role of non-exact replications in Software Engineering experiments. Empirical Software Engineering June 2011, Vol. 16, Issue 3, p. 295--324. DOI= 10.1007/s10664-010-9141-9.
44. {ABO027} Juristo, N. and Gómez, O. 2012. Replication of Software Engineering Experiments. Empirical Software Engineering and Verification. Lecture Notes in Computer Science Volume 7007, 2012, pp 60--88. DOI= 10.1007/978-3-642-25231-0_2.
45. {ABO028} Scatalon, L. P., Garcia R. E., Correia, R. C. M. 2011. Packaging Controlled Experiments Using an Evolutionary Approach Based on Ontology(S). In: Proceedings of the 23rd International Conference on Software Engineering & Knowledge Engineering (SEKE'2011), Eden Roc Renaissance, Miami Beach, USA, July 7--9, 2011
46. {ABO029} Gallardo, E. G. E. 2012. Using configuration management and product line software paradigms to support the experimentation process in Software Engineering. In: 6th International Conference on Research Challenges in Information Science (RCIS 2012) p. 1--6, Valencia, Spain.
47. {ABO030} De Magalhaes, C. V. C and Da Silva, F. Q. B. 2013. Towards a Taxonomy of Replications in Empirical Software Engineering Research: A Research Proposal. In: Proceedings of the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER 2013). Baltimore, MD. DOI= 10.1109/RESER.2013.10
48. {ABO031} Ghezzi, G. and Gall H. C. 2013. Replicating mining studies with SOFAS. In: Proceedings of the 10th Working Conference on Mining Software Repositories. p. 363--372.
49. {ABO032} Callele, D., Wnuk, K. and Borg, M. 2013. Confounding factors when conducting industrial replications in requirements engineering. In: 1st International Workshop on Conducting Empirical Studies in Industry (CESI). p. 55--58. DOI= 10.1109/CESI.2013.6618472
50. {ABO033} Squire, M. 2013. A Replicable Infrastructure for Empirical Studies of Email Archives. In: Proceedings of the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER 2013). Baltimore, MD. p. 43--49. DOI= 10.1109/RESER.2013.11.
51. {ABO034} Juristo, N., Vegas, S., Solari, M., Abrahão, S., and Ramos, I. 2013. A process for managing interaction between experimenters to get useful similar replications. Information & Software Technology 55(2): 215--225
52. {ABO035} Runeson, P., Stefik, A. and Andrews, A. 2013. Variation factors in the design and analysis of replicated controlled experiments. Empirical Software Engineering. Springer US. DOI= 10.1007/s10664-013-9262-z.
53. {ABO036} Brooks, A., Daly, J., Miller, J., Roper, M. and Wood, M. 1996. Replication of Experimental Results in Software Engineering. Technical Report ISERN-96-10, Department of Computer Science, University of Strathclyde, Glasgow, Scotland, UK, 1996

Index Terms

• Software and its engineering