The State of Empirical Evaluation in Static Feature Location

Abstract

Feature location (FL) is the task of finding the source code that implements a specific, user-observable functionality in a software system. It plays a key role in many software maintenance tasks and a wide variety of Feature Location Techniques (FLTs), which rely on source code structure or textual analysis, have been proposed by researchers. As FLTs evolve and more novel FLTs are introduced, it is important to perform comparison studies to investigate “Which are the best FLTs?” However, an initial reading of the literature suggests that performing such comparisons would be an arduous process, based on the large number of techniques to be compared, the heterogeneous nature of the empirical designs, and the lack of transparency in the literature. This article presents a systematic review of 170 FLT articles, published between the years 2000 and 2015. Results of the systematic review indicate that 95% of the articles studied are directed towards novelty, in that they propose a novel FLT. Sixty-nine percent of these novel FLTs are evaluated through standard empirical methods but, of those, only 9% use baseline technique(s) in their evaluations to allow cross comparison with other techniques. The heterogeneity of empirical evaluation is also clearly apparent: altogether, over 60 different FLT evaluation metrics are used across the 170 articles, 272 subject systems have been used, and 235 different benchmarks employed. The review also identifies numerous user input formats as contributing to the heterogeneity. Analysis of the existing research also suggests that only 27% of the FLTs presented might be reproduced from the published material. These findings suggest that comparison across the existing body of FLT evaluations is very difficult. We conclude by providing guidelines for empirical evaluation of FLTs that may ultimately help to standardise empirical research in the field, cognisant of FLTs with different goals, leveraging common practices in existing empirical evaluations and allied with rationalisations. This is seen as a step towards standardising evaluation in the field, thus facilitating comparison across FLTs.

References

1. T. Eisenbarth, R. Koschke, and D. Simon. 2003. Locating features in source code. IEEE Trans. Softw. Eng. 29, 3 (2003), 210--224.
2. B. Dit, M. Revelle, M. Gethers, and D. Poshyvanyk. 2013. Feature location in source code: A taxonomy and survey. J. Software: Evolution and Process 25, 1 (2013), 53--95.
3. D. Poshyvanyk, Y. G. Gueheneuc, A. Marcus, G. Antoniol, and V. Rajlich. 2007. Feature location using probabilistic ranking of methods based on execution scenarios and information retrieval. IEEE Trans. Software Eng. 33, 6 (2007), 420--432.
4. Z. Shi, J. Keung, K. E. Bennin, and X. Zhang. 2018. Comparing learning to rank techniques in hybrid bug localization. Appl. Soft Comput. J. 62, 636--648.
5. F. Angerer, H. Prhofer, D. Lettner, A. Grimmer, and P. Grnbacher. 2014. Identifying inactive code in product lines with configuration-aware system dependence graphs. In Proceedings of the 18th International Software Product Line Conference - Volume 1. 52--61.
6. D. Shepherd, L. Pollock, and T. Tourw. 2005. Using language clues to discover crosscutting concerns. SIGSOFT Softw. Eng. Notes 30, 4 (2005), 1--6.
7. A. D. Lucia, F. Fasano, R. Oliveto, and G. Tortora. 2007. Recovering traceability links in software artifact management systems using information retrieval methods. ACM Trans. Softw. Eng. Methodol. 16, 4 (2007), 13.
8. M. Eaddy, T. Zimmermann, K. D. Sherwood, V. Garg, G. C. Murphy, N. Nagappan, and A. V. Aho. 2008. Do crosscutting concerns cause defects? IEEE Trans. Softw. Eng. 34, 4 (2008), 497--515.
9. A. Panichella, B. Dit, R. Oliveto, M. D. Penta, D. Poshyvanyk, and A. D. Lucia. 2013. How to effectively use topic models for software engineering tasks? An approach based on genetic algorithms. In Proceedings of the 2013 International Conference on Software Engineering. 522--531.
10. S. Simmons, D. Edwards, N. Wilde, J. Homan, and M. Groble. 2006. Industrial tools for the feature location problem: An exploratory study. J. Software Maint. Evolut. Res. Pract. 18, 6 (2006), 457--474.
11. J. Rubin and M. Chechik. 2013. A survey of feature location techniques. In Domain Engineering: Product Lines, Languages, and Conceptual Models. Springer, Berlin, 29--58.
12. D. Binkley, D. Lawrie, C. Uehlinger, and D. Heinz. 2015. Enabling improved IR-based feature location. J. Syst. Software 101, 30--42.
13. S. W. Thomas, M. Nagappan, D. Blostein, and A. E. Hassan. 2013. The impact of classifier configuration and classifier combination on bug localization. IEEE Trans. Software Eng. 39, 10 (2013), 1427--1443.
14. N. Wilde, M. Buckellew, H. Page, V. Rajlich, and L. Pounds. 2003. A comparison of methods for locating features in legacy software. J. Syst. Softw. 65, 2 (2003), 105--114.
15. A. Mahmoud and G. Bradshaw. 2015. Estimating semantic relatedness in source code. ACM Trans. Softw. Eng. Methodol. 25, 1 (2015), 1--35.
16. S. Wang, D. Lo, Z. Xing, and L. Jiang. Concern localization using information retrieval: An empirical study on Linux kernel. In Proceedings of the 18th Working Conference on Reverse Engineering (WCRE'11). IEEE, 92--96.
17. M. P. Robillard. 2008. Topology analysis of software dependencies. ACM Trans. Softw. Eng. Methodol 17, 4 (2008), 1--36.
18. M. R. Robillard and G. C. Murphy. 2002. Concern graphs: Finding and describing concerns using structural program dependencies. In Proceedings of the 24th International Conference on Software Engineering. ACM, 406--416.
19. X. Ye, R. Bunescu, and C. Liu. 2016. Mapping bug reports to relevant files: A ranking model, a fine-grained benchmark, and feature evaluation. IEEE Trans. Software Eng. 42, 4 (2016), 379--402.
20. B. Dit, M. Wagner, S. Wen, W. Wang, M. Linares-V, D. Poshyvanyk, and H. Kagdi. 2014. ImpactMiner: A tool for change impact analysis. In Companion Proceedings of the 36th International Conference on Software Engineering. 540--543.
21. N. Wilde, M. Buckellew, H. Page, and V. Rajlich. A case study of feature location in unstructured legacy Fortran code. 68--76.
22. D. Poshyvanyk, M. Gethers, and A. Marcus. 2013. Concept location using formal concept analysis and information retrieval. ACM Trans. Softw. Eng. Methodol. 21, 4 (2013), 1--34, 2013.
23. C. Kästner, A. Dreiling, and K. Ostermann. 2014. Variability mining: consistent semi-automatic detection of product-line features. IEEE Trans. Software Eng. 40, 1 (2014), 67--82.
24. H. Kagdi, M. Gethers, and D. Poshyvanyk. 2013. Integrating conceptual and logical couplings for change impact analysis in software. Empirical Software Eng. 18, 5 (2013), 933--969.
25. P. Rovegård, L. Angelis, and C. Wohlin. 2008. An empirical study on views of importance of change impact analysis issues. IEEE Trans. Software Eng. 34, 4 (2008), 516--530.
26. X. Sun, X. Liu, B. Li, Y. Duan, H. Yang, and J. Hu. 2016. Exploring topic models in software engineering data analysis: A survey. In Proceedings of the 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD'16). IEEE, 357--362.
27. J. Buckley, J. Rosik, S. Herold, A. Wasala, G. Botterweck, and C. Exton. 2016. FLINTS: A tool for architectural-level modeling of features in software systems. In Proceedings of the 10th European Conference on Software Architecture Workshops. 1--7.
28. C. McMillan, D. Poshyvanyk, M. Grechanik, Q. Xie, and C. Fu. 2013. Portfolio: Searching for relevant functions and their usages in millions of lines of code. ACM Trans. Softw. Eng. Methodol. 22, 4 (2013), 1--30.
29. X. Xie, D. Poshyvanyk, and A. Marcus. 2006. 3D visualization for concept location in source code. In Proceedings of the 28th International Conference on Software Engineering. 839--842.
30. M. Petrenko and V. Rajlich. 2009. Variable granularity for improving precision of impact analysis. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09). IEEE, 10--19.
31. B. Cornelissen, A. Zaidman, A. van Deursen, L. Moonen, and R. Koschke. 2009. A systematic survey of program comprehension through dynamic analysis. IEEE Trans. Software Eng. 35, 5 (2009), 684--702.
32. S. Rao and A. Kak. 2011. Retrieval from software libraries for bug localization: A comparative study of generic and composite text models. In Proceedings of the 8th Working Conference on Mining Software Repositories. 43--52.
33. J. Zhou, H. Zhang, and D. Lo. 2012. Where should the bugs be fixed? -- More accurate information retrieval-based bug localization based on bug reports. In Proceedings of the 34th International Conference on Software Engineering. 14--24.
34. B. Cleary, C. Exton, J. Buckley, and M. English. 2009. An empirical analysis of information retrieval based concept location techniques in software comprehension. Empirical Software Eng. 14, 1 (2009), 93--130.
35. A. Mahmoud and N. Niu. 2015. On the role of semantics in automated requirements tracing. Requirements Eng. 20, 3 (2015), 281--300.
36. B. Dit, M. Revelle, and D. Poshyvanyk. 2013. Integrating information retrieval, execution and link analysis algorithms to improve feature location in software. Empirical Software Eng. 18, 2 (2013), 277--309.
37. K. Saha, M. Lease, S. Khurshid, and D. E. Perry. 2013. Improving bug localization using structured information retrieval. In Proceedings of the IEEE/ACM 28th International Conference on Automated Software Engineering (ASE'13). IEEE, 345--355.
38. G. Tóth, P. Hegedűs, Á. Beszédes, T. Gyimóthy, and J. Jász. 2010. Comparison of different impact analysis methods and programmer's opinion: An empirical study. In Proceedings of the 8th International Conference on the Principles and Practice of Programming in Java. 109--118.
39. M. Revelle and D. Poshyvanyk. 2009. An exploratory study on assessing feature location techniques. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC'09). IEEE, 218--222.
40. S. K. Lukins, N. A. Kraft, and L. H. Etzkorn. 2010. Bug localization using latent Dirichlet allocation. Inf. Software Technol. 52, 9 (2010), 972--990.
41. A. T. Nguyen, T. T. Nguyen, J. Al-Kofahi, H. V. Nguyen, and T. N. Nguyen. 2011. A topic-based approach for narrowing the search space of buggy files from a bug report. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering. 263--272.
42. G. Antoniol, G. Canfora, G. Casazza, A. D. Lucia, and E. Merlo. 2002. Recovering traceability links between code and documentation. IEEE Trans. Softw. Eng. 28, 10 (2002), 970--983.
43. A. Marcus and J. I. Maletic. 2003. Recovering documentation-to-source-code traceability links using latent semantic indexing. In Proceedings of the 25th International Conference on Software Engineering. 125--135.
44. B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, 2009. Systematic literature reviews in software engineering -- A systematic literature review. Inf. Software Technol. 51, 1 (2009), 7--15.
45. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, Italy. 68--77.
46. B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. E. Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28, 8 (2002), 721--734.
47. S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test case generation. IEEE Transactions on Software Engineering 36, 6 (2010), 742--762.
48. B. Li, X. Sun, H. Leung, and S. Zhang. 2013. A survey of code‐based change impact analysis techniques. Software Testing, Verification and Reliability 23, 8 (2013), 613--646.
49. C. M. Lott and H. D. Rombach. 1996. Repeatable software engineering experiments for comparing defect-detection techniques. Empirical Software Engineering 1, 3 (1996), 241--277.
50. P. Heck, and A. Zaidman. 2014. Horizontal traceability for just-in-time requirements: The case for open source feature requests. Journal of Software: Evolution and Process 26, 12 (2014), 1280--1296.
51. G. Gay, S. Haiduc, A. Marcus, and T. Menzies. 2009. On the use of relevance feedback in IR-based concept location. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM'09). IEEE, 351--360.
52. O. S. Gómez, N. Juristo, and S. Vegas. 2014. Understanding replication of experiments in software engineering: A classification. Information and Software Technology 56, 8 (2014), 1033--1048.
53. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wessln. 2012. Experimentation in Software Engineering. Springer Publishing Company, Inc., Berlin 2012.
54. N. Juristo and A. M. Moreno. 2010. Basics of Software Engineering Experimentation. Springer Publishing Company, Inc., 2010.
55. M. Galster, D. Weyns, D. Tofan, B. Michalik, and P. Avgeriou. 2014. Variability in software systems -- A systematic literature review. IEEE Transactions on Software Engineering 40, 3 (2014), 282--306.
56. C. Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. 1--10.
57. J. L. Fleiss and J. Cohen. 1973. The equivalence of weighted Kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 33, 3 (1973), 613--619.
58. J. Sim and C. C. Wright. 2005. The Kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy 85, 3 (2005), 257--268.
59. J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159--174.
60. A. Y. Yao. 2001. CVSSearch: Searching through source code using CVS Comments. In Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01). 364.
61. M. Chochlov, M. English, and J. Buckley. 2017. A historical, textual analysis approach to feature location. Information and Software Technology 88, 110--126.
62. D. Diaz, G. Bavota, A. Marcus, R. Oliveto, S. Takahashi, and A. D. Lucia. 2013. Using code ownership to improve IR-based Traceability Link Recovery. In Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13). IEEE, 123--132.
63. T. D. B. Le, S. Wang, and D. Lo. 2013. Multi-abstraction concern localization. In Proceedings of the IEEE International Conference on Software Maintenance. IEEE, 364--367.
64. M. Borg, P. Runeson, and A. Ardö. 2014. Recovering from a decade: A systematic mapping of information retrieval approaches to software traceability. Empirical Software Engineering 19, 6 (2014), 1565--1616.
65. V. Dallmeier and T. Zimmermann. 2007. Extraction of bug localization benchmarks from history. In Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering. 433--436.
66. L. M. Pickard, B. A. Kitchenham, and P. W. Jones. 1998. Combining empirical results in software engineering. Information and Software Technology 40, 14 (1998), 811--821.
67. M. P. Robillard and G. C. Murphy. 2007. Representing concerns in source code. ACM Trans. Softw. Eng. Methodol. 16, 1 (2007), 3.
68. T. Savage, M. Revelle, and D. Poshyvanyk. 2010. FLAT 3: Feature location and textual tracing tool. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2. 255--258.
69. M. Gethers, R. Oliveto, D. Poshyvanyk, and A. D. Lucia. On integrating orthogonal information retrieval methods to improve traceability recovery. 133--142.
70. R. K. Saha, J. Lawall, S. Khurshid, and D. E. Perry. 2014. On the effectiveness of information retrieval based bug localization for C programs. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME'14). IEEE, 161--170.
71. M. Revelle, M. Gethers, and D. Poshyvanyk. 2011. Using structural and textual information to capture feature coupling in object-oriented software. Empirical Software Engineering 16, 6 (2011), 773--811.
72. B. Bassett and N. A. Kraft. 2013. Structural information based term weighting in text retrieval for feature location. In Proceedings of the IEEE 21st International Conference on Program Comprehension (ICPC'13). IEEE, 133--141.
73. S. Wang and D. Lo. 2014. Version history, similar report, and structure: putting them together for improved bug localization. In Proceedings of the 22nd International Conference on Program Comprehension. 53--63.
74. M. M. Carey and G. C. Gannod. 2007. Recovering concepts from source code with automated concept identification. In Proceedings of the 15th IEEE International Conference on Program Comprehension (ICPC'07). IEEE, 27--36.
75. T. M. Meyers and D. Binkley. 2007. An empirical study of slice-based cohesion and coupling metrics. ACM Trans. Softw. Eng. Methodol 17, 1 (2007), 1--27.
76. D. Liu, A. Marcus, D. Poshyvanyk, and V. Rajlich. 2007. Feature location via information retrieval based filtering of a single scenario execution trace. In Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering. 234--243.
77. D. Poshyvanyk, A. Marcus, R. Ferenc, and T. Gyimóthy. 2009. Using information retrieval based coupling measures for impact analysis. Empirical Software Engineering 14, 1 (2009), 5--32.
78. B. Dit, A. Holtzhauer, D. Poshyvanyk, and H. Kagdi. 2013. A dataset from change history to support evaluation of software maintenance tasks. In Proceedings of the 10th Working Conference on Mining Software Repositories. 131--134.
79. D. Kim, Y. Tao, S. Kim, and A. Zeller. 2013. Where should we fix this bug? A two-phase recommendation model. IEEE Transactions on Software Engineering 39, 11 (2013), 1597--1610.
80. Y. Shin, J. H. Hayes, and J. Cleland-Huang. 2012. A framework for evaluating traceability benchmark metrics. Technical Reports. 21. https://via.library.depaul.edu/tr/21.
81. E. Hill, A. Bacchelli, D. Binkley, B. Dit, D. Lawrie, and R. Oliveto. Which feature location technique is better? 408--411.
82. S. Wang, D. Lo, and J. Lawall. Compositional vector space models for improved bug localization. 171--180.
83. B. Sisman and A. C. Kak. 2013. Assisting code search with automatic query reformulation for bug localization. In Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, 309--318.
84. L. Moreno, G. Bavota, S. Haiduc, M. D. Penta, R. Oliveto, B. Russo, and A. Marcus. 2015. Query-based configuration of text retrieval solutions for software engineering tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 567--578.
85. C. Mills, G. Bavota, S. Haiduc, R. Oliveto, A. Marcus, and A. D. Lucia. 2017. Predicting query quality for applications of text retrieval to software engineering tasks. ACM Trans. Softw. Eng. Methodol 26, 1 (2017), 1--45.
86. S. K. Lukins, N. A. Kraft, and L. H. Etzkorn. 2008. Source code retrieval for bug localization using latent Dirichlet allocation. In Proceedings of the 15th Working Conference on Reverse Engineering. IEEE, 155--164.
87. M. Würsch, G. Ghezzi, G. Reif, and H. C. Gall. 2010. Supporting developers with natural language queries. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1. 165--174.
88. J. C. Carver, N. Juristo, M. T. Baldassarre, and S. Vegas. 2014. Replications of software engineering experiments. Empirical Softw. Eng. 19, 2 (2014), 267--276.
89. A. Kuhn, S. Ducasse, and T. Gîrba. 2007. Semantic clustering: Identifying topics in source code. Inf. Softw. Technol. 49, 3 (2007), 230--243.
90. J. I. Maletic and M. L. Collard. 2015. Exploration, analysis, and manipulation of source code using srcML. In Proceedings of the 37th International Conference on Software Engineering, Vol. 2. IEEE Press, 951--952.
91. X. Peng, Z. Xing, X. Tan, Y. Yu, and W. Zhao. 2011. Iterative context-aware feature location (NIER track). In Proceedings of the 33rd International Conference on Software Engineering. 900--903.
92. G. Scanniello, A. Marcus, and D. Pascale. 2015. Link analysis algorithms for static concept location: An empirical assessment. Empirical Software Engineering 20, 6 (2015), 1666--1720.
93. N. Ali, Y. G. Guéhéneuc, and G. Antoniol. 2013. Trustrace: Mining software repositories to improve the accuracy of requirement traceability links. IEEE Transactions on Software Engineering 39, 5 (2013), 725--741.
94. F. Shull, J. Singer, and D. I. K. Sjberg. 2007. Guide to Advanced Empirical Software Engineering: Springer-Verlag Inc, New York, 2007.
95. A. Marcus, A. Sergeyev, V. Rajlich, and J. I. Maletic. 2004. An information retrieval approach to concept location in source code. In Proceedings of the 11th Working Conference on Reverse Engineering. IEEE, 214--223.
96. E. Hill, D. Shepherd, and L. Pollock. 2015. Exploring the use of concern element role information in feature location evaluation. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. 140--150.
97. H. Cai and R. Santelices. 2016. Method-level program dependence abstraction and its application to impact analysis. J. Syst. Software 122, 311--326.
98. M. Cataldo, A. Mockus, J. A. Roberts, and J. D. Herbsleb. 2009. Software dependencies, work dependencies, and their impact on failures. IEEE Trans. Software Eng. 35, 6 (2009), 864--878.
99. K. Chen and V. Rajlich. 2000. Case study of feature location using dependence graph. In Proceedings of the 8th International Workshop on Program Comprehension (IWPC'00). IEEE, 241--247.
100. L. R. Biggers, C. Bocovich, R. Capshaw, B. P. Eddy, L. H. Etzkorn, and N. A. Kraft. 2014. Configuring latent Dirichlet allocation-based feature location. Empirical Software Eng. 19, 3 (2014), 465--500.
101. B. Dit, L. Guerrouj, D. Poshyvanyk, and G. Antoniol. Can better identifier splitting techniques help feature location? 11--20.
102. A. Panichella, B. Dit, R. Oliveto, M. D. Penta, D. Poshyvanyk, and A. D. Lucia. 2016. Parameterizing and assembling IR-based solutions for SE tasks using genetic algorithms. In Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER'16). IEEE, 314--325.
103. R. D. Peng. 2011. Reproducible research in computational science. Science 334, 6060 (2011), 1226--1227.
104. B. Dit, E. Moritz, M. Linares-Vásquez, D. Poshyvanyk, and J. Cleland-Huang. 2015. Supporting and accelerating reproducible empirical research in software evolution and maintenance using TraceLab Component Library. Empirical Software Eng. 20, 5 (2015), 1198--1236.
105. S. Zamani, S. P. Lee, R. Shokripour, and J. Anvik. 2014. A noun-based approach to feature location using time-aware term-weighting. Inf. Software Technol. 56, 8 (2014), 991--1011.
106. A. Jedlitschka and D. Pfahl. Reporting guidelines for controlled experiments in software engineering. 10 pp.
107. Y. Xue, Z. Xing, and S. Jarzabek. 2012. Feature location in a collection of product variants. In Proceedings of the 19th Working Conference on Reverse Engineering (WCRE'12). IEEE, 145--154.
108. T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell. 2012. A systematic literature review on fault prediction performance in software engineering. IEEE Trans. Software Eng. 38, 6 (2012), 1276--1304.
109. B. Kitchenham, H. Al-Khilidar, M. A. Babar, M. Berry, K. Cox, J. Keung, F. Kurniawati, M. Staples, H. Zhang, and L. Zhu. 2008. Evaluating guidelines for reporting empirical software engineering studies. Empirical Software Eng. 13, 1 (2008), 97--121.
110. C. Collberg and T. A. Proebsting. 2016. Repeatability in computer systems research. Communications of the ACM 59, 3 (2016), 62--69.
111. A. Hosny. 2016. Is your research reproducible? XRDS: Crossroads, The ACM Magazine for Students 22, 4 (2016), 14--15.

Index Terms

• Software and its engineering

  • Software notations and tools

    • Software maintenance tools