The type of evidence produced by empirical software engineers

ABSTRACT

This paper reports on the research published between the years 1997 and 2003 inclusive in the journal of Empirical Software Engineering, drawing on the taxonomy developed by Glass et al. in [3]. We found that the research was somewhat narrow in topic with about half the papers focusing on measurement/metrics, review and inspection; that researchers were almost as interested in formulating as in evaluating; that hypothesis testing and laboratory experiments dominated evaluations; that research was not very likely to focus on people and extremely unlikely to refer to other disciplines. We discuss our findings in the context of making empirical software engineering more relevant to practitioners.

REFERENCES

• 1: Boehm , Richard Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2003
• 2: Alistair Cockburn, Agile software development, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 3: Glass, R. L., Vessey, I., Ramesh, V. Research in software engineering: an analysis of the literature. Information and Software Technology, 44, 491--506, 2002.
• 4: A. S. Lee, A scientific methodology for MIS case studies, MIS Quarterly, v.13 n.1, p.33-50, March 1989 [doi>10.2307/248698]
• 5: Carolyn B. Seaman, Qualitative Methods in Empirical Studies of Software Engineering, IEEE Transactions on Software Engineering, v.25 n.4, p.557-572, July 1999 [doi>10.1109/32.799955 ]
• 6: Judith Segal, The Nature of Evidence in Empirical Software Engineering, Proceedings of the Eleventh Annual International Workshop on Software Technology and Engineering Practice (STEP'03), p.40-47, September 19-21, 2003
• 7: Susan Elliott Sim , Janice Singer , Margaret-Anne Storey, Beg, Borrow, or Steal: Using Multidisciplinary Approachesin Empirical Software Engineering Research, Empirical Software Engineering, v.6 n.1, p.85-93, March 2001 [doi>10.1023/A:1009809824225 ]
• 8: Walter F. Tichy , Paul Lukowicz , Lutz Prechelt , Ernst A. Heinz, Experimental evaluation in computer science: a quantitative study, Journal of Systems and Software, v.28 n.1, p.9-18, Jan. 1995 [doi>10.1016/0164-1212(94)00111-Y ]
• 9: Walter F. Tichy, Hints for Reviewing Empirical Work in Software Engineering, Empirical Software Engineering, v.5 n.4, p.309-312, December 2000 [doi>10.1023/A:1009844119158 ]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.0 General

General Terms:
Experimentation

Keywords:
empirical software engineering, evidence, field studies, research taxonomy