Towards building a solid empirical body of knowledge in testing techniques

ABSTRACT

Testing technique-related empirical studies have been performed for 25 years. We have managed to accumulate a fair number of experiments in this time, which might lead us to think that we now could have a sizeable empirically backed body of knowledge (BoK) on testing techniques. However, the experiments in this field have some flaws, and, consequently, the empirical BoK we have on testing techniques is far from solid. In this paper, we use the results of a survey that we did on empirical testing techniques studies to identify and discuss solutions that could lead to the formation of a solid empirical BoK. The solutions are related to two fundamental experimental issues: (1) the rigorousness of the experimental design and analysis, and (2) the need for a series of community-wide agreements to coordinate empirical research and assure that studies ratify and complement each other.

REFERENCES

• 1: Victor R. Basili , Richard W. Selby, Comparing the effectiveness of software testing strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1, 1987 [doi>10.1109/TSE.1987.232881]
• 2: James M. Bieman , Janet L. Schultz, An empirical evaluation (and specification) of the all-du-paths testing criterion, Software Engineering Journal, v.7 n.1, p.43-51, Jan. 1992
• 3: Phyllis G. Frankl , Oleg Iakounenko, Further empirical studies of test effectiveness, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, p.153-162, November 01-05, 1998, Lake Buena Vista, Florida, United States
• 4: Phyllis G. Frankl , Stewart N. Weiss , Cang Hu, All-uses vs mutation testing: an experimental comparison of effectiveness, Journal of Systems and Software, v.38 n.3, p.235-253, Sept. 1997 [doi>10.1016/S0164-1212(96)00154-9 ]
• 5: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581 ]
• 6: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 7: Natalia Juristo , Ana M. Moreno , Sira Vegas, Reviewing 25 Years of Testing Technique Experiments, Empirical Software Engineering, v.9 n.1-2, p.7-44, March 2004
• 8: Erik Kamsties , Christopher M. Lott, An Empirical Evaluation of Three Defect-Detection Techniques, Proceedings of the 5th European Software Engineering Conference, p.362-383, September 25-28, 1995
• 9: A. Jefferson Offutt , Ammei Lee , Gregg Rothermel , Roland H. Untch , Christian Zapf, An experimental determination of sufficient mutant operators, ACM Transactions on Software Engineering and Methodology (TOSEM), v.5 n.2, p.99-118, April 1996 [doi>10.1145/227607.227610]
• 10: A. J. Offutt , S. D. Lee, An Empirical Evaluation of Weak Mutation, IEEE Transactions on Software Engineering, v.20 n.5, p.337-344, May 1994 [doi>10.1109/32.286422 ]
• 11: S. Vegas, N. Juristo, V. Basili. Identifying Relevant Information for Testing Technique Selection. An Instantiated Characterisation Schema. Kluwer 2003.
• 12: Elaine J. Weyuker, The Cost of Data Flow Testing: An Empirical Study, IEEE Transactions on Software Engineering, v.16 n.2, p.121-128, February 1990 [doi>10.1109/32.44376 ]
• 13: E. Wong and A. P. Mathur. Fault Detection Effectiveness of Mutation and Data-flow Testing. Software Quality Journal. Volume 4. Pages 69--83. 1995.
• 14: Murray Wood , Marc Roper , Andrew Brooks , James Miller, Comparing and combining software defect detection techniques: a replicated empirical study, Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering, p.262-277, September 22-25, 1997, Zurich, Switzerland

INDEX TERMS

Primary Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features

General Terms:
Experimentation, Verification

Keywords:
Keywords are your own designated keywords