A controlled experiment on white-box database testing

ABSTRACT

Controlled experiments are a powerful way to assess and compare the effectiveness of different techniques. In this paper we present the experimental results of the evaluation of the effectiveness of a structural test coverage criterion developed for SQL queries when used by a tester to guide the selection of database test cases. We describe a controlled experiment designed for comparing this criterion with other conventional criteria such as equivalence partitioning and boundary value analysis. The results show that 1) the use of the structural coverage allows the tester to develop more effective test cases, 2) the effectiveness is higher when considering the kind of faults that are more specifically related to SQL than other kinds of faults, and 3) the results give us some insight into how to improve the coverage criterion.

REFERENCES

• 1: Bowen, P. L. and Rohde, F. H. (2002): Further evidence of the effects of normalization on end-user query errors: an experimental evaluation, International Journal of Accounting Information Systems, 3(4) 255--290.
• 2: Stefan Brass , Christian Goldberg, Semantic errors in SQL queries: a quite complete list, Journal of Systems and Software, v.79 n.5, p.630-644, May 2006 [doi>10.1016/j.jss.2005.06.028]
• 3: Natalia Juristo , Ana M. Moreno , Sira Vegas, Reviewing 25 Years of Testing Technique Experiments, Empirical Software Engineering, v.9 n.1-2, p.7-44, March 2004
• 4: K. N. King , A. Jefferson Offutt, A Fortran language system for mutation-based software testing, Software—Practice & Experience, v.21 n.7, p.685-718, July 1991 [doi>10.1002/spe.4380210704]
• 5: P. Reisner, Use of Psychological Experimentation as an Aid to Development of a Query Language, IEEE Transactions on Software Engineering, v.3 n.3, p.218-229, May 1977 [doi>10.1109/TSE.1977.231131]
• 6: Siau, K. L., Chan, H. C. and Wei, K. L. (2004): Effects of Query Complexity and Learning on Novice User Query Performance With Conceptual and Logical Database Interfaces, IEEE Transactions on Systems, Man and Cybernetics-Part A: Systems and Humans, 34(2) 276--281.
• 7: María José Suárez-Cabal , Javier Tuya, Using an SQL coverage measurement for testing database applications, Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering, October 31-November 06, 2004, Newport Beach, CA, USA
• 8: Suarez-Cabal, M. J. (2006): Improvement of test cases for database applications using coverage measurements of SQL Queries, PhD Thesis, University of Oviedo.
• 9: Javier Tuya , Ma Jose Suarez-Cabal , Claudio de la Riva, SQLMutation: A tool to generate mutants of SQL database queries, Proceedings of the Second Workshop on Mutation Analysis, p.1, November 07-10, 2006 [doi>10.1109/MUTATION.2006.13]
• 10: Javier Tuya , Ma José Suárez-Cabal , Claudio de la Riva, Mutating database queries, Information and Software Technology, v.49 n.4, p.398-417, April, 2007 [doi>10.1016/j.infsof.2006.06.009]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Validation

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.0 General

General Terms:
Design, Management, Verification

Keywords:
SQL testing, controlled experiment, database testing, empirical validation, software testing