Daniel J. Farrar
ABSTRACT
The testing of database management software to ensure correct, stable, and efficient operation requires the application not only of fixed regression test cases, but of ad hoc experiments to search for previously-unknown bugs and performance optimizations. This paper describes the Dexterity system, a declarative framework for the rapid description, execution, and documentation of such experiments, using a relational database. In this design, experimental specifications and results are stored in the database, and dependencies between the deliverables of each step of an experiment are modelled as foreign key relationships in the schema (with additional annotations to indicate the type and duration of the dependency relationship). Worker processes performing the testing tasks communicate with the server via stateless web services, and are scheduled by querying the data and schema of the database. This design concentrates the effort of the human experimenter into two phases: an experiment description phase, and a data analysis phase, while allowing (sometimes lengthy) test execution over a large number of dynamic independent variables to be conducted with minimal intervention.
- I. T. Bowman, P. Bumbulis, D. Farrar, A. K. Goel, B. Lucier, A. Nica, G. N. Paulley, J. Smirnios, and M. Young-Lai. SQL Anywhere: A holistic approach to database self-management. In Proceedings, ICDE Workshops (Self-Managing Database Systems), pages 414--423, Istanbul, Turkey, Apr. 2007. ieee Computer Society Press. Google Scholar
Digital Library
- R. T. Fielding and R. N. Taylor. Principled design of the modern web architecture. ACM Trans. Internet Tech., 2(2):115--150, May 2002. Google ScholarDigital Library
- J. Hughes. Why Functional Programming Matters. Computer Journal, 32(2):98--107, 1989. Google ScholarDigital Library
- http://jquery.com.Google Scholar
- J. Spillner and E. Schill. Analysis on inference mechanisms for schema-driven forms generation. http://www.rn.inf.tu-dresden.de/uploads/Publikationen/wsgui-inference.pdf.Google Scholar
Index Terms
- Schema-driven experiment management: declarative testing with dexterity
Recommendations
HYpothesis-driven experiment design in computer simulation studies
WSC '17: Proceedings of the 2017 Winter Simulation ConferenceIn simulation studies, the goal specifies the objective or purpose of the study and thus drives the entire experimentation process. Relevant experiments and respective experiment hypotheses are derived from the study's goal and the model's observed ...
Toward a model-driven engineering framework for reproducible simulation experiment lifecycle management
WSC '14: Proceedings of the 2014 Winter Simulation ConferenceGoal-directed reproducible experimentation with simulation models is still a significant challenge. The underutilization of design of experiments, limited transparency in the collection and analysis of results, and ad-hoc adaptation of experiments as ...
Experiment databases: towards an improved experimental methodology in machine learning
ECMLPKDD'07: Proceedings of the 11th European Conference on Principles and Practice of Knowledge Discovery in DatabasesMachine learning research often has a large experimental component. While the experimental methodology employed in machine learning has improved much over the years, repeatability of experiments and generalizability of results remain a concern. In this ...
Comments