Are fit tables really talking?: a series of experiments to understand whether fit tables are useful during evolution tasks

ABSTRACT

Test-driven software development tackles the problem of operationally defining the features to be implemented by means of test cases. This approach was recently ported to the early development phase, when requirements are gathered and clarified. Among the existing proposals, Fit (Framework for Integrated Testing) supports the precise specification of requirements by means of so called Fit tables, which express relevant usage scenarios in a tabular format, easily understood also by the customer. Fit tables can be turned into executable test cases through the creation of pieces of glue code, called fixtures.

In this paper, we test the claimed benefits of Fit through a series of three controlled experiments in which Fit tables and related fixtures are used to clarify a set of change requirements, in a software evolution scenario. Results indicate improved correctness achieved with no significant impact on time, however benefits of Fit vary in a substantial way depending on the developers' experience. Preliminary results on the usage of Fit in combination with pair programming revealed another relevant source of variation.

REFERENCES

• 1: J. Aarniala. Acceptance testing. In whitepaper. www.cs.helsinki.fi/u/jaarnial/jaarnial-testing.pdf, October 30 2006.
• 2: G. Bache, J. Andersson, and P. Sutton. Xp with acceptance test driven development - a rewrite project for a resource optimization system. In Proceedings of the 8th International Conference on Agile Processes in Software Engineering and eXtreme Programming (XP 2003), pages 180--188. Springer, 2003.
• 3: J. Cohen. Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates, Hillsdale, NJ, 1988.
• 4: C. Deng, P. Wilson, and F. Maurer. Fitclipse: A fit-based eclipse plug-in for executable acceptance test driven development. In Proceedings of the 8th International Conference on Agile Processes in Software Engineering and eXtreme Programming (XP 2007). Springer, 2007.
• 5: N. Juristo and A. Moreno. Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Englewood Cliffs, NJ, 2001.
• 6: Karl R. P. H. Leung , W. L. Yeung, Generating User Acceptance Test Plans from Test Cases, Proceedings of the 31st Annual International Computer Software and Applications Conference - Vol. 2- (COMPSAC 2007), p.737-742, July 24-27, 2007 [doi>10.1109/COMPSAC.2007.125]
• 7: Grigori Melnik , Frank Maurer , Mike Chiasson, Executable Acceptance Tests for Communicating Business Requirements: Customer Perspective, Proceedings of the conference on AGILE 2006, p.35-46, July 23-28, 2006 [doi>10.1109/AGILE.2006.26]
• 8: G. Melnik, K. Read, and F. Maurer. Suitability of fit user acceptance tests for specifying functional requirements: Developer perspective. In Extreme programming and agile methods - XP/Agile Universe 2004, pages 60--72, August 2004.
• 9: B. Meyer, On Formalism in Specifications, IEEE Software, v.2 n.1, p.6-26, January 1985 [doi>10.1109/MS.1985.229776]
• 10: Rick Mugridge , Ward Cunningham, Fit for Developing Software: Framework for Integrated Tests (Robert C. Martin), Prentice Hall PTR, Upper Saddle River, NJ, 2005
• 11: A. N. Oppenheim. Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London, 1992.
• 12: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 13: K. Read, G. Melnik, and F. Maurer. Examining usage patters of the fit acceptance testing framework. In Proc. 6th International Conference on eXtreme Programming and Agile Processes in Software Engineering (XP2005), pages Lecture Notes in Computer Science, Vol. 3556, Springer Verlag: 127--136 2005, June 18-23 2005.
• 14: Filippo Ricca , Marco Torchiano , Mariano Ceccato , Paolo Tonella, Talking tests: an empirical assessment of the role of fit acceptance tests in clarifying requirements, Ninth international workshop on Principles of software evolution: in conjunction with the 6th ESEC/FSE joint meeting, September 03-04, 2007, Dubrovnik, Croatia [doi>10.1145/1294948.1294962]
• 15: F. Ricca, M. Torchiano, M. Di Penta, M. Ceccato, and P. Tonella. The use of executable fit tables to support maintenance and evolution tasks. Electronic Communications of the EASST, 8, 2008.
• 16: Jacques Philippe Sauvé , Osório Lopes Abath Neto , Walfredo Cirne, EasyAccept: a tool to easily create, run and drive development with automated acceptance tests, Proceedings of the 2006 international workshop on Automation of software test, May 23-23, 2006, Shanghai, China [doi>10.1145/1138929.1138951]
• 17: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000
• 18: Ralph R. Young, Effective requirements practices, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2001

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
          Subjects: Methodologies (e.g., object-oriented, structured)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
          Subjects: Tools

General Terms:
Experimentation, Measurement

Keywords:
acceptance test, empirical studies, software maintenance