Automated test case generation for spreadsheets

ABSTRACT

Spreadsheet languages, which include commercial spreadsheets and various research systems, have had a substantial impact on end-user computing. Research shows, however, that spreadsheets often contain faults. Thus, in previous work, we presented a methodology that assists spreadsheet users in testing their spreadsheet formulas. Our empirical studies have shown that this methodology can help end-users test spreadsheets more adequately and efficiently; however, the process of generating test cases can still represent a significant impediment. To address this problem, we have been investigating how to automate test case generation for spreadsheets in ways that support incremental testing and provide immediate visual feedback. We have utilized two techniques for generating test cases, one involving random selection and one involving a goal-oriented approach. We describe these techniques, and report results of an experiment examining their relative costs and benefits.

REFERENCES

• 1: Nicolas J. Belkin, Helping people find what they don't know, Communications of the ACM, v.43 n.8, p.58-61, Aug. 2000 [doi>10.1145/345124.345143]
• 2: David L Bird , Carlos Urias Munoz, Automatic generation of random self-checking test cases, IBM Systems Journal, v.22 n.3, p.229-245, 1983
• 3: Polly S. Brown , John D. Gould, An experimental study of people creating spreadsheets, ACM Transactions on Information Systems (TOIS), v.5 n.3, p.258-272, July 1987 [doi>10.1145/27641.28058]
• 4: Margaret Burnett , John Atwood , Rebecca Walpole Djang , James Reichwein , Herkimer Gottfried , Sherry Yang, Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm, Journal of Functional Programming, v.11 n.2, p.155-206, March 2001
• 5: Ed Huai-hsin Chi , Joseph Konstan , Phillip Barry , John Riedl, A spreadsheet approach to information visualization, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.79-80, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263513]
• 6: L. Clarke. A system to generate test data and symbolically execute programs. IEEE Transactions on Software Engineering, 2(3):215-222, Sept. 1976.
• 7: C. Corritore, B. Kracher, and S. Wiedenbeck. Trust in the online environment. In Proceedings of HCI International, pages 1548-1552, Aug. 2001.
• 8: Richard A. DeMillo , A. Jefferson Offutt, Constraint-Based Automatic Test Data Generation, IEEE Transactions on Software Engineering, v.17 n.9, p.900-910, September 1991 [doi>10.1109/32.92910 ]
• 9: E. Duesterwald, R. Gupta, and M. L. Soffa. Rigorous data flow testing through output influences. In Proceedings of the 2nd Irvine Software Symposium, Mar. 1992.
• 10: Roger Ferguson , Bogdan Korel, The chaining approach for software test data generation, ACM Transactions on Software Engineering and Methodology (TOSEM), v.5 n.1, p.63-86, Jan. 1996 [doi>10.1145/226155.226158]
• 11: M. Fisher, M. Cao, D. Brown, G. Rothermel, C. R. Cook, and M. M. Burnett. Integrating automated test case generation into the WYSIWYT spreadsheet testing methodology. Technical Report TR: 02-60-01, Oregon State University, Jan. 2002.
• 12: P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194 ]
• 13: Arnaud Gotlieb , Bernard Botella , Michel Rueher, Automatic test data generation using constraint solving techniques, Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis, p.53-62, March 02-04, 1998, Clearwater Beach, Florida, United States
• 14: B. Korel. A dynamic approach of automated test data generation. In Proceedings of the International Conference on Software Maintenance, pages 311-317, Nov. 1990.
• 15: B. Korel, Automated Software Test Data Generation, IEEE Transactions on Software Engineering, v.16 n.8, p.870-879, August 1990 [doi>10.1109/32.57624 ]
• 16: Jennifer L. Leopold , Allen L. Ambler, Keyboardless Visual Programming Using Voice, Handwriting, and Gesture, Proceedings of the 1997 IEEE Symposium on Visual Languages (VL '97), p.28, April 23-26, 1997
• 17: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 18: A. J. Offutt. An integrated automatic test data generation system. Journal of Systems Integration, 1(3):391-409, Nov. 1991.
• 19: H. D. Pande , W. A. Landi , B. G. Ryder, Interprocedural Def-Use Associations for C Systems with Single Level Pointers, IEEE Transactions on Software Engineering, v.20 n.5, p.385-403, May 1994 [doi>10.1109/32.286418 ]
• 20: Raymond R. Panko, What we know about spreadsheet errors, Journal of End User Computing, v.10 n.2, p.15-21, Spring 1998
• 21: C. Ramamoorthy, S. Ho, and W. Chen. On the automated generation of program test data. IEEE Transactions on Software Engineering, 2(4):293-300, Dec. 1976.
• 22: Gregg Rothermel , Margaret Burnett , Lixin Li , Christopher Dupuis , Andrei Sheretov, A methodology for testing spreadsheets, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.1, p.110-147, Jan. 2001 [doi>10.1145/366378.366385]
• 23: Gregg Rothermel , Lixin Li , Margaret Burnett, Testing Strategies for Form-Based Visual Programs, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.96, November 02-05, 1997
• 24: Gregg Rothermel , Lixin Li , Christopher DuPuis , Margaret Burnett, What you see is what you test: a methodology for testing form-based visual programs, Proceedings of the 20th international conference on Software engineering, p.198-207, April 19-25, 1998, Kyoto, Japan
• 25: Karen J. Rothermel , Curtis R. Cook , Margaret M. Burnett , Justin Schonfeld , T. R. G. Green , Gregg Rothermel, WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation, Proceedings of the 22nd international conference on Software engineering, p.230-239, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337206]
• 26: B. Shneiderman. Designing the User Interface. Addison-Wesley, Reading, MA, 3rd edition, 1998.
• 27: Trevor J. Smedley , Philip T. Cox , Shannon L. Byrne, Expanding the utility of spreadsheets through the integration of visual programming and user interface objects, Proceedings of the workshop on Advanced visual interfaces, May 27-29, 1996, Gubbio, Italy [doi>10.1145/948449.948473]
• 28: E. J. Weyuker, More Experience with Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.9, p.912-919, September 1993 [doi>10.1109/32.241773 ]
• 29: E. M. Wilcox , J. W. Atwood , M. M. Burnett , J. J. Cadiz , C. R. Cook, Does continuous visual feedback aid debugging in direct-manipulation programming systems?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.258-265, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258721]

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.8 PERSONAL COMPUTING
      K.8.1 Application Packages
          Subjects: Spreadsheets

Additional Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.2 Nonnumerical Algorithms and Problems
          Subjects: Computations on discrete structures

  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Spreadsheets

General Terms:
Algorithms, Experimentation, Performance