Integrating automated test generation into the WYSIWYT spreadsheet testing methodology

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Integrating automated test generation into the WYSIWYT spreadsheet testing methodology

Full text

Pdf (1.94 MB)

Source

ACM Transactions on Software Engineering and Methodology (TOSEM) archive
Volume 15 , Issue 2 (April 2006) table of contents

Pages: 150 - 194

Year of Publication: 2006

ISSN:1049-331X

Authors

Marc Fisher, II	University of Nebraska Lincoln, Lincoln, NE
Gregg Rothermel	University of Nebraska Lincoln, Lincoln, NE
Darren Brown	Oregon State University, Corvallis, OR
Mingming Cao	Oregon State University, Corvallis, OR
Curtis Cook	Oregon State University, Corvallis, OR
Margaret Burnett	Oregon State University, Corvallis, OR

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 20, Downloads (12 Months): 146, Citation Count: 1

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTex EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1131421.1131423 What is a DOI?

ABSTRACT

Spreadsheet languages, which include commercial spreadsheets and various research systems, have had a substantial impact on end-user computing. Research shows, however, that spreadsheets often contain faults. Thus, in previous work we presented a methodology that helps spreadsheet users test their spreadsheet formulas. Our empirical studies have shown that end users can use this methodology to test spreadsheets more adequately and efficiently; however, the process of generating test cases can still present a significant impediment. To address this problem, we have been investigating how to incorporate automated test case generation into our testing methodology in ways that support incremental testing and provide immediate visual feedback. We have used two techniques for generating test cases, one involving random selection and one involving a goal-oriented approach. We describe these techniques and their integration into our testing environment, and report results of an experiment examining their effectiveness and efficiency.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Alberto Avritzer , Elaine J. Weyuker, The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software, IEEE Transactions on Software Engineering, v.21 n.9, p.705-716, September 1995 [doi>10.1109/32.464549 ]
	2	André Baresel , David Binkley , Mark Harman , Bogdan Korel, Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
	3	Nicolas J. Belkin, Helping people find what they don't know, Communications of the ACM, v.43 n.8, p.58-61, Aug. 2000 [doi>10.1145/345124.345143]
	4	David L Bird , Carlos Urias Munoz, Automatic generation of random self-checking test cases, IBM Systems Journal, v.22 n.3, p.229-245, 1983
	5	Chandrasekhar Boyapati , Sarfraz Khurshid , Darko Marinov, Korat: automated testing based on Java predicates, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
	6	Brown, D., Burnett, M., Rothermel, G., Fujita, H., and Negoro, F. 2003. Generalizing WYSIWYT visual testing to screen transition languages. In Proceedings of the IEEE Symposium on Human-Centric Computing Languages and Environments (Auckland, New Zealand). IEEE Computer Society Press, Los Alamitos, Calif.
	7	Polly S. Brown , John D. Gould, An experimental study of people creating spreadsheets, ACM Transactions on Information Systems (TOIS), v.5 n.3, p.258-272, July 1987 [doi>10.1145/27641.28058]
	8	Margaret Burnett , John Atwood , Rebecca Walpole Djang , James Reichwein , Herkimer Gottfried , Sherry Yang, Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm, Journal of Functional Programming, v.11 n.2, p.155-206, March 2001
	9	Margaret Burnett , Curtis Cook , Omkar Pendse , Gregg Rothermel , Jay Summet , Chris Wallace, End-user software engineering with assertions in the spreadsheet paradigm, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
	10	Margaret Burnett , Curtis Cook , Gregg Rothermel, End-user software engineering, Communications of the ACM, v.47 n.9, September 2004 [doi>10.1145/1015864.1015889]
	11	Margaret Burnett , Richard Hossli , Tim Pulliam , Brian VanVoorst , Xiaoyang Yang, Toward Visual Programming Languages for Steering Scientific Computations, IEEE Computational Science & Engineering, v.1 n.4, p.44-62, December 1994 [doi>10.1109/99.338768 ]
	12	Margaret Burnett , Andrei Sheretov , Bing Ren , Gregg Rothermel, Testing Homogeneous Spreadsheet Grids with the "What You See Is What You Test" Methodology, IEEE Transactions on Software Engineering, v.28 n.6, p.576-594, June 2002 [doi>10.1109/TSE.2002.1010060 ]
	13	Margaret Burnett , Andrei Sheretov , Gregg Rothermel, Scaling Up a "What You See Is What You Test" Methodology to Spreadsheet Grids, Proceedings of the IEEE Symposium on Visual Languages, p.30, September 13-16, 1999
	14	Juei Chang , Debra J. Richardson, Structural specification-based testing: automated support and experimental evaluation, Proceedings of the 7th European software engineering conference held jointly with the 7th ACM SIGSOFT international symposium on Foundations of software engineering, p.285-302, September 06-10, 1999, Toulouse, France
	15	E. H. -H. Chi , P. Barry , J. Riedl , J. Konstan, A spreadsheet approach to information visualization, Proceedings of the 1997 IEEE Symposium on Information Visualization (InfoVis '97), p.17, October 18-25, 1997
	16	Clarke, L. 1976. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 2, 3 (Sept.), 215--222.
	17	Corritore, C., Kracher, B., and Wiedenbeck, S. 2001. Trust in the online environment. In Proceedings of the HCI International (New Orleans, La). Lawrence Erlbaum, Mahwah, N.J., 1548--1552.
	18	Cullen, D. 2003. Excel snafu costs firm $24m. The Register.
	19	Richard A. DeMillo , A. Jefferson Offutt, Constraint-Based Automatic Test Data Generation, IEEE Transactions on Software Engineering, v.17 n.9, p.900-910, September 1991 [doi>10.1109/32.92910 ]
	20	Duesterwald, E., Gupta, R., and Soffa, M. L. 1992. Rigorous data flow testing through output influences. In Proceedings of the 2nd Irvine Software Symposium (Irvine, Calif). University of California, Irvine.
	21	Michael D. Ernst , Jake Cockrell , William G. Griswold , David Notkin, Dynamically Discovering Likely Program Invariants to Support Program Evolution, IEEE Transactions on Software Engineering, v.27 n.2, p.99-123, February 2001 [doi>10.1109/32.908957 ]
	22	Roger Ferguson , Bogdan Korel, The chaining approach for software test data generation, ACM Transactions on Software Engineering and Methodology (TOSEM), v.5 n.1, p.63-86, Jan. 1996 [doi>10.1145/226155.226158]
	23	Marc Fisher , Mingming Cao , Gregg Rothermel , Curtis R. Cook , Margaret M. Burnett, Automated test case generation for spreadsheets, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581359]
	24	Marc Fisher II , Dalai Jin , Gregg Rothermel , Margaret Burnett, Test Reuse in the Spreadsheet Paradigm, Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE'02), p.257, November 12-15, 2002
	25	Marc Fisher , Gregg Rothermel, The EUSES spreadsheet corpus: a shared resource for supporting experimentation with spreadsheet dependability mechanisms, Proceedings of the first workshop on End-user software engineering, p.1-5, May 21-21, 2005, St. Louis, Missouri
	26	P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194 ]
	27	Arnaud Gotlieb , Bernard Botella , Michel Rueher, Automatic test data generation using constraint solving techniques, Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis, p.53-62, March 02-04, 1998, Clearwater Beach, Florida, United States
	28	Arnaud Gotlieb , Bernard Botella , Michel Rueher, A CLP Framework for Computing Structural Test Data, Proceedings of the First International Conference on Computational Logic, p.399-413, July 01, 2000
	29	Neelam Gupta , Aditya P. Mathur , Mary Lou Soffa, Automated test data generation using an iterative relaxation method, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, p.231-244, November 01-05, 1998, Lake Buena Vista, Florida, United States
	30	Hartman, A. and Nagin, K. 2003. Model driven testing---Agedis architecture interfaces and tools. In Proceedings of the 1st European Conference on Model Driven Software Engineering (Nuremberg). imbus AG, Möhrendorf, Germany, 1--11.
	31	Korel, B. 1990a. A dynamic approach of automated test data generation. In Proceedings of the International Conference on Software Maintenance (San Diego, Calif.). IEEE Computer Society Press, Los Alamitos, Calif., 311--317.
	32	B. Korel, Automated Software Test Data Generation, IEEE Transactions on Software Engineering, v.16 n.8, p.870-879, August 1990 [doi>10.1109/32.57624 ]
	33	Incorporating Incremental Validation and Impact Analysis into Spreadsheet Maintenance: An Empirical Study, Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01), p.72, November 07-09, 2001
	34	Laski, J. and Korel, B. 1993. A data flow oriented program testing strategy. IEEE Trans. Softw. Eng. 9, 3 (May), 347--354.
	35	Jennifer L. Leopold , Allen L. Ambler, Keyboardless Visual Programming Using Voice, Handwriting, and Gesture, Proceedings of the 1997 IEEE Symposium on Visual Languages (VL '97), p.28, April 23-26, 1997
	36	Darko Marinov , Sarfraz Khurshid, TestEra: A Novel Framework for Automated Testing of Java Programs, Proceedings of the 16th IEEE international conference on Automated software engineering, p.22, November 26-29, 2001
	37	Martina Marré , Antonia Bertolino, Reducing and estimating the cost of test coverage criteria, Proceedings of the 18th international conference on Software engineering, p.486-494, March 25-29, 1996, Berlin, Germany
	38	Martina Marré , Antonia Bertolino, Unconstrained duals and their use in achieving all-uses coverage, Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis, p.147-157, January 08-10, 1996, San Diego, California, United States
	39	C. C. Michael , G. McGraw , M. A. Schatz, Generating Software Test Data by Evolution, IEEE Transactions on Software Engineering, v.27 n.12, p.1085-1110, December 2001 [doi>10.1109/32.988709 ]
	40	Brad A. Myers, Graphical techniques in a spreadsheet for specifying user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.243-249, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108903]
	41	Offutt, A. 1991. An integrated automatic test data generation system. J. Syst. Integration 1, 3 (Nov.), 391--409.
	42	Offutt, J. and Abdurazik, A. 1999. Generating tests from uml specifications. In Proceedings of the International Conference on the Unified Modeling Language (Fort Collins, Colo.). Springer Verlag, New York.
	43	H. D. Pande , W. A. Landi , B. G. Ryder, Interprocedural Def-Use Associations for C Systems with Single Level Pointers, IEEE Transactions on Software Engineering, v.20 n.5, p.385-403, May 1994 [doi>10.1109/32.286418 ]
	44	Panko, R. 1995. Finding spreadsheet errors: Most spreadsheet errors have design flaws that may lead to long-term miscalculation. Information Week, 100.
	45	Raymond R. Panko, What we know about spreadsheet errors, Journal of End User Computing, v.10 n.2, p.15-21, Spring 1998
	46	Ramamoorthy, C., Ho, S., and Chen, W. 1976. On the automated generation of program test data. IEEE Trans. Softw. Eng. 2, 4 (Dec.), 293--300.
	47	Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
	48	Orna Raz , Philip Koopman , Mary Shaw, Semantic anomaly detection in online data sources, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581378]
	49	Gregg Rothermel , Margaret Burnett , Lixin Li , Christopher Dupuis , Andrei Sheretov, A methodology for testing spreadsheets, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.1, p.110-147, Jan. 2001 [doi>10.1145/366378.366385]
	50	Gregg Rothermel , Lixin Li , Margaret Burnett, Testing Strategies for Form-Based Visual Programs, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.96, November 02-05, 1997
	51	Gregg Rothermel , Lixin Li , Christopher DuPuis , Margaret Burnett, What you see is what you test: a methodology for testing form-based visual programs, Proceedings of the 20th international conference on Software engineering, p.198-207, April 19-25, 1998, Kyoto, Japan
	52	Karen J. Rothermel , Curtis R. Cook , Margaret M. Burnett , Justin Schonfeld , T. R. G. Green , Gregg Rothermel, WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation, Proceedings of the 22nd international conference on Software engineering, p.230-239, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337206]
	53	Scott, A. 2003. Shurgard stock dives after auditor quits over company's accounting. The Seattle Times.
	54	Trevor J. Smedley , Philip T. Cox , Shannon L. Byrne, Expanding the utility of spreadsheets through the integration of visual programming and user interface objects, Proceedings of the workshop on Advanced visual interfaces, May 27-29, 1996, Gubbio, Italy [doi>10.1145/948449.948473]
	55	Smith, R. 2004. University of Toledo loses $2.4m in projected revenue. The Toledo Blade.
	56	Nguyen Tran Sy , Yves Deville, Consistency techniques for interprocedural test data generation, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
	57	Viehstaedt, G. and Ambler, A. 1992. Visual representation and manipulation of matrices. J. Visual Lang. Comput. 3, 3 (Sept.), 273--298.
	58	Willem Visser , Corina S. Pǎsǎreanu , Sarfraz Khurshid, Test input generation with java PathFinder, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
	59	E. J. Weyuker, More Experience with Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.9, p.912-919, September 1993 [doi>10.1109/32.241773 ]
	60	E. M. Wilcox , J. W. Atwood , M. M. Burnett , J. J. Cadiz , C. R. Cook, Does continuous visual feedback aid debugging in direct-manipulation programming systems?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.258-265, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258721]
	61	Aaron Wilson , Margaret Burnett , Laura Beckwith , Orion Granatir , Ledah Casburn , Curtis Cook , Mike Durham , Gregg Rothermel, Harnessing curiosity to increase correctness in end-user programming, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642665]