Automated replay and failure detection for web applications

ABSTRACT

User-session-based testing of web applications gathers user sessions to create and continually update test suites based on real user input in the field. To support this approach during maintenance and beta testing phases, we have built an automated framework for testing web-based software that focuses on scalability and evolving the test suite automatically as the application's operational profile changes. This paper reports on the automation of the replay and oracle components for web applications, which pose issues beyond those in the equivalent testing steps for traditional, stand-alone applications. Concurrency, nondeterminism, dependence on persistent state and previous user sessions, a complex application infrastructure, and a large number of output formats necessitate developing different replay and oracle comparator operators, which have tradeoffs in fault detection effectiveness, precision of analysis, and efficiency. We have designed, implemented, and evaluated a set of automated replay techniques and oracle comparators for user-session-based testing of web applications. This paper describes the issues, algorithms, heuristics, and an experimental case study with user sessions for two web applications. From our results, we conclude that testers performing user-session-based testing should consider their expectations for program coverage and fault detection when choosing a replay and oracle technique.

REFERENCES

• 1: A. Andrews, J. Offutt, and R. Alexander. Testing web applications by modeling with FSMs. Software Systems and Modeling, 4(2), April 2005.
• 2: J. H. Andrews , L. C. Briand , Y. Labiche, Is mutation an appropriate tool for testing experiments?, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062530]
• 3: L. Baresi and M. Young. Test oracles. Technical Report CIS-TR01-02, University of Oregon, 2001.
• 4: Robert V. Binder, Testing object-oriented systems: models, patterns, and tools, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 5: J. Callahan, F. Schneider, and S. Easterbrook. Automated software testing using model-checking. In SPIN Wksp, 1996.
• 6: David Chays , Yuetang Deng , Phyllis G. Frankl , Saikat Dan , Filippos I. Vokolos , Elaine J. Weyuker, An AGENDA for testing relational database applications: Research Articles, Software Testing, Verification & Reliability, v.14 n.1, p.17-44, March 2004 [doi>10.1002/stvr.v14:1]
• 7: Clover: Code coverage tool for Java. <http://www.cenqua.com/clover/>, 2005.
• 8: Laura K. Dillon , Qing Yu, Oracles for checking temporal properties of concurrent systems, Proceedings of the 2nd ACM SIGSOFT symposium on Foundations of software engineering, p.140-153, December 06-09, 1994, New Orleans, Louisiana, United States
• 9: Sebastian Elbaum , Srikanth Karre , Gregg Rothermel, Improving web application testing with user session data, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 10: Sebastian Elbaum , Gregg Rothermel , Srikanth Karre , Marc Fisher II, Leveraging User-Session Data to Support Web Application Testing, IEEE Transactions on Software Engineering, v.31 n.3, p.187-202, March 2005 [doi>10.1109/TSE.2005.36]
• 11: Open source web applications with source code. <http://www.gotocode.com>, 2003.
• 12: M. Jean Harrold , Rajiv Gupta , Mary Lou Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.3, p.270-285, July 1993 [doi>10.1145/152388.152391]
• 13: Chris Hunter , Paul Strooper, Systematically deriving partial oracles for testing concurrent programs, Proceedings of the 24th Australasian conference on Computer science, p.83-91, January 29-February 02, 2001, Gold Coast, Queensland, Australia
• 14: Ivar Jacobson, The use-case construct in object-oriented software engineering, Scenario-based design: envisioning work and technology in system development, John Wiley & Sons, Inc., New York, NY, 1995
• 15: Engin Kirda , Mehdi Jazayeri , Clemens Kerer , Markus Schranz, Experiences in Engineering Flexible Web Services, IEEE MultiMedia, v.8 n.1, p.58-65, January 2001 [doi>10.1109/93.923954 ]
• 16: B. Long, D. Hoffman, and P. Strooper. Tool support for testing concurrent Java components. Trans on Software Engineering, 29(6):555--566, June 2003.
• 17: G. Di Lucca , A. Fasolino , F. Faralli, Testing Web Applications, Proceedings of the International Conference on Software Maintenance (ICSM'02), p.310, October 03-06, 2002
• 18: Atif M. Memon , Martha E. Pollack , Mary Lou Soffa, Automated test oracles for GUIs, Proceedings of the 8th ACM SIGSOFT international symposium on Foundations of software engineering: twenty-first century applications, p.30-39, November 06-10, 2000, San Diego, California, United States
• 19: Alessandro Orso , Bryan Kennedy, Selective capture and replay of program executions, Proceedings of the third international workshop on Dynamic analysis, p.1-7, May 17-17, 2005, St. Louis, Missouri
• 20: Parasoft WebKing. <http://www.parsoft.com>, 2004.
• 21: Rational Robot. <http://www-306.ibm.com/software/ awdtools/tester/robot/>, 2005.
• 22: Caucho resin. <http://www.caucho.com/resin/>, 2005.
• 23: Debra J. Richardson, TAOS: Testing with Analysis and Oracle Support, Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis, p.138-153, August 17-19, 1994, Seattle, Washington, United States [doi>10.1145/186258.187158]
• 24: Debra J. Richardson , Stephanie Leif Aha , T. Owen O'Malley, Specification-based test oracles for reactive systems, Proceedings of the 14th international conference on Software engineering, p.105-118, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143100]
• 25: Sreedevi Sampath , Valentin Mihaylov , Amie Souter , Lori Pollock, Composing a Framework to Automate Testing of Operational Web-Based Software, Proceedings of the 20th IEEE International Conference on Software Maintenance, p.104-113, September 11-14, 2004
• 26: Sreedevi Sampath , Valentin Mihaylov , Amie Souter , Lori Pollock, A Scalable Approach to User-Session based Testing of Web Applications through Concept Analysis, Proceedings of the 19th IEEE international conference on Automated software engineering, p.132-141, September 20-24, 2004 [doi>10.1109/ASE.2004.6]
• 27: Jessica Sant , Amie Souter , Lloyd Greenwald, An exploration of statistical models for automated test case generation, Proceedings of the third international workshop on Dynamic analysis, p.1-7, May 17-17, 2005, St. Louis, Missouri
• 28: Sara Sprenkle , Sreedevi Sampath , Emily Gibson , Lori Pollock , Amie Souter, An Empirical Comparison of Test Suite Reduction Techniques for User-Session-Based Testing of Web Applications, Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05), p.587-596, September 25-30, 2005 [doi>10.1109/ICSM.2005.18]
• 29: S. D. Stoller. Model-checking multi-threaded distributed Java programs. Int Journal on Soft Tools for Technology Transfer, 4(1):71--91, Oct. 2002.
• 30: E. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465--70, 1982.
• 31: GNU wget. <http://www.gnu.org/software/wget/>, 2005.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Experimentation, Reliability

Keywords:
replay, software testing, test oracles, web applications