Automatic test factoring for java

ABSTRACT

Test factoring creates fast, focused unit tests from slow system-wide tests; each new unit test exercises only a subset of the functionality exercised by the system test. Augmenting a test suite with factored unit tests should catch errors earlier in a test run.One way to factor a test is to introduce mock objects. If a test exercises a component T, which interacts with another component E (the "environment"), the implementation of E can be replaced by a mock. The mock checks that T's calls to E are as expected, and it simulates E's behavior in response. We introduce an automatic technique for test factoring. Given a system test for T and E, and a record of T's and E's behavior when the system test is run, test factoring generates unit tests for T in which E is mocked. The factored tests can isolate bugs in T from bugs in E and, if E is slow or expensive, improve test performance or cost.Our implementation of automatic dynamic test factoring for the Java language reduces the running time of a system test suite by up to an order of magnitude.

REFERENCES

• 1: Michael D. Ernst , Jake Cockrell , William G. Griswold , David Notkin, Dynamically Discovering Likely Program Invariants to Support Program Evolution, IEEE Transactions on Software Engineering, v.27 n.2, p.99-123, February 2001 [doi>10.1109/32.908957 ]
• 2: Michael Factor , Assaf Schuster , Konstantin Shagin, Instrumentation of standard libraries in object-oriented languages: the twin class hierarchy approach, Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 24-28, 2004, Vancouver, BC, Canada
• 3: Michael Feathers, Working Effectively with Legacy Code, Prentice Hall PTR, Upper Saddle River, NJ, 2004
• 4: Refactoring: improving the design of existing code, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 5: M. Jean Harrold , Rajiv Gupta , Mary Lou Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.3, p.270-285, July 1993 [doi>10.1145/152388.152391]
• 6: H. K. N. Leung and L. White. Insights into regression testing. In ICSM, pages 60--69, Oct. 1989.
• 7: Alessandro Orso , Bryan Kennedy, Selective capture and replay of program executions, Proceedings of the third international workshop on Dynamic analysis, p.1-7, May 17-17, 2005, St. Louis, Missouri
• 8: Thomas Ostrand , Aaron Anodide , Herbert Foster , Tarak Goradia, A visual test development environment for GUI systems, Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis, p.82-92, March 02-04, 1998, Clearwater Beach, Florida, United States
• 9: Rational Robot. http://www-306.ibm.com/software/awdtools/tester/robot/.
• 10: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955 ]
• 11: Gregg Rothermel , Roland J. Untch , Chengyun Chu, Prioritizing Test Cases For Regression Testing, IEEE Transactions on Software Engineering, v.27 n.10, p.929-948, October 2001 [doi>10.1109/32.962562 ]
• 12: David Saff , Michael D. Ernst, Reducing wasted development time via continuous testing, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.281, November 17-21, 2003
• 13: David Saff , Michael D. Ernst, Mock object creation for test factoring, Proceedings of the ACM-SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, June 07-08, 2004, Washington DC, USA [doi>10.1145/996821.996838]
• 14: D. Saff and M. D. Ernst. Continuous testing in Eclipse. In 2nd Eclipse Technology Exchange Workshop (eTX), Barcelona, Spain, Mar. 2004.
• 15: SilkTest. http://www.segue.com/products/functional-regressional-testing/silktest.asp.
• 16: Amitabh Srivastava , Jay Thiagarajan, Effectively prioritizing tests in development environment, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 17: Mercury WinRunner. http://www.mercury.com/us/products/quality-center/functional-testing/winrunner/.
• 18: W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Algorithms, Experimentation, Performance, Verification

Keywords:
mock objects, test factoring, unit testing