A comparative study of coarse- and fine-grained safe regression test-selection techniques

ABSTRACT

Regression test-selection techniques reduce the cost of regression testing by selecting a subset of an existing test suite to use in retesting a modified program. Over the past two decades, numerous regression test-selection techniques have been described in the literature. Initial empirical studies of some of these techniques have suggested that they can indeed benefit testers, but so far, few studies have empirically compared different techniques. In this paper, we present the results of a comparative empirical study of two safe regression test-selection techniques. The techniques we studied have been implemented as the tools DejaVu and TestTube; we compared these tools in terms of a cost model incorporating precision (ability to eliminate unnecessary test cases), analysis cost, and test execution cost. Our results indicate, that in many instances, despite its relative lack of precision, TestTube can reduce the time required for regression testing as much as the more precise DejaVu. In other instances, particularly where the time required to execute test cases is long, DejaVu's superior precision gives it a clear advantage over TestTube. Such variations in relative performance can complicate a tester's choice of which tool to use. Our experimental results suggest that a hybrid regression test-selection tool that combines features of TestTube and DejaVu may be an answer to these complications; we present an initial case study that demonstrates the potential benefit of such a tool.

REFERENCES

• 1: Hiralal Agrawal , Joseph R. Horgan , Edward W. Krauser , Saul London, Incremental Regression Testing, Proceedings of the Conference on Software Maintenance, p.348-357, September 01, 1993
• 2: M. Balcer , W. Hasling , T. Ostrand, Automatic generation of test scripts from formal test specifications, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.210-218, December 13-15, 1989, Key West, Florida, United States
• 3: Thomas Ball, On the limit of control flow analysis for regression test selection, Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis, p.134-142, March 02-04, 1998, Clearwater Beach, Florida, United States
• 4: BENEDUSI, P., CIMITILE, A., AND DE CARLINI, U. 1988. Post-maintenance testing based on path change analysis. In Proceedings of the Conference on Software Maintenance (Oct.). 352-361.
• 5: David Binkley, Semantics Guided Regression Test Cost Reduction, IEEE Transactions on Software Engineering, v.23 n.8, p.498-516, August 1997 [doi>10.1109/32.624306 ]
• 6: Yih-Farn Chen , David S. Rosenblum , Kiem-Phong Vo, TestTube: a system for selective regression testing, Proceedings of the 16th international conference on Software engineering, p.211-220, May 16-21, 1994, Sorrento, Italy
• 7: Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, Proceedings of the 20th international conference on Software engineering, p.188-197, April 19-25, 1998, Kyoto, Japan
• 8: GUPTA, R., HARROLD,M.J.,AND SOFFA, M. 1992. An approach to regression testing using slicing. In Proceedings of the 1992 Conference on Software Maintenance (Nov.). 299-308.
• 9: HARROLD,M.J.AND ROTHERMEL, G. 1997. Aristotle: A system for research on and development of program-analysis-based tools. OSU-CISRC-3/97-TR17. Ohio State University, Columbus, OH.
• 10: HARROLD,M.J.AND SOFFA, M. L. 1988. An incremental approach to unit testing during maintenance. In Proceedings of the Conference on Software Maintenance (Oct.). 362-367.
• 11: Mary Jean Harrold , David Rosenblum , Gregg Rothermel , Elaine Weyuker, Empirical Studies of a Prediction Model for Regression Test Selection, IEEE Transactions on Software Engineering, v.27 n.3, p.248-263, March 2001 [doi>10.1109/32.910860 ]
• 12: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 13: LASKI,J.AND SZERMER, W. 1992. Identification of program modifications and its applications in software maintentance. In Proceedings of the 1992 Conference on Software Maintenance (Nov.). 282-290.
• 14: LEUNG,H.AND WHITE, L. 1990. A study of integration testing and software regression at the integration level. In Proceedings of the Conference on Software Maintenance. 290-300.
• 15: LEUNG,H.AND WHITE, L. 1991. A cost model to compare regression test strategies. In Proceedings of the Conference on Software Maintenance (Oct.). 201-208.
• 16: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 17: ROSENBLUM,D.AND ROTHERMEL, G. 1997. A comparative study of regression test-selection techniques. In Proceedings of the International Workshop on Empirical Studies of Software Maintenance (Oct.). 89-94.
• 18: David S. Rosenblum , Elaine J. Weyuker, Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies, IEEE Transactions on Software Engineering, v.23 n.3, p.146-156, March 1997 [doi>10.1109/32.585502 ]
• 19: Gregg Rothermel , Mary Jean Harrold, Efficient, effective regression testing using safe test selection techniques, 1996
• 20: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955 ]
• 21: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 22: Gregg Rothermel , Mary Jean Harrold, Empirical Studies of a Safe Regression Test Selection Technique, IEEE Transactions on Software Engineering, v.24 n.6, p.401-419, June 1998 [doi>10.1109/32.689399 ]
• 23: F. I. Vokolos , P. G. Frankl, Pythia: a regression test selection tool based on textual differencing, IFIP TC5 WG5.4 3rd internatinal conference on on Reliability, quality and safety of software-intensive systems, p.3-21, January 1997, Athens, Greece
• 24: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998
• 25: WHITE,L.AND LEUNG, H. 1992. A firewall concept for both control-flow and data-flow in regression integration testing. In Proceedings of the 1992 Conference on Software Maintenance (Nov.). 262-270.
• 26: Lee J. White , Venkatasubramaniam Narayanswamy , Ted Friedman , Marc Kirschenbaum , Paul Piwowarski , Mitsuru Oha, Test Manager: A Regression Testing Tool, Proceedings of the Conference on Software Maintenance, p.338-347, September 01, 1993

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Algorithms, Verification

Keywords:
regression test selection, regression testing