skip to main content
10.1145/1062455.1062531acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
Article

An empirical evaluation of test case filtering techniques based on exercising complex information flows

Published: 15 May 2005 Publication History

Abstract

Some software defects trigger failures only when certain complex information flows occur within the software. Profiling and analyzing such flows therefore provides a potentially important basis for filtering test cases. We report the results of an empirical evaluation of several test case filtering techniques that are based on exercising complex information flows. Both coverage-based and profile-distribution-based filtering techniques are considered. They are compared to filtering techniques based on exercising basic blocks, branches, function calls, and def-use pairs, with respect to their effectiveness for revealing defects.

References

[1]
Agrawal H., Horgan J., Krauser E., London S. Incremental Regression Testing. Proceedings of the IEEE Conference on Software Maintenance (Montreal, Canada, 1993).
[2]
Bates, S. and Horwitz, S. Incremental Program Testing Using Program Dependence Graphs. 20th ACM Symposium on Principles of Programming Languages (January 1993), 384--396.
[3]
Clarke, L. A., Podgurski, A., Richardson, D. J., and Zeil, S. J. A formal evaluation of data flow path selection criteria. IEEE Transactions on Software Engineering, Vol. 15, No. 11 (November 1989), 1381--1332.
[4]
Denning D.E. and Denning P.J. Certification of programs for secure information flow. Communication of the ACM 20, 7 (1977), 504--513.
[5]
Dickinson, W., Leon, D., and Podgurski, A. Finding failures by cluster analysis of execution profiles. 23rd Intl. Conf. on Software Engineering (Toronto, May 2001), 339--348.
[6]
Dickinson, W., Leon, D., and Podgurski, A. Pursuing failure: the distribution of program failures in a profile space. 10th European Software Engineering Conf. and 9th ACM SIGSOFT Symp. on the Foundations of Software Engineering (Vienna, September 2001), ACM Press, 246--255.
[7]
Elbaum, S., Malishevsky, A.G., and Rothermel, G. Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering 28, 2 (February 2002), 159--182.
[8]
Ferrante J., Ottenstein K.J., and Warren J.D. The Program Dependence Graph and its Use in Optimization. ACM Transactions on Programming Languages and Systems 9, 3 (October 1987), 319--349.
[9]
Frankl, P. and Iakounenko, O. Further Empirical Studies of Test Effectiveness. 6th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, Nov. 1998), 153--162.
[10]
Frankl, P. G. and Y. Deng. A Comparison of Delivered Reliability of Branch, Data Flow, and Operational Testing: A Case Study. 2000 International Symposium on Software Testing and Analysis (Portland, OR, August 2000), 124--134.
[11]
Graves, T. L, Harrold, M. J., Kim, J. M., Porter, A., and Rothermel, G. An empirical study of regression test selection techniques. ACM Transactions on Software Engineering and Methodology 10, 2 (April 2001), 184--208.
[12]
Gupta, R., Harrold, M. J., and Soffa, M. L. Program Slicing-Based Regression Testing Techniques. Journal of Software Testing, Verification, and Reliability 6, 2 (June 1996), 83--112.
[13]
Harold, M. J., Rothermel, G., Sayre, K., Wu, R., and Yi, L. An Empirical Investigation of the Relationship Between Spectra Differences and Regression Faults. Journal of Softare Testing, Verification, and Reliability, 10, 3 (September 2000).
[14]
Hochbaum, D. S. (editor). Approximation algorithms for NP-hard problems. PWS Publishing, Boston, MA, 1997.
[15]
Jacks, IBM, Jacks Project, www.ibm.com/developerworks/oss/cvs/jacks/, 2002.
[16]
Java Language Specification, Sun Microsystems, java.sun.com/docs/books/jls/second_edition/html/j.title.doc.html, 2000.
[17]
javac, Sun Microsystems Inc., Java 2 Platform, Standard Edition, java.sun.com/j2se/1.3/, 1995-2002.
[18]
JTidy, jtidy.sourceforge.net, World Wide Web Consortium (Massachusetts Institute of Technology, Institut National de Recherche en Informatique et en Automatique, Keio University), 1998-2000.
[19]
Kim, J. M. and Porter, A. A history-based test prioritization technique for regression testing in resource constrained environments. 2002 International Conference on Software Engineering (Orlando, FL, May 2002).
[20]
Korel, B. The Program Dependence Graph in Static Program Testing. Information Processing Letters 24 (January 1987), 103--108.
[21]
Korel B. and Laski J. Dynamic Program Slicing. Information Processing Letters 29 (October 1988), 155--163.
[22]
Korel B. and Yalamanchili S. Forward Computation of Dynamic Program Slices. ISSTA (1994), 66--79.
[23]
Laski, J. W. and Korel, B. A Data Flow Oriented Program Testing Strategy. IEEE Transactions on Software Engineering 9, 3 (May 1983), 347--354.
[24]
Leon, D. and Podgurski, A. A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases. International Symposium on Software Reliability Engineering (Denver, CO, November 2003), 442--454.
[25]
Leon, D., Podgurski, A., and White, L.J. Multivariate visualization in observation-based testing. Proceedings of the 22nd International Conference on Software Engineering (Limerick, Ireland, June 2000), ACM Press, 116--125.
[26]
Masri, W., Podgurski, A., and Leon, D. Detecting and Debugging Insecure Information Flows. 15th. IEEE International Symposium on Software Reliability Engineering, ISSRE 2004. St. Malo, France Nov 2-5, 2004.
[27]
Ntafos, S. C. On Required Element Testing. IEEE Transactions on Software Engineering 10, 6 (November 1984), 795--803.
[28]
Podgurski A. and Clarke L. A Formal Model of Program Dependencies and its Implications for Software Testing, Debugging, and Maintenance. IEEE TSE, 16(9):965--979, September 1990.
[29]
Podgurski. A., Leon, D., Francis, P., Masri, W., Minch, M., Sun, J. and Wang, B. Automated support for classifying software failure reports. To appear in 2003 International Conference on Software Engineering (Portland, OR, May 2003).
[30]
Rapps, S. and E. J. Weyuker. Selecting Software Test Data Using Data Flow Information. IEEE Transactions on Software Engineering 11, 4 (April 1985), 367--375.
[31]
Rothermel, G. and Harrold, M.J. A Safe, Efficient Algorithm for Regression Test Selection, 1993 Conference on Software Maintenance (September 1993), pages 358--367.
[32]
Rothermel, G. and Harrold, M. J. Selecting Tests and Identifying Test Coverage Requirements for Modified Software. 1994 International Symposium on Software Testing and Analysis (August 1994), 169--184.
[33]
Rothermel, G., Untch, R., Chu, C., and Harrold, M.J. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering 27, 10 (October 2001), 929--948.
[34]
Thompson, M. C., Richardson, D. J., and Clarke, L. A. An Information Flow Model of Fault Detection. International Symposium on Software Testing and Analysis (Cambridge, MA, June 1993), 182--192.
[35]
Weiser M. Program Slicing. IEEE Transactions On Software Engineering 10, 4 (1984), 352--357.
[36]
Wong, W. E., Horgan, J. R., London, S., and Mathur, A. P. Effect of test set size minimization and fault detection effectiveness. Software Practice and Experience 28, 4 (April 1998), 347--369.
[37]
Wong, W. E., Horgan, J. R., Mathur, A. P., and Pasquini, A. Test set size minimization and fault detection effectiveness: a case study in a space application. 21st Annual International Computer Software and Applications Conference (Washington, D.C., August 1997), 522--528.
[38]
Xerces. The Apache XML Project: xml.apache.org/xerces-j.
[39]
XML Conformance Test Suite. www.w3.org/XML/Test

Cited By

View all
  • (2018)A Software Quality Testing Evaluation Method and Its Power Grid ApplicationInternational Conference on Applications and Techniques in Cyber Security and Intelligence ATCI 201810.1007/978-3-319-98776-7_42(400-404)Online publication date: 5-Nov-2018
  • (2013)On the adoption of MC/DC and control-flow adequacy for a tight integration of program testing and statistical fault localizationInformation and Software Technology10.1016/j.infsof.2012.10.00155:5(897-917)Online publication date: 1-May-2013
  • (2012)Comparing the Effectiveness of Equivalence Partitioning, Branch Testing and Code Reading by Stepwise Abstraction Applied by SubjectsProceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation10.1109/ICST.2012.113(330-339)Online publication date: 17-Apr-2012
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '05: Proceedings of the 27th international conference on Software engineering
May 2005
754 pages
ISBN:1581139632
DOI:10.1145/1062455
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 May 2005

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. dynamic information flow analysis
  2. dynamic slicing
  3. observation-based testing
  4. program dependences
  5. software testing
  6. test case filtering

Qualifiers

  • Article

Conference

ICSE05
Sponsor:

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2018)A Software Quality Testing Evaluation Method and Its Power Grid ApplicationInternational Conference on Applications and Techniques in Cyber Security and Intelligence ATCI 201810.1007/978-3-319-98776-7_42(400-404)Online publication date: 5-Nov-2018
  • (2013)On the adoption of MC/DC and control-flow adequacy for a tight integration of program testing and statistical fault localizationInformation and Software Technology10.1016/j.infsof.2012.10.00155:5(897-917)Online publication date: 1-May-2013
  • (2012)Comparing the Effectiveness of Equivalence Partitioning, Branch Testing and Code Reading by Stepwise Abstraction Applied by SubjectsProceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation10.1109/ICST.2012.113(330-339)Online publication date: 17-Apr-2012
  • (2012)How well does test case prioritization integrate with statistical fault localization?Information and Software Technology10.1016/j.infsof.2012.01.00654:7(739-758)Online publication date: 1-Jul-2012
  • (2011)Intrusive Test Automation with Failed Test Case ClusteringProceedings of the 2011 18th Asia-Pacific Software Engineering Conference10.1109/APSEC.2011.31(89-96)Online publication date: 5-Dec-2011
  • (2009)Algorithms and tool support for dynamic information flow analysisInformation and Software Technology10.1016/j.infsof.2008.05.00851:2(385-404)Online publication date: 1-Feb-2009
  • (2008)Experiments with test case prioritization using relevant slicesJournal of Systems and Software10.5555/1326359.132642381:2(196-221)Online publication date: 1-Feb-2008
  • (2008)Experiments with test case prioritization using relevant slicesJournal of Systems and Software10.1016/j.jss.2007.05.00681:2(196-221)Online publication date: Feb-2008
  • (2008)Locating dependence structures using search-based slicingInformation and Software Technology10.1016/j.infsof.2007.11.00150:12(1189-1209)Online publication date: 1-Nov-2008
  • (2008)Application-based anomaly intrusion detection with dynamic information flow analysisComputers and Security10.1016/j.cose.2008.06.00227:5-6(176-187)Online publication date: 1-Oct-2008
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media