Article

An experimental comparison of four test suite reduction techniques

Authors:

Hong MeiAuthors Info & Claims

ICSE '06: Proceedings of the 28th international conference on Software engineering

Pages 636 - 640

https://doi.org/10.1145/1134285.1134380

Published: 28 May 2006 Publication History

Abstract

As a test suite usually contains redundancy, a subset of the test suite (representative set) may still satisfy all the test objectives. As the redundancy increases the cost of executing the test suite, many test suite reduction techniques have been brought out in spite of the NP-completeness of the general problem of finding the optimal representative set of the test suite. In the literature, some experimental studies of test suite reduction techniques have already been reported, but there are still shortcomings of the studies of these techniques. This paper presents an experimental comparison of the four typical test suite reduction techniques: heuristic H, heuristic GRE, genetic algorithm-based approach and ILP-based approach. The aim of the study is to provide a guideline for choosing the appropriate test suite reduction techniques.

References

[1]

J. Black, E Melachrinoudis, and D. Kaeli. "Bi-Criteria Models for All-Uses Test Suite Reduction," International Conference on Software Engineering, 2004, pp. 106--115.

Digital Library

[2]

T.Y. Chen and M.F. Lau, "A New Heuristic for Test Suite Reduction," Information and Software Technology, Vol. 40, No. 5, 1998, pp. 347--354.

[3]

T.Y. Chen and M.F. Lau, "A Simulation Study on Some Heuristics for Test Suite Reduction," Information and Software Technology, Vol. 40, No. 13, 1998, pp. 777-787.

[4]

M.J. Harrold, R. Gupta, and M.L. Soffa. "A Methodology for Controlling the Size of a Test Suite," ACM Transactions on Software Engineering and Methodology, Vol. 2, No.3, 1993, pp. 270--285.

Digital Library

[5]

M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, "Experiments on the Effectiveness of Dataflow- and Control Flow-Based Test Adequacy Criteria," International Conference on Software Engineering, 1994, pp. 191--200.

Digital Library

[6]

N. Mansour and K. El-Fakin. "Simulated Annealing and Genetic Algorithms for Optimal Regression Testing," Journal of Software Maintenance: Research and Practice, Vol. 11, No. 1, 1999, pp. 19--34.

Digital Library

[7]

T. Ralphs and M. Guzelsoy, "The SYMPHONY Callable Library for Mixed Integer Programming," The Ninth INFORMS Computing Society Conference, 2005, pp. 61--73.

[8]

G. Rothermel, M.J. Harrold, J. von Ronne, and C. Hong. "Empirical Studies of Test-Suite Reduction," Software Testing Verification and Reliability, Vol. 12, No. 4, 2002, pp. 219--249.

[9]

P. Thevenod-Fosse, H. Waeselynck, and Y. Crouzet, "An Experimental Study on Software Structural Testing: Deterministic verses Random Input Generation," IEEE International Symposium on Fault Tolerant Computing, 1991, pp. 410--417.

[10]

M. B. Wall, A Genetic Algorithm for Resource-Constrained Scheduling. MIT, PhD thesis, 1996.

Digital Library

[11]

W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur, "Effect of Test Set Minimization on Fault Detection Effectiveness," Proc. of the 17th International Conference on Software Engineering, 1995, pp. 41--50.

Digital Library

[12]

W.E. Wong, J.R. Horgan, A.P. Mathur, and A. Pasquini, "Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application," Annual International Computer Software and Applications Conference (COMPSAC), 1997, pp. 522--528.

Digital Library

[13]

http://www.cc.gatech.edu/aristotle/Tools/subjects/

[14]

http://xmlppm.sourceforge.net/

[15]

http://www.gnu.org/software/tar/

Cited By

Goyal AShyamasundar RSivakumar GJetley RRamaswamy S(2021)Empirical Analysis of Greedy, GE and GRE HeuristicsProceedings of the 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)10.1145/3452383.3452389(1-11)Online publication date: 25-Feb-2021
https://dl.acm.org/doi/10.1145/3452383.3452389
Alsharif AKapfhammer GMcMinn P(2020)Hybrid Methods for Reducing Database Schema Test SuitesProceedings of the IEEE/ACM 1st International Conference on Automation of Software Test10.1145/3387903.3389305(41-50)Online publication date: 7-Oct-2020
https://dl.acm.org/doi/10.1145/3387903.3389305
Coviello CRomano SScanniello GAntoniol G(2020)GASSERProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3422157(1-6)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3422157
Show More Cited By

Index Terms

An experimental comparison of four test suite reduction techniques
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Software management
        Software maintenance
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

An experimental study of four typical test suite reduction techniques

In software development, developers often rely on testing to reveal bugs. Typically, a test suite should be prepared before initial testing, and new test cases may be added to the test suite during the whole testing process. This may usually cause the ...
Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization ...
Analysis of test suite reduction with enhanced tie-breaking techniques

Test suite minimization techniques try to remove redundant test cases of a test suite. However, reducing the size of a test suite might reduce its ability to reveal faults. In this paper, we present a novel approach for test suite reduction that uses an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '06: Proceedings of the 28th international conference on Software engineering

May 2006

1110 pages

ISBN:1595933751

DOI:10.1145/1134285

General Chair:
Leon J. Osterweil
University of Massachusetts, USA
,
Program Chairs:
Dieter Rombach
TU Kaiserslautern, Germany
,
Mary Lou Soffa
University of Virginia, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 May 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE06

Sponsor:

ICSE06: International Conference on Software Engineering

May 20 - 28, 2006

Shanghai, China

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
699
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Goyal AShyamasundar RSivakumar GJetley RRamaswamy S(2021)Empirical Analysis of Greedy, GE and GRE HeuristicsProceedings of the 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)10.1145/3452383.3452389(1-11)Online publication date: 25-Feb-2021
https://dl.acm.org/doi/10.1145/3452383.3452389
Alsharif AKapfhammer GMcMinn P(2020)Hybrid Methods for Reducing Database Schema Test SuitesProceedings of the IEEE/ACM 1st International Conference on Automation of Software Test10.1145/3387903.3389305(41-50)Online publication date: 7-Oct-2020
https://dl.acm.org/doi/10.1145/3387903.3389305
Coviello CRomano SScanniello GAntoniol G(2020)GASSERProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3422157(1-6)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3422157
Alsharif AKapfhammer GMcMinn P(2020)STICCER: Fast and Effective Database Test Suite Reduction Through Merging of Similar Test Cases2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST46399.2020.00031(220-230)Online publication date: Oct-2020
https://doi.org/10.1109/ICST46399.2020.00031
Marchetto AScanniello GSusi A(2019)Combining Code and Requirements Coverage with Execution Cost for Test Suite ReductionIEEE Transactions on Software Engineering10.1109/TSE.2017.277783145:4(363-390)Online publication date: 1-Apr-2019
https://doi.org/10.1109/TSE.2017.2777831
Liu PLi YLi Z(2019)Some Thoughts on Model-Based Test Optimization2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C)10.1109/QRS-C.2019.00058(268-274)Online publication date: Jul-2019
https://doi.org/10.1109/QRS-C.2019.00058
Mostafa SWang XXie TBultan TSen K(2017)PerfRanker: prioritization of performance regression tests for collection-intensive softwareProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092725(23-34)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092725
Guarnieri MTsankov PBuchs TTorabi Dashti MBasin DBultan TSen K(2017)Test execution checkpointing for web applicationsProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092710(203-214)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092710
Lin CTang KWang JKapfhammer G(2017)Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexityScience of Computer Programming10.1016/j.scico.2017.05.004150:C(1-25)Online publication date: 15-Dec-2017
https://dl.acm.org/doi/10.1016/j.scico.2017.05.004
Sandoval Alcocer JBergel AValente MAvritzer AIosup AZhu XBecker S(2016)Learning from Source Code History to Identify Performance FailuresProceedings of the 7th ACM/SPEC on International Conference on Performance Engineering10.1145/2851553.2851571(37-48)Online publication date: 12-Mar-2016
https://dl.acm.org/doi/10.1145/2851553.2851571
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten