skip to main content
10.1145/1368088.1368116acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

An empirical study of the effects of test-suite reduction on fault localization

Published: 10 May 2008 Publication History

Abstract

Fault-localization techniques that utilize information about all test cases in a test suite have been presented. These techniques use various approaches to identify the likely faulty part(s) of a program, based on information about the execution of the program with the test suite. Researchers have begun to investigate the impact that the composition of the test suite has on the effectiveness of these fault-localization techniques. In this paper, we present the first experiment on one aspect of test-suite composition--test-suite reduction. Our experiment studies the impact of the test-suite reduction on the effectiveness of fault-localization techniques. In our experiment, we apply 10 test-suite reduction strategies to test suites for eight subject programs. We then measure the differences between the effectiveness of four existing fault-localization techniques on the unreduced and reduced test suites. We also measure the reduction in test-suite size of the 10 test-suite reduction strategies. Our experiment shows that fault-localization effectiveness varies depending on the test-suite reduction strategy used, and it demonstrates the trade-offs between test-suite reduction and fault-localization effectiveness.

References

[1]
R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference, Practice and Research Techniques, Windsor, UK, September 2007.
[2]
H. Agrawal, J. Horgan, S. London, and W. Wong. Fault localization using execution slices and dataflow tests. In Proceedings of IEEE Software Reliability Engineering, pages 143--151, 1995.
[3]
Aristotle Research Group. Aristotle analysis system, 2007. http://www.cc.gatech.edu/aristotle/.
[4]
B. Baudrey, F. Fleurey, and Y. L. Traon. Improving test suites for efficient fault localization. In International Conference on Software Engineering, pages 82--91, Shanghai, China, May 2006.
[5]
T. Y. Chen and M. F. Lau. Dividing strategies for the optimization of a test suite. Information Processing Letters, 60(3):135--141, Mar. 1996.
[6]
H. Cleve and A. Zeller. Locating causes of program failures. In Proceedings of the International Conference on Software Engineering, pages 342--351, St. Louis, Missouri, May 2005.
[7]
D. Hao, Y. Pan, L. Zhang, W. Zhao, H. Mei, and J. Sun. A similarity-aware approach to testing based fault localization. In Proceedings of the Conference on Automated Software Engineering, pages 291--294, November 2005.
[8]
M. J. Harrold, R. Gupta, and M. L. Soffa. A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology, 2(3):270--285, July 1993.
[9]
J. R. Horgan and S. A. London. ATAC: A data flow coverage testing tool for C. In Proceedings of the Symp. on Assessment of Quality Software Development Tools, pages 2--10, May 1992.
[10]
M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the International Conference on Software Engineering, pages 191--200, May 1994.
[11]
J. Jones and M. J. Harrold. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the International Conference on Automated Software Engineering, pages 273--282, November 2005.
[12]
J. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the International Conference on Software Engineering, pages 467--477, Orlando, Florida, May 2002.
[13]
B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In Proceedings of the conference on Programming language design and implementation, pages 15--26, 2005.
[14]
C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff. SOBER: statistical model-based bug localization. In Proceedings of European Software Engineering Conference and Foundations on Software Engineering, pages 286--295, September 2005.
[15]
J. Offutt, J. Pan, and J. M. Voas. Procedures for reducing the size of coverage-based test sets. In Proceedings of the International Conference on Testing Comp. Software, pages 111--123, June 1995.
[16]
M. Renieris and S. Reiss. Fault localization with nearest neighbor queries. In Proceedings of the International Conference on Automated Software Engineering, pages 30--39, Montreal, Quebec, October 2003.
[17]
F. Vokolos and P. Frankl. Empirical evaluation of the textual differencing regression testing techniques. In Proceedings of the International Conference on Software Maintenance, November 1998.

Cited By

View all
  • (2024)Racing on the negative forceProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699137(4229-4246)Online publication date: 14-Aug-2024
  • (2024)HetFL: Heterogeneous Graph-Based Software Fault LocalizationIEEE Transactions on Software Engineering10.1109/TSE.2024.345460550:11(2884-2905)Online publication date: 1-Nov-2024
  • (2024)Isolating Compiler Bugs by Generating Effective Witness Programs with Large Language ModelsIEEE Transactions on Software Engineering10.1109/TSE.2024.3397822(1-20)Online publication date: 2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '08: Proceedings of the 30th international conference on Software engineering
May 2008
558 pages
ISBN:9781605580791
DOI:10.1145/1368088
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 May 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. empirical study
  2. fault localization
  3. test-suite reduction

Qualifiers

  • Research-article

Conference

ICSE '08
Sponsor:

Acceptance Rates

ICSE '08 Paper Acceptance Rate 56 of 370 submissions, 15%;
Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)30
  • Downloads (Last 6 weeks)4
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Racing on the negative forceProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699137(4229-4246)Online publication date: 14-Aug-2024
  • (2024)HetFL: Heterogeneous Graph-Based Software Fault LocalizationIEEE Transactions on Software Engineering10.1109/TSE.2024.345460550:11(2884-2905)Online publication date: 1-Nov-2024
  • (2024)Isolating Compiler Bugs by Generating Effective Witness Programs with Large Language ModelsIEEE Transactions on Software Engineering10.1109/TSE.2024.3397822(1-20)Online publication date: 2024
  • (2023)The MADAG Strategy for Fault Location TechniquesApplied Sciences10.3390/app1302081913:2(819)Online publication date: 6-Jan-2023
  • (2023)Information Retrieval-Based Fault Localization for Concurrent Programs2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00122(1467-1479)Online publication date: 11-Sep-2023
  • (2023)A fault localization approach based on fault propagation contextInformation and Software Technology10.1016/j.infsof.2023.107245160:COnline publication date: 1-Aug-2023
  • (2023)ContextAug: model-domain failing test augmentation with contextual informationFrontiers of Computer Science10.1007/s11704-023-2521-218:2Online publication date: 13-Sep-2023
  • (2023)Fault localization in DSLTrans model transformations by combining symbolic execution and spectrum-based analysisSoftware and Systems Modeling10.1007/s10270-023-01123-323:3(737-763)Online publication date: 29-Sep-2023
  • (2023)Automation of User Interface Testing by Reinforcement Learning-Based Monkey AgentsComplex Computational Ecosystems10.1007/978-3-031-44355-8_1(3-15)Online publication date: 26-Oct-2023
  • (2023)Spectrum‐Based Techniques for Software Fault LocalizationHandbook of Software Fault Localization10.1002/9781119880929.ch4(201-270)Online publication date: 21-Apr-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media