Article

An empirical comparison between direct and indirect test result checking approaches

Authors:

T. H. TseAuthors Info & Claims

SOQUA '06: Proceedings of the 3rd international workshop on Software quality assurance

Pages 6 - 13

https://doi.org/10.1145/1188895.1188901

Published: 06 November 2006 Publication History

Abstract

An oracle on software testing is a mechanism for checking whether the system under test has behaved correctly for any executions. In some situations, oracles are unavailable or too expensive to apply. This is known as the oracle problem. It is crucial to develop techniques to address it, and metamorphic testing (MT) was one of such proposals. This paper conducts a controlled experiment to investigate the cost effectiveness of using MT by 38 testers on three open-source programs. The fault detection capability and time cost of MT are compared with the popular assertion checking method. Our results show that MT is cost-efficient and has potentials for detecting more faults than the assertion checking method.

References

[1]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th International Conference on Software Engineering (ICSE 2005), pages 402--411. ACM Press, New York, 2005.

Digital Library

[2]

S. Ar, M. Blum, B. Codenotti, and P. Gemmell. Checking approximate computations over the reals. In Proceedings of the 25th Annual ACM Symposium on Theory of Computing (STOC '93), pages 786--795. ACM Press, New York, 1993.

Digital Library

[3]

B. Beizer. Software Testing Techniques. Van Nostrand Reinhold, New York, 1990.

Digital Library

[4]

R. V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison Wesley, Reading, Massachusetts, 2000.

Digital Library

[5]

M. Blum and S. Kannan. Designing programs that check their work. Journal of the ACM, 42(1):269--291, 1995.

Digital Library

[6]

M. Blum, M. Luby, and R. Rubinfeld. Self-testing/correcting with applications to numerical problems. Journal of Computer and System Sciences, 47(3):549--595, 1993.

Digital Library

[7]

J. F. Bowring, J. M. Rehg, and M. J. Harrold. Active learning for automatic classification of software behavior. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004), pages 195--205. ACM Press, New York, 2004.

Digital Library

[8]

L. C. Briand, M. Di Penta, and Y. Labiche. Assessing and improving state-based class testing: a series of experiments. IEEE Transactions on Software Engineering, 30(11):770--783, 2004.

Digital Library

[9]

F. T. Chan, T. Y. Chen, S. C. Cheung, M. F. Lau, and S. M. Yiu. Application of metamorphic testing in numerical analysis. In Proceedings of the IASTED International Conference on Software Engineering (SE '98), pages 191--197. ACTA Press, Calgary, Canada, 1998.

[10]

W. K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau. A metamorphic approach to integration testing of context-sensitive middleware-based applications. In Proceedings of the 5th International Conference on Quality Software (QSIC 2005), pages 241--249. IEEE Computer Society Press, Los Alamitos, California, 2005.

Digital Library

[11]

W. K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau. Integration testing of context-sensitive middleware-based applications: a metamorphic approach. International Journal of Software Engineering and Knowledge Engineering, to appear.

[12]

W. K. Chan, M. Y. Cheng, S. C. Cheung, and T. H. Tse. Automatic goal-oriented classification of failure behaviors for testing XML-based multimedia software applications: an experimental case study. Journal of Systems and Software, 79(5):602--612, 2006.

Digital Library

[13]

W. K. Chan, S. C. Cheung, and K. R. P. H. Leung. Towards a metamorphic testing methodology for service-oriented software applications. The 1st International Conference on Services Engineering (SEIW 2005), in Proceedings of the 5th International Conference on Quality Software (QSIC 2005), pages 470--476. IEEE Computer Society Press, Los Alamitos, California, 2005.

Digital Library

[14]

W. K. Chan, S. C. Cheung, and K. R. P. H. Leung. A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research, to appear.

[15]

D. Chapman. A program testing assistant. Communications of the ACM, 25(9):625--634, 1982.

Digital Library

[16]

T. Y. Chen, J. Feng, and T. H. Tse. Metamorphic testing of programs on partial differential equations: a case study. In Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC 2002), pages 327--333. IEEE Computer Society Press, Los Alamitos, California, 2002.

Digital Library

[17]

T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2002), pages 191--195. ACM Press, New York, 2002.

Digital Library

[18]

T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 45(1):1--9, 2003.

Digital Library

[19]

R. L. Cobleigh, G. S. Avrunin, and L. A. Clarke. User guidance for creating precise and accessible property specifications. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT 2006/FSE-14). ACM Press, New York, 2006.

Digital Library

[20]

P. Francis, D. Leon, M. Minch, and A. Podgurski. Tree-based methods for classifying software failures. In Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE 2004), pages 451--462. IEEE Computer Society Press, Los Alamitos, California, 2004.

Digital Library

[21]

P. G. Frankl and S. N. Weiss. An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Transactions on Software Engineering, 19(8):774--787, 1993.

Digital Library

[22]

A. Gotlieb and B. Botella. Automated metamorphic testing. In Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC 2003), pages 34--40. IEEE Computer Society Press, Los Alamitos, California, 2003.

Digital Library

[23]

R. Helm, I. M. Holland, and D. Gangopadhyay. Contracts: specifying behavioral compositions in object-oriented systems. In Proceedings of the 5th Annual Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA '90), ACM SIGPLAN Notices, 25(10):169--180, 1990.

Digital Library

[24]

M. Last, M. Friedman, and A. Kandel. The data mining approach to automated software testing. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2003), pages 388--396. ACM Press, New York, 2003.

Digital Library

[25]

H. Lu, W. K. Chan, and T. H. Tse. Testing context-aware middleware-centric programs: a data flow approach and an RFID-based experimentation. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT 2006/FSE-14). ACM Press, New York, 2006.

Digital Library

[26]

Y.-S. Ma, A. J. Offutt, and Y.-R. Kwon. MuJava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97--133, 2005.

Digital Library

[27]

L. I. Manolache and D. G. Kourie. Software testing using model programs. Software: Practice and Experience, 31 (13): 1211--1236, 2001.

Digital Library

[28]

A. M. Memon, M. E. Pollack, and M. L. Soffa. Automated test oracles for GUIs. In Proceedings of the 8th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT2000/FSE-8), pages 30--39. ACM Press, New York, 2000.

Digital Library

[29]

A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology, 5 (2):99--118, 1996.

Digital Library

[30]

A. Podgurski, D. Leon, P. Francis, W. Masri, M. Minch, J. Sun, and B. Wang. Automated support for classifying software failure reports. In Proceedings of the 25th International Conference on Software Engineering (ICSE 2003), pages 465--475. IEEE Computer Society Press, Los Alamitos, California, 2003.

Digital Library

[31]

Y. Sun and E. L. Jones. Specification-driven automated testing of GUI-based Java programs. In Proceedings of the 42nd Annual Southeast Regional Conference (ACM-SE 42), pages 140--145. ACM Press, New York, 2004.

Digital Library

[32]

R. N. Taylor. Assertions in programming languages. ACM SIGPLAN Notices, 15(1):105--114, 1980.

Digital Library

[33]

T. H. Tse, S. S. Yau, W. K. Chan, H. Lu, and T. Y Chen. Testing context-sensitive middleware-based software applications. In Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC 2004), volume 1, pages 458--465. IEEE Computer Society Press, Los Alamitos, California, 2004.

Digital Library

[34]

M. Vanmali, M. Last, and A. Kandel. Using a neural network in the software testing process. International Journal of Intelligent Systems, 17 (1):45--62, 2002.

[35]

E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25 (4):465--470, 1982.

[36]

Q. Xie and A. Memon. Designing and comparing automated test oracles for GUI-based software applications. ACM Transactions on Software Engineering and Methodology, to appear.

Digital Library

Cited By

Fu LLei YYan MXu LXu ZZhang X(2023)MetaFLIET Software10.1049/sfw2.1210217:2(137-153)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1049/sfw2.12102
Sun CFu APoon PXie XLiu HChen T(2019)METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output DomainsIEEE Transactions on Software Engineering10.1109/TSE.2019.2934848(1-1)Online publication date: 2019
https://doi.org/10.1109/TSE.2019.2934848
Ding JLi XHu X(2019)Testing Scientific Software with Invariant Relations: A Case Study2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS.2019.00057(406-417)Online publication date: Jul-2019
https://doi.org/10.1109/QRS.2019.00057
Show More Cited By

Index Terms

An empirical comparison between direct and indirect test result checking approaches
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Impacts of Test Suite's Class Imbalance on Spectrum-Based Fault Localization Techniques
QSIC '13: Proceedings of the 2013 13th International Conference on Quality Software

Spectrum-based fault localization (SBFL) uses the execution results of test cases to debug. There are two types of SBFL techniques: one using conventional slices, and the other using metamorphic slices. This paper investigates the ratio between non-...
Backward-Slice-Based Statistical Fault Localization without Test Oracles
QSIC '13: Proceedings of the 2013 13th International Conference on Quality Software

A recent promising technique for fault localization, Backward-Slice-based Statistical Fault Localization (BSSFL), statistically analyzes the backward slices and results of a set of test cases to evaluate the suspiciousness of a statement being faulty. ...
An empirical study of inadequate and adequate test suite reduction approaches
ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Background. Regression testing is conducted after changes are made to a system in order to ensure that these changes did not alter its expected behavior. The problem with regression testing is that it can require too much time and/or too many resources. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SOQUA '06: Proceedings of the 3rd international workshop on Software quality assurance

November 2006

86 pages

ISBN:1595935843

DOI:10.1145/1188895

General Chair:
Johannes Mayer
Ulm University, Germany
,
Program Chairs:
Neelam Gupta,
Yves Ledru

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGSOFT06/FSE-14

Sponsor:

SIGSOFT

SIGSOFT06/FSE-14: SIGSOFT 2006 -14th International Symposium on the Foundations of Software Engineering

November 6, 2006

Oregon, Portland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
860
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fu LLei YYan MXu LXu ZZhang X(2023)MetaFLIET Software10.1049/sfw2.1210217:2(137-153)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1049/sfw2.12102
Sun CFu APoon PXie XLiu HChen T(2019)METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output DomainsIEEE Transactions on Software Engineering10.1109/TSE.2019.2934848(1-1)Online publication date: 2019
https://doi.org/10.1109/TSE.2019.2934848
Ding JLi XHu X(2019)Testing Scientific Software with Invariant Relations: A Case Study2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS.2019.00057(406-417)Online publication date: Jul-2019
https://doi.org/10.1109/QRS.2019.00057
Ding JHu X(2017)Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation programSoftware Quality Journal10.1007/s11219-016-9337-325:3(841-869)Online publication date: 1-Sep-2017
https://dl.acm.org/doi/10.1007/s11219-016-9337-3
Yan SYang XLi MLiu HLiu Z(2017)Research of Testing for Scientific Computing Software in the Area of Nuclear Power Based on Metamorphic TestingProceedings of The 20th Pacific Basin Nuclear Conference10.1007/978-981-10-2311-8_46(501-512)Online publication date: 11-Jan-2017
https://doi.org/10.1007/978-981-10-2311-8_46
Ding JZhang DHu X(2016)An application of metamorphic testing for testing scientific softwareProceedings of the 1st International Workshop on Metamorphic Testing10.1145/2896971.2896981(37-43)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2896971.2896981
Segura SFraser GSanchez ARuiz-Cortes A(2016)A Survey on Metamorphic TestingIEEE Transactions on Software Engineering10.1109/TSE.2016.253287542:9(805-824)Online publication date: 1-Sep-2016
https://doi.org/10.1109/TSE.2016.2532875
Ding JZhang D(2016)An Approach for Iteratively Generating Adequate Tests in Metamorphic Testing: A Case Study2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC)10.1109/COMPSAC.2016.48(263-268)Online publication date: Jun-2016
https://doi.org/10.1109/COMPSAC.2016.48
Kanewala UBieman JBen-Hur A(2016)Predicting metamorphic relations for testing scientific softwareSoftware Testing, Verification & Reliability10.1002/stvr.159426:3(245-269)Online publication date: 1-May-2016
https://dl.acm.org/doi/10.1002/stvr.1594
Liu HYusuf ISchmidt HChen TJalote PBriand LHoek A(2014)Metamorphic fault tolerance: an automated and systematic methodology for fault tolerance in the absence of test oracleCompanion Proceedings of the 36th International Conference on Software Engineering10.1145/2591062.2591109(420-423)Online publication date: 31-May-2014
https://dl.acm.org/doi/10.1145/2591062.2591109
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten