Article

Simulation-based test adequacy criteria for distributed systems

Authors:

Matthew J. Rutherford,

Antonio Carzaniga,

Alexander L. WolfAuthors Info & Claims

SIGSOFT '06/FSE-14: Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering

Pages 231 - 241

https://doi.org/10.1145/1181775.1181804

Published: 05 November 2006 Publication History

Abstract

Developers of distributed systems routinely construct discrete-event simulations to help understand and evaluate the behavior of inter-component protocols. Simulations are abstract models of systems and their environments, capturing basic algorithmic functionality at the same time as they focus attention on properties critical to distribution, including topology, timing, bandwidth, and overall scalability. We claim that simulations can be treated as a form of specification, and thereby used within a specification-based testing regime to provide developers with a rich new basis for defining and applying system-level test adequacy criteria. We describe a framework for evaluating distributed system test adequacy criteria, and demonstrate our approach on simulations and implementations of three distributed systems, including DNS, the Domain Name System.

References

[1]

M. Allman and A. Falk. On the effective evaluation of TCP. SIGCOMM Computer Communications Review, 29(5):59--70, 1999.

Digital Library

[2]

P. E. Ammann and P. E. Black. A specification-based coverage metric to evaluate test sets. International Journal of Reliability, Quality and Safety Engineering, 8(4):275--299, 2001.

[3]

G. Bochmann and A. Petrenko. Protocol testing: Review of methods and relevance for software testing. In Proceedings of the 1994 ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 109--124, 1994.

Digital Library

[4]

L. C. Briand, Y. Labiche, and Y. Wang. Using simulation to empirically investigate test coverage criteria based on statecharts. In Proceedings of the 26th International Conference on Software Engineering, pages 86--95, 2004.

Digital Library

[5]

R. H. Carver and K.-C. Tai. Use of sequencing constraints for specification-based testing of concurrent programs. IEEE Transactions on Software Engineering, 24(6):471--490, 1998.

Digital Library

[6]

J. Chang and D. J. Richardson. Structural specification-based testing: Automated support and experimental evaluation. In Proceedings of the 7th European Software Engineering Conference 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 285--302, 1999.

Digital Library

[7]

I. Clarke, S. G. Miller, T. W. Hong, O. Sandberg, and B. Wiley. Protecting free expression online with Freenet. IEEE Internet Computing, 6(1):40--49, 2002.

Digital Library

[8]

R. A. DeMillo, R. Lipton, and F. Sayward. Hints on test data selection: Help for the practicing programmer. IEEE Computer, 11(4):34--41, Apr. 1978.

Digital Library

[9]

J. L. Devore. Probability and Statistics for Engineering and the Sciences. Brooks/Cole, 4th edition, 1995.

[10]

S. C. P. F. Fabbri, J. C. Maldonado, T. Sugeta, and P. C. Masiero. Mutation testing applied to validate specifications based on statecharts. In Proceedings of the 10th International Symposium on Software Reliability Engineering, page 210, Washington, DC, USA, 1999.

Digital Library

[11]

P. Frankl and O. Iakounenko. Further empirical studies of test effectiveness. In Proceedings of the 6th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 153--162, New York, NY, USA, 1998.

Digital Library

[12]

P. Frankl and S. Weiss. An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Transactions on Software Engineering, 19(8):774--787, Aug. 1993.

Digital Library

[13]

P. Frankl and E. Weyuker. An applicable family of data flow testing criteria. IEEE Transactions on Software Engineering, 14(10):1483--1498, 1988.

Digital Library

[14]

M. Harder, J. Mellen, and M. D. Ernst. Improving test suites via operational abstraction. In Proceedings of the 25th International Conference on Software Engineering, pages 60--71, 2003.

Digital Library

[15]

Z. Jin and J. Offutt. Deriving tests from software architectures. In Proceedings of the 12th IEEE International Symposium on Software Reliability Engineering, pages 308--313, Nov. 2001.

Digital Library

[16]

D. R. Kuhn. Fault classes and error detection capability of specification-based testing. ACM Transactions on Software Engineering Methodology, 8(4):411--424, 1999.

Digital Library

[17]

J. F. Kurose and K. W. Ross. Computer Networking: A Top-Down Approach Featuring the Internet. Pearson Benjamin Cummings, 2004.

Digital Library

[18]

Y.-S. Ma, J. Offutt, and Y.-R. Kwon. Mujava: An automated class mutation system. Journal of Software Testing, Verification and Reliability, 15(2):97--133, 2005.

Digital Library

[19]

H. Muccini, A. Bertolino, and P. Inverardi. Using software architecture for code testing. IEEE Transactions on Software Engineering, 30(3):160--171, Mar. 2004.

Digital Library

[20]

A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering Methodology, 5(2):99--118, 1996.

Digital Library

[21]

A. J. Offutt and S. Liu. Generating test data from SOFL specifications. Journal of Systems and Software, 49(1):49--62, 1999.

Digital Library

[22]

J. Offutt, S. Liu, A. Abdurazik, and P. Ammann. Generating test data from state-based specifications. Journal of Software Testing, Verification and Reliability, 13(1):25--53, Mar. 2003.

[23]

T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating fuctional tests. Communications of the ACM, 31(6):676--686, 1988.

Digital Library

[24]

D. E. Perry and A. L. Wolf. Foundations for the study of software architecture. SIGSOFT Software Engineering Notes, 17(4):40--52, 1992.

Digital Library

[25]

M. D. Rice and S. B. Seidman. An approach to architectural analysis and testing. In Proceedings of the 3rd International Workshop on Software Architecture, pages 121--123, 1998.

Digital Library

[26]

D. Richardson, O. O'Malley, and C. Tittle. Approaches to specification-based testing. In Proceedings of the ACM SIGSOFT '89 Third Symposium on Software Testing, Analysis, and Verification, pages 86--96, 1989.

Digital Library

[27]

D. Richardson and M. Thompson. An analysis of test data selection criteria using the RELAY model of fault detection. IEEE Transactions on Software Engineering, 19(6):533--553, 1993.

Digital Library

[28]

D. J. Richardson and A. L. Wolf. Software testing at the architectural level. In Joint Proceedings of the Second International Software Architecture Workshop (ISAW-2) and International Workshop on Multiple Perspectives in Software Development (Viewpoints '96) on SIGSOFT '96 Workshops, pages 68--71, 1996.

Digital Library

[29]

M. J. Rutherford, A. Carzaniga, and A. L. Wolf. Simulation-based testing of distributed systems. Technical Report CU-CS-1004-06, University of Colorado at Boulder, Boulder, CO, Jan. 2006.

[30]

S. D. R. S. D. Souza, J. C. Maldonado, S. C. P. F. Fabbri, and W. L. D. Souza. Mutation testing applied to Estelle specifications. Software Quality Control, 8(4):285--301, 1999.

Digital Library

[31]

I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Transactions on Networking, 11(1):17--32, 2003.

Digital Library

[32]

T. Tsuchiya and T. Kikuno. On fault classes and error detection capability of specification-based testing. ACM Transactions on Software Engineering Methodology, 11(1):58--62, 2002.

Digital Library

[33]

Y. Wang, M. J. Rutherford, A. Carzaniga, and A. L. Wolf. Automating experimentation on distributed testbeds. In Proceedings of 20th IEEE/ACM International Conference on Automated Software Engineering, pages 164--173, Long Beach, CA, Nov. 2005.

Digital Library

[34]

E. Weyuker, T. Goradia, and A. Singh. Automatically generating test data from a boolean specification. IEEE Transactions on Software Engineering, 20(5):353--363, 1994.

Digital Library

Cited By

Crochet CAoga JLegay A(2024)Formally Discovering and Reproducing Network Protocols VulnerabilitiesSecure IT Systems10.1007/978-3-031-79007-2_22(424-443)Online publication date: 6-Nov-2024
https://dl.acm.org/doi/10.1007/978-3-031-79007-2_22
Zarrad AAlsmadi IYassine A(2020)Mutation Testing Framework for Ad-hoc Networks Protocols2020 IEEE Wireless Communications and Networking Conference (WCNC)10.1109/WCNC45663.2020.9120695(1-8)Online publication date: May-2020
https://doi.org/10.1109/WCNC45663.2020.9120695
Lucas CElbaum SRosenblum D(2012)Detecting problematic message sequences and frequencies in distributed systemsACM SIGPLAN Notices10.1145/2398857.238468347:10(915-926)Online publication date: 19-Oct-2012
https://dl.acm.org/doi/10.1145/2398857.2384683
Show More Cited By

Index Terms

Simulation-based test adequacy criteria for distributed systems
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems

Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, ...
A Formal Analysis of the Subsume Relation Between Software Test Adequacy Criteria

Software test adequacy criteria are rules to determine whether a software system has been adequately tested. A central question in the study of test adequacy criteria is how they relate to fault detecting ability. In this paper, we identify two ...
DOM-based test adequacy criteria for web applications
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

To assess the quality of web application test cases, web developers currently measure code coverage. Although code coverage has traditionally been a popular test adequacy criterion, we believe it alone is not adequate for assessing the quality of web ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGSOFT '06/FSE-14: Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering

November 2006

298 pages

ISBN:1595934685

DOI:10.1145/1181775

General Chair:
Michal Young
University of Oregon
,
Program Chair:
Premkumar Devanbu
University of California, Davis

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGSOFT06/FSE-14

Sponsor:

SIGSOFT06/FSE-14: SIGSOFT 2006 -14th International Symposium on the Foundations of Software Engineering

November 5 - 11, 2006

Oregon, Portland, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
559
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Crochet CAoga JLegay A(2024)Formally Discovering and Reproducing Network Protocols VulnerabilitiesSecure IT Systems10.1007/978-3-031-79007-2_22(424-443)Online publication date: 6-Nov-2024
https://dl.acm.org/doi/10.1007/978-3-031-79007-2_22
Zarrad AAlsmadi IYassine A(2020)Mutation Testing Framework for Ad-hoc Networks Protocols2020 IEEE Wireless Communications and Networking Conference (WCNC)10.1109/WCNC45663.2020.9120695(1-8)Online publication date: May-2020
https://doi.org/10.1109/WCNC45663.2020.9120695
Lucas CElbaum SRosenblum D(2012)Detecting problematic message sequences and frequencies in distributed systemsACM SIGPLAN Notices10.1145/2398857.238468347:10(915-926)Online publication date: 19-Oct-2012
https://dl.acm.org/doi/10.1145/2398857.2384683
Lucas CElbaum SRosenblum DLeavens GDwyer M(2012)Detecting problematic message sequences and frequencies in distributed systemsProceedings of the ACM international conference on Object oriented programming systems languages and applications10.1145/2384616.2384683(915-926)Online publication date: 19-Oct-2012
https://dl.acm.org/doi/10.1145/2384616.2384683
PLATON ESEI Y(2008)Security software engineering in wireless sensor networksProgress in Informatics10.2201/NiiPi.2008.5.6(49)Online publication date: Mar-2008
https://doi.org/10.2201/NiiPi.2008.5.6
Lai ZCheung SChan WHarrold MMurphy G(2008)Inter-context control-flow and data-flow test adequacy criteria for nesC applicationsProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering10.1145/1453101.1453115(94-104)Online publication date: 9-Nov-2008
https://dl.acm.org/doi/10.1145/1453101.1453115
Calegari RMusolesi MRaimondi FMascolo CCrnkovic IBertolino A(2007)CTGProceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering10.1145/1287624.1287684(415-424)Online publication date: 7-Sep-2007
https://dl.acm.org/doi/10.1145/1287624.1287684
Herzner WSchlick RLe Guennec AMartin B(2007)Model-Based Simulation of Distributed Real-time Applications2007 5th IEEE International Conference on Industrial Informatics10.1109/INDIN.2007.4384909(989-994)Online publication date: Jul-2007
https://doi.org/10.1109/INDIN.2007.4384909
Bertolino A(2007)Software Testing Research2007 Future of Software Engineering10.1109/FOSE.2007.25(85-103)Online publication date: 23-May-2007
https://dl.acm.org/doi/10.1109/FOSE.2007.25

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten