skip to main content
10.1145/1101908.1101934acmconferencesArticle/Chapter ViewAbstractPublication PagesaseConference Proceedingsconference-collections
Article

Automating experimentation on distributed testbeds

Published: 07 November 2005 Publication History

Abstract

Engineering distributed systems is a challenging activity. This is partly due to the intrinsic complexity of distributed systems, and partly due to the practical obstacles that developers face when evaluating and tuning their design and implementation decisions.This paper addresses the latter aspect, providing techniques for software engineers to automate the experimentation activity. Our approach is founded on a suite of models that characterize the distributed system under experimentation, the testbeds upon which the experiments are to be carried out, and the client behaviors that drive the experiments. The models are used by generative techniques to automate construction of the workloads,as well as construction of the scripts for deploying and executing the experiments on distributed testbeds. The framework is not targeted at a specific system or application model, but rather is a generic, programmable tool. We have validated our approach by performing experiments on a variety of distributed systems. For two of these systems, the experiments were deployed and executed on the PlanetLab wide-area testbed.Our experience shows that this framework can be readily applied to different kinds of distributed system architectures,and that using it for meaningful experimentation,especially in large-scale network environments, is advantageous.

References

[1]
R. Adams. Take command: The m4 macro package. Linux J., 2002(96):6, Apr. 2002.
[2]
M. Caporuscio, A. Carzaniga, and A. L. Wolf. Design and evaluation of a support service for mobile, wireless publish/subscribe applications. IEEE Transactions on Software Engineering, 29(12):1059--1071, Dec. 2003.
[3]
A. Carzaniga, D. S. Rosenblum, and A. L. Wolf. Design and evaluation of a wide-area event notification service. ACM Transactions on Computer Systems, 19(3):332--383, Aug. 2001.
[4]
B. Chun. pssh HOWTO. Intel Research Berkeley, Nov. 2003.
[5]
I. Clarke. Freenet's Next Generation Routing Protocol. Freenet Project, July 2003.
[6]
I. Clarke, S. G. Miller, T. W. Hong, O. Sandberg, and B. Wiley. Protecting free expression online with Freenet. IEEE Internet Computing, 6(1):40--49, 2002.
[7]
S. Ghosh, N. Bawa, G. Craig, and K. Kalgaonkar. A test management and software visualization framework for heterogeneous distributed applications. In Proceedings of the 6th IEEE International Symposium on High Assurance Systems Engineering (HASE '01), pages 106--116, Boca Raton, Florida, Oct. 2001.
[8]
D. Guerrero. System administration: Caching the web. Linux J., 1999(58es):11, 1999.
[9]
A. Hubbard, C. M. Woodside, and C. Schramm. Decals: Distributed experiment control and logging system. In Proceedings of the 1995 Conference of the Centre for Advanced Studies on Collaborative Research, page 32, Toronto, Ontario, Canada, Nov. 1995.
[10]
A. S. Krishna, N. Wang, B. Natarajan, A. Gokhale, D. C. Schmidt, and G. Thaker. CCMPerf: A benchmarking tool for CORBA component model implementations. In Proceedings of the 10th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS'04), pages 140--147, Toronto, Canada, May 2004.
[11]
B. Laurie and P. Laurie. Apache: The Definitive Guide. O'Reilly and Associates, 3 edition, 2002.
[12]
A. Martinez, Y. Dimitriadis, and P. de la Fuente. Towards an XML-based model for the representation of collaborative action. In Proceedings of the Conference on Computer Support for Collaborative Learning (CSCL '03), pages 14--18, Bergen, Norway, June 2003.
[13]
A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. In Proceedings of the 26th IEEE/ACM International Conference on Software Engineering (ICSE '04), pages 459--468, Edinburgh, United Kingdom, May 2004.
[14]
L. Peterson, T. Anderson, D. Culler, and T. Roscoe. A blueprint for introducing disruptive technology into the Internet. ACM SIGCOMM Computer Communication Review, 33(1):59--64, 2003.
[15]
B. Segall and D. Arnold. Elvin has left the building: A publish/subscribe notification service with quenching. In Proceedings of the Australian UNIX and Open Systems User Group Conference (AUUG '97), pages 243--255, Brisbane, Australia, Sept. 1997.
[16]
I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for Internet applications. In Proceedings of the 2001 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM '01), pages 149--160, San Diego, California, Aug. 2001.
[17]
L. Tauscher and S. Greenberg. How people revisit web pages: Empirical findings and implications for the design of history systems. International Journal on Human-Computer Studies, 47(1):97--138, 1997.
[18]
Y. Wang, M. J. Rutherford, A. Carzaniga, and A. L. Wolf. Weevil: A tool to automate experimentation with distributed systems. Technical Report CU-CS-980-04, Department of Computer Science, University of Colorado, Oct. 2004.
[19]
B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad, M. Newbold, M. Hibler, C. Barb, and A. Joglekar. An integrated experimental environment for distributed systems and networks. In Proc. of the Fifth Symposium on Operating Systems Design and Implementation, pages 255--270, Boston, MA, Dec. 2002. USENIX Association.
[20]
A. Wolman, G. M. Voelker, N. Sharma, N. Cardwell, A. Karlin, and H. M. Levy. On the scale and performance of cooperative web proxy caching. In 17th ACM Symposium on Operating Systems Principles (SOSP '99), pages 16--31, Kiawah Island, SC, Dec. 1999.

Cited By

View all
  • (2024)Distributed Automated Testing Framework for Bluetooth Mesh ApplicationsNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575328(1-5)Online publication date: 6-May-2024
  • (2019)ExpRunA : a domain-specific approach for technology-oriented experimentsSoftware and Systems Modeling10.1007/s10270-019-00749-619:2(493-526)Online publication date: 5-Aug-2019
  • (2015)A survey of general-purpose experiment management tools for distributed systemsFuture Generation Computer Systems10.1016/j.future.2014.10.00745:C(1-12)Online publication date: 1-Apr-2015
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ASE '05: Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering
November 2005
482 pages
ISBN:1581139934
DOI:10.1145/1101908
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2005

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. PlanetLab
  2. distributed systems
  3. experiment automation
  4. wide-area testbeds

Qualifiers

  • Article

Conference

ASE05

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)1
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Distributed Automated Testing Framework for Bluetooth Mesh ApplicationsNOMS 2024-2024 IEEE Network Operations and Management Symposium10.1109/NOMS59830.2024.10575328(1-5)Online publication date: 6-May-2024
  • (2019)ExpRunA : a domain-specific approach for technology-oriented experimentsSoftware and Systems Modeling10.1007/s10270-019-00749-619:2(493-526)Online publication date: 5-Aug-2019
  • (2015)A survey of general-purpose experiment management tools for distributed systemsFuture Generation Computer Systems10.1016/j.future.2014.10.00745:C(1-12)Online publication date: 1-Apr-2015
  • (2014)A workflow-inspired, modular and robust approach to experiments in distributed systemsProceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2014.33(112-121)Online publication date: 26-May-2014
  • (2013)Position paperProceedings of the 2013 international workshop on Hot topics in cloud services10.1145/2462307.2462322(63-70)Online publication date: 20-Apr-2013
  • (2013)An automated approach to create, store, and analyze large-scale experimental data in clouds2013 IEEE 14th International Conference on Information Reuse & Integration (IRI)10.1109/IRI.2013.6642493(357-364)Online publication date: Aug-2013
  • (2013)An infrastructure for automating large-scale performance studies and data processing2013 IEEE International Conference on Big Data10.1109/BigData.2013.6691573(187-192)Online publication date: Oct-2013
  • (2013)Automated testing of cloud-based elastic systems with AUToCLESProceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2013.6693140(714-717)Online publication date: 11-Nov-2013
  • (2012)AbstractProceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis10.1109/SC.Companion.2012.240(1435-1436)Online publication date: 10-Nov-2012
  • (2012)ExpertusProceedings of the 2012 IEEE Fifth International Conference on Cloud Computing10.1109/CLOUD.2012.98(115-122)Online publication date: 24-Jun-2012
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media