Automating experimentation on distributed testbeds

ABSTRACT

Engineering distributed systems is a challenging activity. This is partly due to the intrinsic complexity of distributed systems, and partly due to the practical obstacles that developers face when evaluating and tuning their design and implementation decisions.This paper addresses the latter aspect, providing techniques for software engineers to automate the experimentation activity. Our approach is founded on a suite of models that characterize the distributed system under experimentation, the testbeds upon which the experiments are to be carried out, and the client behaviors that drive the experiments. The models are used by generative techniques to automate construction of the workloads,as well as construction of the scripts for deploying and executing the experiments on distributed testbeds. The framework is not targeted at a specific system or application model, but rather is a generic, programmable tool. We have validated our approach by performing experiments on a variety of distributed systems. For two of these systems, the experiments were deployed and executed on the PlanetLab wide-area testbed.Our experience shows that this framework can be readily applied to different kinds of distributed system architectures,and that using it for meaningful experimentation,especially in large-scale network environments, is advantageous.

REFERENCES

• 1: Robert Adams, Take command: The m4 macro package, Linux Journal, v.2002 n.96, p.6, April 2002
• 2: M. Caporuscio, A. Carzaniga, and A. L. Wolf. Design and evaluation of a support service for mobile, wireless publish/subscribe applications. IEEE Transactions on Software Engineering, 29(12):1059--1071, Dec. 2003.
• 3: Design and evaluation of a wide-area event notification service, ACM Transactions on Computer Systems (TOCS), v.19 n.3, p.332-383, Aug. 2001 [doi>10.1145/380749.380767]
• 4: B. Chun. pssh HOWTO. Intel Research Berkeley, Nov. 2003.
• 5: I. Clarke. Freenet's Next Generation Routing Protocol. Freenet Project, July 2003.
• 6: Ian Clarke , Scott G. Miller , Theodore W. Hong , Oskar Sandberg , Brandon Wiley, Protecting Free Expression Online with Freenet, IEEE Internet Computing, v.6 n.1, p.40-49, January 2002 [doi>10.1109/4236.978368 ]
• 7: Sudipto Ghosh , Nishant Bawa , Gerald Craig , Ketaki Kalgaonkar, A Test Management and Software Visualization Framework for Heterogeneous Distributed Applications, The 6th IEEE International Symposium on High-Assurance Systems Engineering: Special Topic: Impact of Networking, p.106-116, October 24-26, 2001
• 8: David Guerrero, System Administration, Linux Journal, v.1999 n.58es, Feb. 1999
• 9: Alex Hubbard , C. Murray Woodside , Cheryl Schramm, DECALS: distributed experiment control and logging system, Proceedings of the 1995 conference of the Centre for Advanced Studies on Collaborative research, p.32, November 07-09, 1995, Toronto, Ontario, Canada
• 10: CCMPerf: A Benchmarking Tool for CORBA Component Model Implementations, Proceedings of the 10th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS'04), p.140, May 25-28, 2004
• 11: Ben Laurie , Peter Laurie, Apache: The Definitive Guide, O'Reilly & Associates, Inc., Sebastopol, CA, 2002
• 12: A. Martinez, Y. Dimitriadis, and P. de la Fuente. Towards an XML-based model for the representation of collaborative action. In Proceedings of the Conference on Computer Support for Collaborative Learning (CSCL '03), pages 14--18, Bergen, Norway, June 2003.
• 13: A. Memon , A. Porter , C. Yilmaz , A. Nagarajan , D. Schmidt , B. Natarajan, Skoll: Distributed Continuous Quality Assurance, Proceedings of the 26th International Conference on Software Engineering, p.459-468, May 23-28, 2004
• 14: Larry Peterson , Tom Anderson , David Culler , Timothy Roscoe, A blueprint for introducing disruptive technology into the Internet, ACM SIGCOMM Computer Communication Review, v.33 n.1, p.59-64, January 2003 [doi>10.1145/774763.774772]
• 15: B. Segall and D. Arnold. Elvin has left the building: A publish/subscribe notification service with quenching. In Proceedings of the Australian UNIX and Open Systems User Group Conference (AUUG '97), pages 243--255, Brisbane, Australia, Sept. 1997.
• 16: Ion Stoica , Robert Morris , David Karger , M. Frans Kaashoek , Hari Balakrishnan, Chord: A scalable peer-to-peer lookup service for internet applications, Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications, p.149-160, August 2001, San Diego, California, United States
• 17: Linda Tauscher , Saul Greenberg, How people revisit web pages: empirical findings and implications for the design of history systems, International Journal of Human-Computer Studies, v.47 n.1, p.97-137, July 1997 [doi>10.1006/ijhc.1997.0125 ]
• 18: Y. Wang, M. J. Rutherford, A. Carzaniga, and A. L. Wolf. Weevil: A tool to automate experimentation with distributed systems. Technical Report CU-CS-980-04, Department of Computer Science, University of Colorado, Oct. 2004.
• 19: Brian White , Jay Lepreau , Leigh Stoller , Robert Ricci , Shashi Guruprasad , Mac Newbold , Mike Hibler , Chad Barb , Abhijeet Joglekar, An integrated experimental environment for distributed systems and networks, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060313]
• 20: Alec Wolman , M. Voelker , Nitin Sharma , Neal Cardwell , Anna Karlin , Henry M. Levy, On the scale and performance of cooperative Web proxy caching, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.16-31, December 12-15, 1999, Charleston, South Carolina, United States

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques

General Terms:
Experimentation

Keywords:
PlanetLab, distributed systems, experiment automation, wide-area testbeds