Beyond Beowulf Clusters: As clusters grow in size and complexity, it becomes harder and harder to manage their configurations.

Abstract

In the early ’90s, the Berkeley NOW Project under David Culler posited that groups of less capable machines could be used to solve scientific and other computing problems at a fraction of the cost of larger computers. In 1994, Donald Becker and Thomas Sterling worked to drive the costs even lower by adopting the then-fledgling Linux operating system to build Beowulf clusters at NASA’s Goddard Space Flight Center. By tying desktop machines together with open source tools such as PVM, MPI, and PBS, early clusters—which were often PC towers stacked on metal shelves with a nest of wires interconnecting them—fundamentally altered the balance of scientific computing. Before these first clusters appeared, distributed/parallel computing was prevalent at only a few computing centers, national laboratories, and a very few university departments. Since the introduction of clusters, distributed computing is now, literally, everywhere.

References

1. Venter, J.C., et. al. 2004. Environmental genome shotgun sequencing of the Sargasso Sea. Science 304(5667): 66-74.
2. Altschul, S.F., Madden, T.L., Schaffer, A.A., Zhang, J., Zhang, Z., Miller, W., Lipman, D.J. 1997. Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. Nucleic Acids Research 25(17): 3389-3402.
3. Grethe, J.S., Jovicich, J., Martone, M.E., Pieper, S., Brown, G.G., Ellisman, M.H., and Gollub, R.L. 2003. The Biomedical Informatics Research Network: Educational and Teaching Resources. Society for Neuroscience Annual Meeting, New Orleans.
4. Durbin, R., Eddy, S., Krogh, A., Mitchison, G. 1998. The theory behind profile HMMs. Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press.
5. Sonnhammer, E.L.L., Eddy, S.R., Durbin, R. 1997. Pfam: A comprehensive database of protein domain families based on seed alignments. Proteins 28(3): 405-420.
6. Papadopoulos, P.M., Katz, M.J., Bruno, G. 2001. NPACI Rocks: Tools and techniques for easily deploying manageable Linux clusters. IEEE Cluster 2001, Newport Beach, CA (October).
7. Katz, M.J., Papadopoulos, P.M., Bruno, G. 2002. Leveraging standard core technologies to programmatically build Linux cluster appliances. Clusters 2002: IEEE International Conference on Cluster Computing, Chicago (April).

Index Terms

• Computer systems organization

  • Architectures

    • Distributed architectures

• Software and its engineering

  • Software organization and properties

    • Contextual software domains

      • Operating systems

        • File systems management

        • Process management

    • Software system structures

      • Distributed systems organizing principles