skip to main content
10.1145/1996109.1996119acmconferencesArticle/Chapter ViewAbstractPublication PageshpdcConference Proceedingsconference-collections
research-article

Magellan: experiences from a science cloud

Published: 08 June 2011 Publication History

Abstract

Cloud resources promise to be an avenue to address new categories of scientific applications including data-intensive science applications, on-demand/surge computing, and applications that require customized software environments. However, there is a limited understanding on how to operate and use clouds for scientific applications. Magellan, a project funded through the Department of Energy's (DOE) Advanced Scientific Computing Research (ASCR) program, is investigating the use of cloud computing for science at the Argonne Leadership Computing Facility (ALCF) and the National Energy Research Scientific Computing Facility (NERSC). In this paper, we detail the experiences to date at both sites and identify the gaps and open challenges from both a resource provider as well as application perspective.

References

[1]
The FutureGrid Project. https://portal.futuregrid.org/.
[2]
G. Aldering et al. Overview of the Nearby Supernova Factory. In J. A. Tyson & S. Wol, editor, Society of Photo-Optical Instrumentation Engineers (SPIE) Series, volume 4836 of Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference, pages 61--72, Dec. 2002.
[3]
bcfg2. http://trac.mcs.anl.gov/projects/bcfg2.
[4]
Cern Virtual Machines. http://rbuilder.cern.ch/project/cernvm/releases.
[5]
J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI '04, pages 137--150, 2004.
[6]
E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good. The Cost of Doing Science on the Cloud: The Montage Example. In Proceedings of SC'08, Austin, TX, 2008. IEEE.
[7]
E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good. The cost of doing science on the cloud: the montage example. In Proceedings of the 2008 ACM/IEEE conference on Supercomputing, pages 1--12. IEEE Press, 2008.
[8]
C. Evangelinos and C. Hill. Cloud Computing for parallel Scientific HPC Applications: Feasibility of running Coupled Atmosphere-Ocean Climate Models on AmazonaAZs EC2. ratio, 2(2.40):2--34, 2008.
[9]
M. Fenn, S. Goasguen, and J. Lauret. Contextualization in practice: The clemson experience. In ACAT Proceedings, 2010.
[10]
S. Hazelhurst. Scientific computing using virtual high-performance computing: a case study using the Amazon elastic computing cloud. In Proceedings of the 2008 annual research conference of the South African Institute of Computer Scientists and Information Technologists on IT research in developing countries: riding the wave of technology, pages 94--103. ACM, 2008.
[11]
Heckle. http://trac.mcs.anl.gov/projects/Heckle.
[12]
K. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. Wasserman, and N. Wright. Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud. In 2nd IEEE International Conference on Cloud Computing Technology and Science, 2010.
[13]
G. Juve and E. Deelman. Scientific work flows and clouds. Crossroads, 16:14--18, March 2010.
[14]
K. Keahey. Cloud Computing for Science. In Proceedings of the 21st International Conference on Scientific and Statistical Database Management, page 478. Springer-Verlag, 2009.
[15]
K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, and M. Tsugawa. Science clouds: Early experiences in cloud computing for scientific applications. Cloud Computing and Applications, 2008, 2008.
[16]
K. Keahey, T. Freeman, J. Lauret, and D. Olson. Virtual workspaces for scientific applications. In Journal of Physics: Conference Series, volume 78, page 012038. Institute of Physics Publishing, 2007.
[17]
J. Lauret, M. Walker, S. Goasguen, and L. Hajdu. From Grid to cloud, the STAR experience. SciDAC 2010 Proceedings, 2010.
[18]
J. Li, D. Agarwal, M. Humphrey, C. van Ingen, K. Jackson, and Y. Ryu. eScience in the Cloud: A MODIS Satellite Data Reprojection and Reduction Pipeline in the Windows Azure Platform. In IPDPS, 2010.
[19]
J. Napper and P. Bientinesi. Can cloud computing reach the top500? In Proceedings of the combined workshops on UnConventional high performance computing workshop plus memory access workshop, pages 17--20. ACM, 2009.
[20]
Nist cloud. http://csrc.nist.gov/groups/SNS/cloud-computing/.
[21]
F. T. C. on a Cloud Computing Cluster: Using Parallel Virtualization for Large-Scale Climate Simulation Analysis. Daren Hasenkamp and Alex Sim and Michael Wehner and Kesheng Wu. In 2nd IEEE International Conference on Cloud Computing Technology and Science, 2010.
[22]
S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema. An early performance analysis of cloud computing services for scientific computing. Delft University of Technology, Tech. Rep, 2008.
[23]
M. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel. Amazon S3 for science grids: a viable solution? In Proceedings of the 2008 international workshop on Data-aware distributed computing, pages 55--64. ACM, 2008.
[24]
L. Ramakrishnan et al. VGrADS: Enabling e-Science Work flows on Grids and Clouds with Fault Tolerance. In Proceedings of the ACM/IEEE SC2009 Conference on High Performance Computing, Networking, Storage and Analysis, Portland, Oregon, Portland, Oregon, November 2009.
[25]
J. Rehr, F. Vila, J. Gardner, L. Svec, and M. Prange. Scientific computing in the cloud. Computing in Science and Engineering, 99(PrePrints), 2010.
[26]
M. C. Schatz. CloudBurst: highly sensitive read mapping with MapReduce. Bioinformatics, pages 1363--1369, June 2009.
[27]
G. Wang and T. E. Ng. The impact of virtualization on network performance of amazon ec2 data center. In Proceedings of IEEE INFOCOM, 2010.
[28]
From Clusters To Clouds: xCAT 2 Is Out Of The Bag. http://www.linux-mag.com/id/7230/.

Cited By

View all
  • (2024)Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows2024 IEEE 20th International Conference on e-Science (e-Science)10.1109/e-Science62913.2024.10678669(1-9)Online publication date: 16-Sep-2024
  • (2024)Jetstream2: Research Clouds as a Convergence AcceleratorComputing in Science and Engineering10.1109/MCSE.2024.340238926:3(9-19)Online publication date: 21-May-2024
  • (2024)‘The Cloud is Not Not IT’: Ecological Change in Research Computing in the CloudComputer Supported Cooperative Work (CSCW)10.1007/s10606-024-09490-1Online publication date: 14-Mar-2024
  • Show More Cited By

Index Terms

  1. Magellan: experiences from a science cloud

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ScienceCloud '11: Proceedings of the 2nd international workshop on Scientific cloud computing
    June 2011
    74 pages
    ISBN:9781450306997
    DOI:10.1145/1996109
    • General Chairs:
    • Ioan Raicu,
    • Pete Beckman,
    • Ian T. Foster,
    • Program Chair:
    • Yogesh Simmhan
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 June 2011

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. cloud computing
    2. mapreduce
    3. programming model
    4. science
    5. virtual machines

    Qualifiers

    • Research-article

    Conference

    HPDC '11
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 44 of 151 submissions, 29%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)7
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows2024 IEEE 20th International Conference on e-Science (e-Science)10.1109/e-Science62913.2024.10678669(1-9)Online publication date: 16-Sep-2024
    • (2024)Jetstream2: Research Clouds as a Convergence AcceleratorComputing in Science and Engineering10.1109/MCSE.2024.340238926:3(9-19)Online publication date: 21-May-2024
    • (2024)‘The Cloud is Not Not IT’: Ecological Change in Research Computing in the CloudComputer Supported Cooperative Work (CSCW)10.1007/s10606-024-09490-1Online publication date: 14-Mar-2024
    • (2021)A BitTorrent Mechanism-Based Solution for Massive System DeploymentIEEE Access10.1109/ACCESS.2021.30525259(21043-21058)Online publication date: 2021
    • (2019)Enabling HPC Workloads on Cloud Infrastructure Using Kubernetes Container Orchestration Mechanisms2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)10.1109/CANOPIE-HPC49598.2019.00007(11-20)Online publication date: Nov-2019
    • (2019)An efficient cloud scheduler design supporting preemptible instancesFuture Generation Computer Systems10.1016/j.future.2018.12.05795:C(68-78)Online publication date: 1-Jun-2019
    • (2018)Dynamically negotiating capacity between on-demand and batch clustersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291707(1-11)Online publication date: 11-Nov-2018
    • (2018)Dynamically negotiating capacity between on-demand and batch clustersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC.2018.00041(1-11)Online publication date: 11-Nov-2018
    • (2018)Orchestrating Complex Application Architectures in Heterogeneous CloudsJournal of Grid Computing10.1007/s10723-017-9418-y16:1(3-18)Online publication date: 1-Mar-2018
    • (2017)Understanding the Performance and Potential of Cloud Computing for Scientific ApplicationsIEEE Transactions on Cloud Computing10.1109/TCC.2015.24048215:2(358-371)Online publication date: 1-Apr-2017
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media