Jetstream: performance, early experiences, and early results

ABSTRACT

Jetstream is a first-of-a-kind system for the NSF - a distributed production cloud resource. The NSF awarded funds to create Jetstream in November 2014. Here we review the purpose for creating Jetstream, present the acceptance test results that define Jetstream's key characteristics, describe our experiences in standing up an OpenStack-based cloud environment, and share some of the early scientific results that have been obtained by researchers and students using this system. Jetstream offers unique capability within the XSEDE-supported US national cyberinfrastructure, delivering interactive virtual machines (VMs) via the Atmosphere interface developed by the University of Arizona. As a multi-region deployment that operates as a single integrated system, Jetstream is proving effective in supporting modes and disciplines of research traditionally underrepresented on larger XSEDE-supported clusters and supercomputers. Already, researchers in biology, network science, economics, earth science, and computer science have used Jetstream to perform research -- much of it research in the "long tail of science."

References

1. Stewart, C.A., Simms, S., Plale, B., Link, M., Hancock, D., and Fox, G. What is Cyberinfrastructure? in SIGUCCS 2010. 2010. Norfolk, VA. http://dx.doi.org/10.1145/1878335.1878347
2. National Science Foundation. NSF Solicitation 14-536. 2014; Available from: http://www.nsf.gov/pubs/2014/nsf14536/nsf14536.htm.
3. Heidorn, P.B., Shedding Light on the Dark Data in the Long Tail of Science. Library Trends, 2008. 57(2): p. 280--299. http://muse.jhu.edu/journals/library_trends/v057/57.2.heidorn.html
4. Stewart, C.A., Cockerill, T.M., Foster, I., Hancock, D., Merchant, N., Skidmore, E., Stanzione, D., Taylor, J., Tuecke, S., Turner, G., Vaughn, M., and Gaffney, N.I., Jetstream: a self-provisioned, scalable science and engineering cloud environment. 2015, In Proceedings of the 2015 XSEDE Conference. St. Louis, Missouri. ACM: 2792774. p. 1--8. http://dx.doi.org/10.1145/2792745.2792774
5. Stodden, V. and Miguez, S., Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. Journal of Open Research Software, 2014. 2(1): p. e21. http://doi.org/10.5334/jors.ay
6. Hurlin, C. and Perignon, C., RunMyCode.org: A Research-Reproducibility Tool for Computational Sciences, in Implementing Reproducible Computational Research, V. Stodden, F. Leisch, and R. and Peng, Editors. 2014, Chapman & Hall/CRC: Boca Raton.
7. Lawrence, K.A., Zentner, M., Wilkins-Diehr, N., Wernert, J.A., Pierce, M., Marru, S., and Michael, S., Science gateways today and tomorrow: positive perspectives of nearly 5000 members of the research community. Concurrency and Computation: Practice and Experience, 2015. 27(16): p. 4252--4268. http://dx.doi.org/10.1002/cpe.3526
8. Venkatesh, V., Morris, M.G., Davis, F.D., Davis, G.B, User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly, 2003. 27(3): p. 425--478. http://csdl-techreports.googlecode.com/svn/trunk/techreports/2005/05-06/doc/Venkatesh2003.pdf
9. iPlant Collaborative. Home page. Available from: http://iplantcollaborative.org/.
10. OpenStack. Home page. Available from: http://www.openstack.org/.
11. Hancock, D.Y., Link, M.R., and Stewart, C.A., Acceptance Test for Jetstream Test Cluster -- Jetstream-Arizona (JA) Dell PowerEdge Test and Development Cluster. PTI Technical Report PTI-TR15-007, 2015. http://hdl.handle.net/2022/20355
12. Stewart, C.A., Foster, I., Merchant, N.C., Taylor, J., and Vaughn, M.W., Jetstream -- A Self-Provisioned, Scalable Science and Engineering Cloud Environment - NSF Acceptance Report. PTI Technical Report, 2016. http://hdl.handle.net/2022/20889
13. Fox, G., Fortes, J., Grimshaw, A.S., Keahey, K., and Smith, W., FutureGrid: An Experimental, High-Performance Grid Test-bed. 2009, National Science Foundation: Indiana University.
14. Chamelon: A configurable experimental environment for large-scale cloud research. Available from: https://www.chameleoncloud.org/.
15. Cloudlab. Available from: http://www.cloudlab.us/.
16. Center for Computational Research - University at Buffalo. Open XDMoD: Metrics on Demand. Available from: http://xdmod.sourceforge.net/.
17. Voss, B.D., Jung-Gribble, Diane., Stewart, Craig A. The Leveraged Support Model. 1996. http://scholarworks.iu.edu/dspace/handle/2022/14011
18. Sieslack, N. Jack Dongarra on TOP500: Past, Present, and Future. 2015 {cited 2016 June 15}; Available from: http://www.top500.org/news/jack-dongarra-on-top500-past-present-and-future/.
19. DelValle, R., Rattihalli, G., Beltre, A., Madhusudhan, G., and Lewis, M., Exploring the Design Space for Optimizations with Apache Aurora and Mesos. accepted for publication in the IEEE International Conference on Cloud Computing (CLOUD), Applications Track, 2016.
20. Shen, N., Fan, Y., and Pamidighantam, S., E-science infrastructures for molecular modeling and parametrization. Journal of Computational Science, 2014. 5(4): p. 576--589. http://www.sciencedirect.com/science/article/pii/S1877750314000064
21. Stewart, C., Stanzione, D., Cockerill, T., Skidmore, E., Fischer, J., Lowe, J.M., Hammond, B., Turner, G., Hancock, D., and Miller, T., NSF High Performance Computing System Acquisition system description: Jetstream - a self-provisioned, scalable science and engineering cloud environment. 2016. http://doi.org/10.5967/K85Q4T1N