Glenn K. Lockwood
Mahidhar Tatineni
ABSTRACT
The demand for virtualization within high-performance computing is rapidly growing as new communities, driven by both new application stacks and new computing modalities, continue to grow and expand. While virtualization has traditionally come with significant penalties in I/O performance that have precluded its use in mainstream large-scale computing environments, new standards such as Single Root I/O Virtualization (SR-IOV) are emerging that promise to diminish the performance gap and make high-performance virtualization possible.
To this end, we have evaluated SR-IOV in the context of both virtualized InfiniBand and virtualized 10 gigabit Ethernet (GbE) using micro-benchmarks and real-world applications. We compare the performance of these interconnects on non-virtualized environments, Amazon's SR-IOV-enabled C3 instances, and our own SR-IOV-enabled InfiniBand cluster and show that SR-IOV significantly reduces the performance losses caused by virtualization. InfiniBand demonstrates less than 2% loss of bandwidth and less than 10% increase in latency when virtualized with SR-IOV. Ethernet also benefits, although less dramatically, when SR-IOV is enabled on Amazon's cloud.
- Bollen, J., Fox, G., and Singhal, P.R. How and where the TeraGrid supercomputing infrastructure benefits science. Journal of Informetrics 5, 1 (2011), 114--121.Google Scholar
Cross Ref
- Dong, Y., Yang, X., Li, J., Liao, G., Tian, K., and Guan, H. High performance network virtualization with SR-IOV. Journal of Parallel and Distributed Computing 72, 11 (2012), 1471--1480. Google ScholarDigital Library
- Eldred, C. (Pittsburgh S.C. and Michalakes, J. (National C. for A.R. WRF V3 Parallel Benchmark Page. 2008. http://www.mmm.ucar.edu/wrf/WG2/benchv3/.Google Scholar
- Ghoshal, D., Canon, R.S., and Ramakrishnan, L. I/O performance of virtualized cloud environments. Proceedings of the Second International Workshop on Data Intensive Computing in the Clouds, ACM Press (2011), 71--80. Google ScholarDigital Library
- Giannozzi, P., Baroni, S., Bonini, N., et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. Journal of Physics: Condensed Matter 21, 39 (2009), 395502.Google ScholarCross Ref
- Hines, M.L. and Carnevale, N.T. Expanding NEURON's Repertoire of Mechanisms with NMODL. Neural Computation 12, 5 (2000), 995--1007. Google ScholarDigital Library
- Huang, Z., Ma, R., Li, J., Chang, Z., and Guan, H. Adaptive and Scalable Optimizations for High Performance SR-IOV. 2012 IEEE International Conference on Cluster Computing, IEEE (2012), 459--467. Google ScholarDigital Library
- Jackson, K.R., Ramakrishnan, L., Muriki, K., et al. Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud. 2010 IEEE Second International Conference on Cloud Computing Technology and Science, IEEE (2010), 159--168. Google ScholarDigital Library
- Jose, J., Li, M., Lu, X., Kandalla, K.C., Arnold, M.D., and Panda, D.K. SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience. 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, IEEE (2013), 385--392.Google Scholar
- Liu, J. Evaluating standard-based self-virtualizing devices: A performance study on 10 GbE NICs with SR-IOV support. 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), IEEE (2010), 1--12.Google ScholarCross Ref
- Luszczek, P.R., Bailey, D.H., Dongarra, J.J., et al. The HPC Challenge (HPCC) benchmark suite. Proceedings of the 2006 ACM/IEEE conference on Supercomputing - SC '06, ACM Press (2006), 213. Google ScholarDigital Library
- Mehrotra, P., Djomehri, J., Heistand, S., et al. Performance evaluation of Amazon EC2 for NASA HPC applications. Proceedings of the 3rd workshop on Scientific Cloud Computing Date - ScienceCloud '12, ACM Press (2012), 41--49. Google ScholarDigital Library
- Miller, M.A., Pfeiffer, W., and Schwartz, T. Creating the CIPRES Science Gateway for inference of large phylogenetic trees. Gateway Computing Environments Workshop (GCE), 2010, (2010).Google ScholarCross Ref
- Pekurovsky, D. P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions. SIAM Journal on Scientific Computing 34, 4 (2012), C192--C209.Google Scholar
- Shainer, G., Liu, T., Michalakes, J., et al. Weather Research and Forecast (WRF) Model: Performance Analysis on Advanced Multi-core HPC Clusters. The 10th LCI International Conference on High-Performance Clustered Computing, (2009).Google Scholar
- Sivagnanam, S., Astakhov, V., Yoshimoto, K., et al. A neuroscience gateway: software and implementation. Proceedings of the Conference on Extreme Science and Engineering Discovery Environment Gateway to Discovery-XSEDE '13, ACM Press (2013). Google ScholarDigital Library
- Skamarock, W.C., Klemp, J.B., Dudhia, J., et al. NCAR Technical Note NCAR/TN--475+STR: A Description of the Advanced Research WRF Version 3. Boulder, CO, 2008.Google Scholar
- QuantumESPRESSO - Deisa. 2008. http://www.deisa.eu/science/benchmarking/codes/quantumespresso.Google Scholar
- Single Root I/O Virtualization and Sharing 1.1 specification. 2010. http://www.pcisig.com/specifications/iov/single_root/.Google Scholar
- HPC Advisory Council Best Practices. 2010. http://www.hpcadvisorycouncil.com/best_practices.php.Google Scholar
- AMD I/O Virtualization Technology (IOMMU) Specification. 2011. http://developer.amd.com/wordpress/media/2012/10/48882.pdf.Google Scholar
- OSU Microbenchmarks. 2011. http://mvapich.cse.ohio-state.edu/benchmarks/.Google Scholar
- Intel Virtualization Technology for Directed I/O. 2013. http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf.Google Scholar
Index Terms
- SR-IOV: Performance Benefits for Virtualized Interconnects
Recommendations
A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices
Special TopicsSingle-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (physical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly ...
MVAPICH2 over openstack with SR-IOV: an efficient approach to build HPC clouds
CCGRID '15: Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid ComputingCloud Computing with Virtualization offers attractive flexibility and elasticity to deliver resources by providing a platform for consolidating complex IT resources in a scalable manner. However, efficiently running HPC applications on Cloud Computing ...
Towards the InfiniBand SR-IOV vSwitch Architecture
CLUSTER '15: Proceedings of the 2015 IEEE International Conference on Cluster ComputingTo meet the demands of the Exascale era and facilitate Big Data analytics in the cloud while maintaining flexibility, cloud providers will have to offer efficient virtualized High Performance Computing clusters in a pay-as-you-go model. As a consequence,...
Comments