Shrinking the hypervisor one subsystem at a time: a userspace packet switch for virtual machines

Abstract

Efficient and secure networking between virtual machines is crucial in a time where a large share of the services on the Internet and in private datacenters run in virtual machines. To achieve this efficiency, virtualization solutions, such as Qemu/KVM, move toward a monolithic system architecture in which all performance critical functionality is implemented directly in the hypervisor in privileged mode. This is an attack surface in the hypervisor that can be used from compromised VMs to take over the virtual machine host and all VMs running on it.

We show that it is possible to implement an efficient network switch nfor virtual machines as an unprivileged userspace component running in the host system including the driver for the upstream network adapter. Our network switch relies on functionality already present in the KVM hypervisor and requires no changes to Linux, the host operating system, and the guest.

Our userspace implementation compares favorably to the existing in-kernel implementation with respect to throughput and latency. We reduced per-packet overhead by using a run-to-completion model an are able to outperform the unmodified system for VM-to-VM traffic by a large margin when packet rates are high.

References

1. Memtest86+ - an advanced memory diagnostic tool. URL http://www.memtest.org/.
2. N. Amit, M. Ben-Yehuda, D. Tsafrir, and A. Schuster. viommu: Efficient iommu emulation. In Proceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC'11, pages 6--6, Berkeley, CA, USA, 2011. USENIX Association. URL http://dl.acm.org/citation.cfm?id=2002181.2002187.
3. F. Bellard. Qemu, a fast and portable dynamic translator. In Proceedings of the annual conference on USENIX Annual Technical Conference, ATEC '05, pages 41--41, Berkeley, CA, USA, 2005. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1247360.1247401.
4. T. Benson, A. Akella, and D. A. Maltz. Network traffic char- acteristics of data centers in the wild. In Proceedings of the 10th ACM SIGCOMM conference on Internet measurement, IMC '10, pages 267--280, New York, NY, USA, 2010. ACM. ISBN 978--1--4503-0483--2. . URL http://doi.acm.org/10.1145/1879141.1879175.
5. P. Colp, M. Nanavati, J. Zhu, W. Aiello, G. Coker, T. Deegan, P. Loscocco, and A. Warfield. Breaking up is hard to do: Security and functionality in a commodity hypervisor. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP '11, pages 189--202, New York, NY, USA, 2011. ACM. ISBN 978--1--4503-0977--6. . URL http://doi.acm.org/10.1145/2043556.2043575.
6. Z. Cui, P. G. Bridges, J. R. Lange, and P. A. Dinda. Virtual TCP offload: optimizing ethernet overlay performance on advanced interconnects. In Proceedings of the 22nd international symposium on High-performance parallel and distributed computing, HPDC '13, pages 49--60, New York, NY, USA, 2013. ACM. ISBN 978--1--4503--1910--2. . URL http://doi.acm.org/10.1145/2462902.2462912.
7. M. Desnoyers, P. McKenney, A. Stern, M. Dagenais, and J. Walpole. User-level implementations of read-copy update. Parallel and Distributed Systems, IEEE Transactions on , 23 (2):375--382, 2012. ISSN 1045--9219. .
8. K. Elphinstone and G. Heiser. From L3 to seL4 -- what have we learnt in 20 years of L4 microkernels? In ACM SIGOPS Symposium on Operating Systems Principles (SOSP), pages 133--150, Farmington, PA, USA, November 2013.
9. genode. Genode operating system framework. URL http://www.genode.org/.
10. A. Gordon, N. Har'El, A. Landau, M. Ben-Yehuda, and A. Traeger. Towards exitless and efficient paravirtual i/o. In Proceedings of the 5th Annual International Systems and Storage Conference, SYSTOR '12, pages 10:1--10:6, New York, NY, USA, 2012. ACM. ISBN 978--1--4503--1448-0. . URL http://doi.acm.org/10.1145/2367589.2367593.
11. J. N. Herder, H. Bos, B. Gras, P. Homburg, and A. S. Tanenbaum. Minix 3: A highly reliable, self-repairing operating system. SIGOPS Oper. Syst. Rev., 40(3):80--89, July 2006. ISSN 0163--5980. . URL http://doi.acm.org/10.1145/1151374.1151391.
12. T. Hruby, D. Vogt, H. Bos, and A. S. Tanenbaum. Keep net working - on a dependable and fast networking stack. In Proceedings of Dependable Systems and Networks (DSN 2012), Boston, MA, June 2012.
13. A. Kantee. Rump file systems: kernel code reborn. In Proceedings of the 2009 conference on USENIX Annual technical conference, USENIX'09, pages 15--15, Berkeley, CA, USA, 2009. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1855807.1855822.
14. A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. KVM: the Linux virtual machine monitor. In Proceedings of the Linux Symposium, volume 1, pages 225--230, 2007.
15. A. Lackorzynski and A. Warg. Taming subsystems: capabilities as universal resource access control in L4. In Proceedings of the Second Workshop on Isolation and Integration in Embedded Systems, IIES '09, pages 25--30, New York, NY, USA, 2009. ACM. ISBN 978--1--60558--464--5. . URL http://doi.acm.org/10.1145/1519130.1519135.
16. B. Leslie, P. Chubb, N. Fitzroy-Dale, S. Götz, C. Gray, L. Macpherson, D. Potts, Y.-T. Shen, K. Elphinstone, and G. Heiser. User-level device drivers: Achieved performance. Journal of Computer Science and Technology, 20(5):654--664, 2005. ISSN 1000--9000. . URL http://dx.doi.org/10.1007/s11390-005-0654--4.
17. A. Menon, A. L. Cox, and W. Zwaenepoel. Optimizing network virtualization in Xen. In Proceedings of the annual conference on USENIX '06 Annual Technical Conference, ATEC'06, pages 2--2, Berkeley, CA, USA, 2006. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1267359.1267361.
18. J. Nakajima. Enabling optimized interrupt/APIC virtualization in KVM. In KVM Forum, 2012.
19. netperf. netperf. URL http://www.netperf.org/.
20. nuttcp. nuttcp network performance measurement tool. URL https://www.nuttcp.net/.
21. K. K. Ram, A. L. Cox, M. Chadha, and S. Rixner. Hyper-Switch: A scalable software virtual switching architecture. In Proceedings of the 2013 USENIX conference on Annual Technical Conference, USENIX ATC'13, Berkeley, CA, USA, 2013. USENIX Association.
22. L. Rizzo. Netmap: a novel framework for fast packet I/O. In Proceedings of the 2012 USENIX conference on Annual Technical Conference, USENIX ATC'12, pages 9--9, Berkeley, CA, USA, 2012. USENIX Association. URL http://dl.acm.org/citation.cfm?id=2342821.2342830.
23. L. Rizzo and G. Lettieri. VALE, a switched ethernet for virtual machines. In Proceedings of the 8th international conference on Emerging networking experiments and technologies, CoNEXT '12, pages 61--72, New York, NY, USA, 2012. ACM. ISBN 978--1--4503--1775--7. URL http://doi.acm.org/10.1145/2413176.2413185.
24. R. Russel. virtio: towards a de-facto standard for virtual I/Odevices. SIGOPS Operating Systems Review, 42(5):95--103, 2008.
25. L. Shalev, J. Satran, E. Borovik, and M. Ben-Yehuda. IsoStack: Highly Efficient Network Processing on Dedicated Cores. In Proceedings of the 2010 USENIX conference on USENIX annual technical conference, USENIX ATC'10, pages 5--5, Berkeley, CA, USA, 2010. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1855840.1855845.
26. snabb. Snabbswitch. URL https://github.com/SnabbCo/snabbswitch/wiki.
27. L. Soares and M. Stumm. Flexsc: Flexible system call scheduling with exception-less system calls. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, OSDI'10, pages 1--8, Berkeley, CA, USA, 2010. USENIX Association. URL http://dl.acm.org/citation.cfm?id=1924943.1924946.
28. U. Steinberg and B. Kauer. NOVA: a microhypervisor-based secure virtualization architecture. In Proceedings of the 5th European conference on Computer systems, EuroSys '10, pages 209--222, New York, NY, USA, 2010. ACM. ISBN 978--1--60558--577--2. . URL http://doi.acm.org/10.1145/1755913.1755935.
29. R. Uhlig, G. Neiger, D. Rodgers, A. L. Santoni, F. C. M. Martins, A. V. Anderson, S. M. Bennett, A. Kagi, F. H. Leung, and L. Smith. Intel virtualization technology. Computer, 38(5):48--56, May 2005. ISSN 0018--9162. . URL http://dx.doi.org/10.1109/MC.2005.163.
30. vfio. VFIO driver: Non-privileged user level pci drivers, 2010. URL http://lwn.net/Articles/391459/.
31. G. Wang and T. Ng. The impact of virtualization on network performance of amazon ec2 data center. In INFOCOM, 2010 Proceedings IEEE, pages 1--9, 2010. .
32. D. Wentzlaff and A. Agarwal. Factored operating systems (fos): the case for a scalable operating system for multicores. SIGOPS Oper. Syst. Rev., 43(2):76--85, Apr. 2009. ISSN 0163--5980. . URL http://doi.acm.org/10.1145/1531793.1531805.

Index Terms

• Networks

  • Network protocols

• Software and its engineering

  • Software creation and management

    • Designing software

  • Software organization and properties

    • Contextual software domains

      • Operating systems