Bringing Virtualization to the x86 Architecture with the Original VMware Workstation

Abstract

This article describes the historical context, technical challenges, and main implementation techniques used by VMware Workstation to bring virtualization to the x86 architecture in 1999. Although virtual machine monitors (VMMs) had been around for decades, they were traditionally designed as part of monolithic, single-vendor architectures with explicit support for virtualization. In contrast, the x86 architecture lacked virtualization support, and the industry around it had disaggregated into an ecosystem, with different vendors controlling the computers, CPUs, peripherals, operating systems, and applications, none of them asking for virtualization. We chose to build our solution independently of these vendors.

As a result, VMware Workstation had to deal with new challenges associated with (i) the lack of virtualization support in the x86 architecture, (ii) the daunting complexity of the architecture itself, (iii) the need to support a broad combination of peripherals, and (iv) the need to offer a simple user experience within existing environments. These new challenges led us to a novel combination of well-known virtualization techniques, techniques from other domains, and new techniques.

VMware Workstation combined a hosted architecture with a VMM. The hosted architecture enabled a simple user experience and offered broad hardware compatibility. Rather than exposing I/O diversity to the virtual machines, VMware Workstation also relied on software emulation of I/O devices. The VMM combined a trap-and-emulate direct execution engine with a system-level dynamic binary translator to efficiently virtualize the x86 architecture and support most commodity operating systems. By relying on x86 hardware segmentation as a protection mechanism, the binary translator could execute translated code at near hardware speeds. The binary translator also relied on partial evaluation and adaptive retranslation to reduce the overall overheads of virtualization.

Written with the benefit of hindsight, this article shares the key lessons we learned from building the original system and from its later evolution.

References

1. Adams, K. and Agesen, O. 2006. A comparison of software and hardware techniques for x86 virtualization. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-XII). 2--13.
2. Agesen, O. 2006. Binary translation of returns. In Workshop on Binary Instrumentation and Applications. 7--14.
3. Agesen, O. and Sheldon, J. W. 2004. Restricting memory access to protect data when sharing a common address space. U.S. Patent 7,277,999.
4. Agesen, O., Garthwaite, A., Sheldon, J., and Subrahmanyam, P. 2010. The evolution of an x86 virtual machine monitor. Operating Systems Review 44, 4, 3--18.
5. AMD Corporation. 1998. Network Products: Ethernet Controllers Books 2.
6. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T. L., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A. 2003. Xen and the art of virtualization. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP). 164--177.
7. Barr, K., Bungale, P. P., Deasy, S., Gyuris, V., Hung, P., Newell, C., Tuch, H., and Zoppis, B. 2010. The VMware mobile virtualization platform: Is that a hypervisor in your pocket? Operating Systems Review 44, 4, 124--135.
8. Bellard, F. 2005. QEMU, a fast and portable dynamic translator. In Proceedings of the USENIX Annual Technical Conference, FREENIX Track. 41--46.
9. Bhargava, R., Serebrin, B., Spadini, F., and Manne, S. 2008. Accelerating two-dimensional page walks for virtualized systems. In Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-XIII). 26--35.
10. Birrell, A. and Nelson, B. J. 1984. Implementing remote procedure calls. ACM Trans. Comput. Syst. 2, 1, 39--59.
11. Bugnion, E. 1998. Dynamic binary translator with a system and method for updating and maintaining coherency of a translation cache. U.S. Patent 6,704,925.
12. Bugnion, E., Devine, S., Govil, K., and Rosenblum, M. 1997. Disco: Running commodity operating systems on scalable multiprocessors. ACM Trans. Comput. Syst. 15, 4, 412--447.
13. Bugnion, E., Devine, S. W., and Rosenblum, M. 1998. System and method for virtualizing computer systems. U.S. Patent 6,496,847.
14. Chappell, G. 1994. DOS Internals. Addisson-Wesley.
15. Chen, Y.-H. 2009. Dynamic binary translation from x86-32 code to x86-64 code for virtualization. M.S. thesis, Massachusetts Institute of Technology.
16. Cmelik, R. F. and Keppel, D. 1994. Shade: A fast instruction-set simulator for execution profiling. In Proceedings of the 1994 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems. 128--137.
17. Compaq, Phoenix, Intel. 1996. BIOS Boot Specification, v1.0.1. http://www.phoenix.com/resources/specs-bbs101.pdf.
18. Connectix Corporation. 2001. Connectix Virtual PC for Windows (Press Release). Retrieved on the Internet Archive’s Wayback Machine.
19. Creasy, R. 1981. The origin of the VM/370 Time-sharing system. IBM J. Res. Develop 25, 5, 483--490.
20. Custer, H. 1993. Inside Windows NT. Microsoft Press.
21. Dehnert, J. C., Grant, B., Banning, J. P., Johnson, R., Kistler, T., Klaiber, A., and Mattson, J. 2003. The Transmeta Code Morphing - Software: Using speculation, recovery, and adaptive retranslation to address real-life challenges. In Proceedings of the 1st IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 15--24.
22. Devine, S. W., Bugnion, E., and Rosenblum, M. 1998. Virtualization system including a virtual machine monitor for a computer with a segmented architecture. U.S. Patent 6,397,242.
23. Dike, J. 2001. User Mode Linux. In Proceedings of the 5th Annual Ottawa Linux Symposium (OLS).
24. Drepper, U. and Molnar, I. 2003. The Native POSIX Thread Library for Linux. RedHat White Paper.
25. Ebcioglu, K. and Altman, E. R. 1997. DAISY: Dynamic compilation for 100% architectural compatibility. In Proceedings of the 24th International Symposium on Computer Architecture (ISCA). 26--37.
26. Ford, B. and Cox, R. 2008. Vx32: Lightweight user-level sandboxing on the x86. In Proceedings of the USENIX Annual Technical Conference. 293--306.
27. Gelsinger, P. 1998. Personal Communication (Intel Corp. CTO).
28. Goldberg, R. P. 1972. Architectural principles for virtual computer systems. Ph.D. thesis, Harvard University, Cambridge, MA.
29. Goldberg, R. P. 1974. Survey of virtual machine research. IEEE Computer Magazine 7, 6, 34--45.
30. IDC. 2009. Server Virtualization Hits Inflection Point as Number of Virtual Machines to Exceed Physical Systems in 2009 (Press Release). http://www.idc.com/about/viewpressrelease.jsp?containerId=prUK21840309.
31. Intel Corporation. 2008. Intel Virtualization Technology FlexMigration Application Note. Tech. rep.
32. Intel Corporation. 2010. Intel64 and IA-32 Architectures Software Developer’s Manual Volume 2 (2A and 2B).
33. International Standards Organization. 1988. Information processing -- Volume and file structure of CD-ROM for information interchange. In ISO 9660-1988.
34. Jones, N. D. 1996. An introduction to partial evaluation. ACM Comput. Surv. 28, 3, 480--503.
35. King, A. 1995. Inside Windows 95. Microsoft Press.
36. King, S. T., Dunlap, G. W., and Chen, P. M. 2003. Operating system support for virtual machines. In USENIX Annual Technical Conference, General Track. 71--84.
37. Kivity, A. 2007. KVM: The Linux virtual machine monitor. In Proceedings of the 2007 Ottawa Linux Symposium (OLS). 225--230.
38. Lim, B.-H., Le, B. C., and Bugnion, E. 2000. Deferred shadowing of segment descriptors in a virtual machine monitor for a segmented computer architecture. U.S. Patent 6,785,886.
39. Ludloff, C. 1996. Sandpile: The world’s leading source for technical x86 processor information. http://www.sandpile.org.
40. Meushaw, R. and Simard, D. 2000. NetTop: Commercial technology in high assurance applications. NSA Tech Trend Notes 9, 4.
41. Microsoft Corporation. 2008. Windows Server 2008R2 Hyper-V. http://www.microsoft.com/en-us/server-cloud/windows-server/hyper-v.aspx.
42. Microsoft Corporation. 2009. Windows XP Mode and Windows Virtual PC. http://www.microsoft.com/windows/virtual-pc/.
43. Nelson, M., Lim, B.-H., and Hutchins, G. 2005. Fast transparent migration for virtual machines. In Proceedings of the USENIX Annual Technical Conference, General Track. 391--394.
44. Parallels Corporation. 2006. Parallels Desktop for the Mac. http://www.parallels.com/products/desktop/.
45. Popek, G. J. and Goldberg, R. P. 1974. Formal requirements for virtualizable third generation architectures. Commun. ACM 17, 7, 412--421.
46. Robin, J. S. and Irvine, C. E. 2000. Analysis of the Intel Pentium’s ability to support a secure virtual machine monitor. In Proceedings of the 9th Conference on USENIX Security Symposium, vol. 9.
47. Rosenblum, M., Herrod, S. A., Witchel, E., and Gupta, A. 1995. Complete computer system simulation: The SimOS approach. IEEE Parall. Distrib. Tech. 3, 34--43.
48. Rosenblum, M., Bugnion, E., Devine, S., and Herrod, S. A. 1997. Using the SimOS machine simulator to study complex computer systems. ACM Trans. Model. Comput. Simul. 7, 1, 78--103.
49. Sieh, V. and Buchacker, K. 2002. UMLinux -- A versatile SWIFI tool. In Proceedings of the 4th European Dependable Computing Conference (EDCC). 159--171.
50. Sites, R. L., Chernoff, A., Kirk, M. B., Marks, M. P., and Robinson, S. G. 1993. Binary translation. Commun. ACM 36, 2, 69--81.
51. Solomon, D. A. and Russinovich, M. E. 2000. Inside Microsoft Windows 2000 3rd Ed. Microsoft Press.
52. Sugerman, J., Venkitachalam, G., and Lim, B.-H. 2001. Virtualizing I/O devices on VMware Workstation’s hosted virtual machine monitor. In Proceedings of the USENIX Annual Technical Conference, General Track. 1--14.
53. Uhlig, R., Neiger, G., Rodgers, D., Santoni, A. L., Martins, F. C. M., Anderson, A. V., Bennett, S. M., Kägi, A., Leung, F. H., and Smith, L. 2005. Intel virtualization technology. IEEE Comput. 38, 5, 48--56.
54. Wahbe, R., Lucco, S., Anderson, T. E., and Graham, S. L. 1993. Efficient software-based fault isolation. In Proceedings of the 14th ACM Symposium on Operating Systems Principles (SOSP). 203--216.
55. Waldspurger, C. A. 2002. Memory resource management in VMware ESX server. In Proceedings of the 5th Symposium on Operating System Design and Implementation (OSDI).
56. Waldspurger, C. A. and Rosenblum, M. 2012. I/O virtualization. Commun. ACM 55, 1, 66--73.
57. Wheeler, D. A. 2001. SLOCCount. http://www.dwheeler.com/sloccount/.
58. Whitaker, A., Shaw, M., and Gribble, S. D. 2002. Scale and performance in the Denali isolation kernel. In Proceedings of the 5th Symposium on Operating System Design and Implementation (OSDI).
59. Witchel, E. and Rosenblum, M. 1996. Embra: Fast and flexible machine simulation. In Proceedings of the 1996 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems. 68--79.
60. Yee, B., Sehr, D., Dardyk, G., Chen, J. B., Muth, R., Ormandy, T., Okasaka, S., Narula, N., and Fullagar, N. 2010. Native client: A sandbox for portable, untrusted x86 native code. Commun. ACM 53, 1, 91--99.

Index Terms

• Computer systems organization

  • Architectures

• Security and privacy

  • Systems security

    • Operating systems security

• Software and its engineering

  • Software creation and management

    • Designing software

  • Software organization and properties

    • Contextual software domains

      • Operating systems