A survey of migration mechanisms of virtual machines

Abstract

In the virtualization area, replication has been considered as a mechanism to provide high availability. A high-availability system should be active most of the time, and this is the reason that its design should consider almost zero downtime and a minimal human intervention if a recovery process is demanded. Several migration and replication mechanisms have been developed to provide high availability inside virtualized environments. In this article, a survey of migration mechanisms is reported. These approaches are classified in three main classes: process migration, memory migration, and suspend/resume migration.

References

1. 2009. The NIST Definition of Cloud Computing. National Institute of Standards and Technology 53, 6 (2009), 50. http://csrc.nist.gov/groups/SNS/cloud-computing/cloud-def-v15.doc
2. Silviu Andrica and George Candea. 2011. WaRR: A tool for high-fidelity web application record and replay. In Proceedings of the 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN'11). IEEE Computer Society, Washington, DC, 403--410. DOI: http://dx.doi.org/10.1109/DSN.2011.5958253
3. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A view of cloud computing. Commun. ACM 53, 4 (April 2010), 50--58. DOI: http://dx.doi.org/10.1145/1721654.1721672
4. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy H. Katz, Andrew Konwinski, Gunho Lee, David A. Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2009. Above the Clouds: A Berkeley View of Cloud Computing. Technical Report UCB/EECS-2009-28. EECS Department, University of California, Berkeley. Retrieved from http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.html
5. Muhammad Atif and Peter Strazdins. 2009. Optimizing live migration of virtual machines in SMP clusters for HPC applications. In Proceedings of the IFIP International Conference on Network and Parallel Computing Workshops. 51--58. DOI:http://dx.doi.org/10.1109/NPC.2009.32
6. Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the art of virtualization. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP'03). ACM, New York, NY, 164--177. http://dx.doi.org/10.1145/945445.945462
7. N. Bobroff, A. Kochut, and K. Beaty. 2007. Dynamic placement of virtual machines for managing SLA violations. In Proceedings of the 10th IFIP/IEEE International Symposium on Integrated Network Management (IM'07). 119--128. DOI: http://dx.doi.org/10.1109/INM.2007.374776
8. Robert Bradford, Evangelos Kotsovinos, Anja Feldmann, and Harald Schiöberg. 2007. Live wide-area migration of virtual machines including local persistent state. In Proceedings of the 3rd International Conference on Virtual Execution Environments (VEE'07). ACM, New York, NY, 169--179. http://dx.doi.org/10.1145/1254810.1254834
9. Ramón Cáceres, Casey Carter, Chandra Narayanaswami, and Mandayam Raghunath. 2005. Reincarnating PCs with portable SoulPads. In Proceedings of the 3rd International Conference on Mobile Systems, Applications, and Services (MobiSys'05). ACM, New York, NY, 65--78. http://dx.doi.org/10.1145/1067170.1067179
10. Ramesh Chandra, Nickolai Zeldovich, Constantine Sapuntzakis, and Monica S. Lam. 2005. The collective: a cache-based system management architecture. In Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation - Volume 2 (NSDI'05). USENIX Association, Berkeley, CA, 259--272.
11. Prabhakar Changarti. 2007. Xen Virtualization: A Practical Handbook (1st ed.). Packt Publishing Ltd., Birmingham, UK.
12. Peter M. Chen and Brian D. Noble. 2001. When virtual is better than real. In Proceedings of the 8th Workshop on Hot Topics in Operating Systems (HOTOS'01). IEEE Computer Society, Washington, DC, 133.
13. David Chisnall. 2007. The Definitive Guide to the Xen Hypervisor. Prentice Hall, Upper Saddle River, NJ.
14. Byung-Gon Chun, Sunghwan Ihm, Petros Maniatis, Mayur Naik, and Ashwin Patti. 2011. CloneCloud: elastic execution between mobile device and cloud. In Proceedings of the 6th Conference on Computer systems (EuroSys'11). ACM, New York, NY, 301--314. DOI: http://dx.doi.org/10.1145/1966445.1966473
15. Byung-Gon Chun and Petros Maniatis. 2009. Augmented smartphone applications through clone cloud execution. In Proceedings of the 12th Conference on Hot Topics in Operating Systems (HotOS'09). USENIX Association, Berkeley, CA, 8--8. http://dl.acm.org/citation.cfm?id=1855568.1855576
16. Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, and Andrew Warfield. 2005. Live migration of virtual machines. In Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation (NSDI'05). USENIX Association, Berkeley, CA, 273--286.
17. Brendan Cully, Geoffrey Lefebvre, Dutch Meyer, Mike Feeley, Norm Hutchinson, and Andrew Warfield. 2008. Remus: High availability via asynchronous virtual machine replication. In Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation (NSDI'08). USENIX Association, Berkeley, CA, 161--174.
18. Rogier Dittner and David Rule. 2007. The Best Damn Server Virtualization Book Period: Including VMware, Xen, and Microsoft Virtual Server. Syngress Publishing.
19. George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and Peter M. Chen. 2002. ReVirt: enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev. 36, SI (December 2002), 211--224. http://dx.doi.org/10.1145/844128.844148
20. George W. Dunlap, Dominic G. Lucchetti, Michael A. Fetterman, and Peter M. Chen. 2008. Execution replay of multiprocessor virtual machines. In Proceedings of the 4th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE'08). ACM, New York, NY, 121--130. DOI: http://dx.doi.org/10.1145/1346256.1346273
21. R. P. Goldberg. 1974. Survey of virtual machine research. IEEE Computer Magazine (June 1974), 34--45.
22. Thilina Gunarathne, Tak-Lon Wu, Judy Qiu, and Geoffrey Fox. 2010. Cloud computing paradigms for pleasingly parallel biomedical applications. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (HPDC'10). ACM, New York, NY, 460--469. DOI: http://dx.doi.org/10.1145/1851476.1851544
23. Diwaker Gupta, Ludmila Cherkasova, Rob Gardner, and Amin Vahdat. 2006. Enforcing performance isolation across virtual machines in Xen. In Proceedings of the ACM/IFIP/USENIX 2006 International Conference on Middleware (Middleware'06). Springer-Verlag, New York, NY, 342--362.
24. Jacob Gorm Hansen and Eric Jul. 2004. Self-migration of operating systems. In Proceedings of the 11th ACM SIGOPS European Workshop (EW'11). ACM, New York, NY, Article 23. http://dx.doi.org/10.1145/1133572.1133616
25. Michael R. Hines, Umesh Deshpande, and Kartik Gopalan. 2009. Post-copy live migration of virtual machines. SIGOPS Oper. Syst. Rev. 43, 3 (July 2009), 14--26. DOI: http://dx.doi.org/10.1145/1618525.1618528
26. Takahiro Hirofuchi, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. 2011. Reactive consolidation of virtual machines enabled by postcopy live migration. In Proceedings of the 5th International Workshop on Virtualization Technologies in Distributed Computing (VTDC'11). ACM, New York, NY, 11--18. DOI: http://dx.doi.org/10.1145/1996121.1996125
27. Takahiro Hirofuchi, Hidemoto Nakada, Hirotaka Ogawa, Satoshi Itoh, and Satoshi Sekiguchi. 2009. A live storage migration mechanism over wan and its performance evaluation. In Proceedings of the 3rd International Workshop on Virtualization Technologies in Distributed Computing (VTDC'09). ACM, New York, NY, 67--74. DOI: http://dx.doi.org/10.1145/1555336.1555348
28. Khaled Z. Ibrahim, Steven Hofmeyr, Costin Iancu, and Eric Roman. 2011. Optimized pre-copy live migration for memory intensive applications. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC'11). ACM, New York, NY, Article 40, 11 pages. DOI: http://dx.doi.org/10.1145/2063384.2063437
29. P. Iványi and B.H.V. Topping (Eds.). 2011. Threat Detection in an Urban Water Distribution Systems with Simulations Conducted in Grids and Clouds. Ajaccio, Corsica, France. DOI: http://dx.doi.org/10.4203/ccp.95.93
30. Bo Jiang, Binoy Ravindran, and Changsoo Kim. 2010. Lightweight live migration for high availability cluster service. In Proceedings of the 12th International Conference on Stabilization, Safety, and Security of Distributed Systems (SSS'10). Springer-Verlag, Berlin, 420--434. Retrieved from http://dl.acm.org/citation.cfm?id=1926829.1926865.
31. James J. Kistler and M. Satyanarayanan. 1992. Disconnected operation in the Coda File System. ACM Trans. Comput. Syst. 10, 1 (Feb. 1992), 3--25. DOI: http://dx.doi.org/10.1145/146941.146942
32. Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. 2007. KVM: The Linux virtual machine monitor. In Proceedings of the Ottawa Linux Symposium. 225--230. Retrieved from http://www.kernel.org/doc/ols/2007/ols2007v1-pages-225-230.pdf.
33. Nadir Kiyanclar. 2005. A survey of virtualization techniques focusing on secure on-demand cluster computing. CoRR abs/cs/0511010 (2005).
34. Michael Kozuch and M. Satyanarayanan. 2002. Internet suspend/resume. In Proceedings of the 4th IEEE Workshop on Mobile Computing Systems and Applications (WMCSA'02). IEEE Computer Society, Washington, DC, 40.
35. Michael A. Kozuch, Michael Kaminsky, and Michael P. Ryan. 2009. Migration without virtualization. In Proceedings of the 12th Conference on Hot Topics in Operating Systems (HotOS'09). USENIX Association, Berkeley, CA, 10.
36. Bo Li, Jianxin Li, Jinpeng Huai, Tianyu Wo, Qin Li, and Liang Zhong. 2009. EnaCloud: An energy-saving application live placement approach for cloud computing environments. In Proceedings of the 2009 IEEE International Conference on Cloud Computing (CLOUD'09). IEEE Computer Society, Washington, DC, 17--24. DOI: http://dx.doi.org/10.1109/CLOUD.2009.72
37. LINBIT. 2011. Distributed Replicated Block Device. Retrieved from http://www.drbd.org/.
38. Haikun Liu, Hai Jin, Xiaofei Liao, Liting Hu, and Chen Yu. 2009. Live migration of virtual machine based on full system trace and replay. In Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing (HPDC'09). ACM, New York, NY, 101--110. http://dx.doi.org/10.1145/1551609.1551630
39. Yingwei Luo, Binbin Zhang, Xiaolin Wang, Zhenlin Wang, Yifeng Sun, and Haogang Chen. 2008. Live and incremental whole-system migration of virtual machines using block-bitmap. In Proceedings of the IEEE International Conference on Cluster Computing. 99--106. DOI: http://dx.doi.org/10.1109/CLUSTR.2008.4663760
40. Kernel Based Virtual Machine. 2012. Main Page. Retrieved from http://www.linux-kvm.org/page/Main_Page.
41. Sean R. Marston, Zhi Li, Subhajyoti Bandyopadhyay, Anand Ghalsasi, and Juheng Zhang. 2009. Cloud computing: The business perspective. SSRN eLibrary (2009).
42. Dejan S. Milojičić, Fred Douglis, Yves Paindaveine, Richard Wheeler, and Songnian Zhou. 2000. Process migration. ACM Comput. Surv. 32, 3 (2000), 241--299. DOI: http://dx.doi.org/10.1145/367701.367728
43. Andrey Mirkin, Alexey Kuznetsov, and Kir Kolyshkin. 2008. Containers checkpointing and live migration. In Proceedings of the Linux Symposium 2008. 85--92.
44. Sape J. Mullender, Guido van Rossum, Andrew S. Tanenbaum, Robbert van Renesse, and Hans van Staveren. 1990. Amoeba: A distributed operating system for the 1990s. Computer 23, 5 (May 1990), 44--53. DOI: http://dx.doi.org/10.1109/2.53354
45. Ripal Nathuji and Karsten Schwan. 2007. VirtualPower: Coordinated power management in virtualized enterprise systems. SIGOPS Oper. Syst. Rev. 41, 6 (Oct. 2007), 265--278. http://dx.doi.org/10.1145/1323293.1294287
46. Michael Nelson, Beng-Hong Lim, and Greg Hutchins. 2005. Fast transparent migration for virtual machines. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (ATEC'05). USENIX Association, Berkeley, CA, USA, 25--25.
47. National Energy Research Scientific Computer Center NERSC. 2011. NERSC Mission. Retrieved from http://www.nersc.gov/about/nersc-mission/.
48. Jason Nieh. 2005. AutoPod: Unscheduled system updates with zero data loss. In Proceedings of the 2nd International Conference on Automatic Computing. IEEE Computer Society, Washington, DC, 367--368. DOI: http://dx.doi.org/10.1109/ICAC.2005.16
49. Steven Osman, Dinesh Subhraveti, Gong Su, and Jason Nieh. 2002. The design and implementation of Zap: a system for migrating computing environments. SIGOPS Oper. Syst. Rev. 36, SI (December 2002), 361--376. DOI: http://dx.doi.org/10.1145/844128.844162
50. P. Padala, X. Zhu, Z. Wang, S. Singhal, and K. Shin. 2007. Performance Evaluation of Virtualization Technologies for Server Consolidation. Technical Report. HP Labs Tech.
51. Future Grid Portal. 2010. Future Grid. Retrieved from https://portal.futuregrid.org/.
52. Mendel Rosenblum. 2004. The reincarnation of virtual machines. Queue 2, 5 (2004), 34--40. http://dx.doi.org/10.1145/1016998.1017000
53. Mendel Rosenblum and Tal Garfinkel. 2005. Virtual machine monitors: Current technology and future trends. Computer 38, 5 (May 2005), 39--47. DOI: http://dx.doi.org/10.1109/MC.2005.176
54. M. Rozier, V. Abrossimov, F. Armand, I. Boule, M. Gien, M. Guillemont, F. Herrmann, C. Kaiser, S. Langlois, P. Léonard, and W. Neuhauser. 1991. Overview of the CHORUS Distributed Operating Systems. Computing Systems 1 (1991), 39--69.
55. J. Sahoo, S. Mohapatra, and R. Lath. 2010. Virtualization: A survey on concepts, taxonomy and associated security issues. In Proceedings of the 2nd International Conference on Computer and Network Technology (ICCNT'10). 222--226. DOI: http://dx.doi.org/10.1109/ICCNT.2010.49
56. Constantine P. Sapuntzakis, Ramesh Chandra, Ben Pfaff, Jim Chow, Monica S. Lam, and Mendel Rosenblum. 2002. Optimizing the migration of virtual computers. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation. 377--390.
57. Mahadev Satyanarayanan, Paramvir Bahl, Ramón Caceres, and Nigel Davies. 2009. The case for VM-based cloudlets in mobile computing. IEEE Pervasive Computing 8, 4 (Oct. 2009), 14--23. DOI: http://dx.doi.org/10.1109/MPRV.2009.82
58. Mahadev Satyanarayanan, B. Gilbert, M. Toups, N. Tolia, D. R. O'Hallaron, Ajay Surie, A. Wolbach, J. Harkes, A. Perrig, D. J. Farber, M. A. Kozuch, C. J. Helfrich, P. Nath, and H. A. Lagar-Cavilla. 2007. Pervasive personal computing in an Internet suspend/resume system. IEEE Internet Computing 11, 2 (March-April 2007), 16--25. DOI: http://dx.doi.org/10.1109/MIC.2007.46
59. M. Satyanarayanan, Michael A. Kozuch, Casey J. Helfrich, and David R. O'Hallaron. 2005. Towards seamless mobility on pervasive hardware. Pervasive Mob. Comput. 1, 2 (July 2005), 157--189. DOI: http://dx.doi.org/10.1016/j.pmcj.2005.03.005
60. Aameek Singh, Madhukar Korupolu, and Dushmanta Mohapatra. 2008. Server-storage virtualization: integration and load balancing in data centers. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (SC'08). IEEE Press, Piscataway, NJ, Article 53, 12 pages. http://dl.acm.org/citation.cfm?id=1413370.1413424
61. Kunwadee Sripanidkulchai, Sambit Sahu, Yaoping Ruan, Anees Shaikh, and Chitra Dorai. 2010. Are clouds ready for large distributed applications? SIGOPS Oper. Syst. Rev. 44, 2 (April 2010), 18--23. DOI: http://dx.doi.org/10.1145/1773912.1773918
62. Ajay Surie, H. Andrés Lagar-Cavilla, Eyal de Lara, and M. Satyanarayanan. 2008. Low-bandwidth VM migration via opportunistic replay. In Proceedings of the 9th Workshop on Mobile Computing Systems and Applications (HotMobile'08). ACM, New York, NY, 74--79. http://dx.doi.org/10.1145/1411759.1411779
63. Nanda Susanta and Chiueh Tzi-Cker. 2005. A Survey on Virtualization Technologies. Technical Report. Stony Brook University. http://www.ecsl.cs.sunysb.edu/tr/TR179.pdf
64. Yoshi Tamura. 2008. Kemari: Virtual machine synchronization for fault tolerance using DomT. In Proceedings of the Xen Summit.
65. Andrew S. Tanenbaum. 1995. Distributed Operating Systems. Prentice Hall.
66. Andrew S. Tanenbaum and Maarten Van Steen. 2001. Distributed Systems: Principles and Paradigms (1st ed.). Prentice Hall, Upper Saddle River, NJ.
67. J. Torres, D. Carrera, K. Hogan, R. Gavalda, V. Beltran, and N. Poggi. 2008. Reducing wasted resources to help achieve green data centers. In Proceedings of the IEEE International Symposium on Parallel and Distributed Processing (IPDPS'08). 1--8. DOI: http://dx.doi.org/10.1109/IPDPS.2008.4536219
68. Franco Travostino. 2006. Seamless live migration of virtual machines over the MAN/WAN. In Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (SC'06). ACM, New York, NY, 290. http://dx.doi.org/10.1145/1188455.1188758
69. Stephen J. Vaughan-Nichols. 2006. New approach to virtualization is a lightweight. Computer 39, 11 (Nov. 2006), 12--14. DOI: http://dx.doi.org/10.1109/MC.2006.393
70. Akshat Verma, Puneet Ahuja, and Anindya Neogi. 2008. pMapper: Power and migration cost aware application placement in virtualized systems. In Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware (Middleware'08). Springer-Verlag, New York, NY, 243--264.
71. VMware. 2012. VMware GSX Server Documentation. Retrieved from http://www.vmware.com/support/pubs/gsx_pubs.html.
72. Jens-Sönke Vöckler, Gideon Juve, Ewa Deelman, Mats Rynge, and Bruce Berriman. 2011. Experiences using cloud computing for a scientific workflow application. In Proceedings of the 2nd International Workshop on Scientific Cloud Computing (ScienceCloud'11). ACM, New York, NY, 15--24.
73. Carl A. Waldspurger. 2002. Memory resource management in VMware ESX server. SIGOPS Oper. Syst. Rev. 36, SI (December 2002), 181--194. DOI: http://dx.doi.org/10.1145/844128.844146
74. A. Whitaker, R. S. Cox, M. Shaw, and S. D. Gribble. 2005. Rethinking the design of virtual machine monitors. Computer 38, 5 (May 2005), 57--62. DOI: http://dx.doi.org/10.1109/MC.2005.169
75. Dan Williams, Hani Jamjoom, and Hakim Weatherspoon. 2012. The Xen-Blanket: virtualize once, run everywhere. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys'12). ACM, New York, NY, 113--126. DOI: http://dx.doi.org/10.1145/2168836.2168849
76. Timothy Wood, K. K. Ramakrishnan, Prashant Shenoy, and Jacobus van der Merwe. 2011. CloudNet: Dynamic pooling of cloud resources by live WAN migration of virtual machines. SIGPLAN Not. 46, 7 (March 2011), 121--132. DOI: http://dx.doi.org/10.1145/2007477.1952699
77. Timothy Wood, Prashant Shenoy, Arun Venkataramani, and Mazin Yousif. 2007. Black-box and gray-box strategies for virtual machine migration. In Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation (NSDI'07). USENIX Association, Berkeley, CA, 17--17.

Index Terms

• Computer systems organization

  • Architectures

    • Distributed architectures

• Hardware

  • Communication hardware, interfaces and storage

• Networks

  • Network protocols

• Software and its engineering

  • Software organization and properties

    • Software system structures

      • Distributed systems organizing principles