Efficient live migration of virtual machines using shared storage

Abstract

Live migration of virtual machines (VM) across distinct physical hosts is an important feature of virtualization technology for maintenance, load-balancing and energy reduction, especially so for data centers operators and cluster service providers. Several techniques have been proposed to reduce the downtime of the VM being transferred, often at the expense of the total migration time. In this work, we present a technique to reduce the total time required to migrate a running VM from one host to another while keeping the downtime to a minimum. Based on the observation that modern operating systems use the better part of the physical memory to cache data from secondary storage, our technique tracks the VM's I/O operations to the network-attached storage device and maintains an updated mapping of memory pages that currently reside in identical form on the storage device. During the iterative pre-copy live migration process, instead of transferring those pages from the source to the target host, the memory-to-disk mapping is sent to the target host which then fetches the contents directly from the network-attached storage device. We have implemented our approach into the Xen hypervisor and ran a series of experiments with Linux HVM guests. On average, the presented technique shows a reduction of up over 30% on average of the total transfer time for a series of benchmarks.

References

1. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. A view of cloud computing. Communications of the ACM, 53(4):50--58, April 2010.
2. Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP '03, pages 164--177, New York, NY, USA, 2003. ACM.
3. Fabrice Bellard. QEMU. http://www.qemu.org, 2013. Online; accessed February 2013.
4. Ravi Bhargava, Benjamin Serebrin, Francesco Spadini, and Srilatha Manne. Accelerating two-dimensional page walks for virtualized systems. In Proceedings of the 13th international conference on Architectural support for programming languages and operating systems, ASPLOS XIII, pages 26--35, New York, NY, USA, 2008. ACM.
5. Nilton Bila, Eyal de Lara, Kaustubh Joshi, H. Andrés Lagar-Cavilla, Matti Hiltunen, and Mahadev Satyanarayanan. Jettison: efficient idle desktop consolidation with partial vm migration. In Proceedings of the 7th ACM european conference on Computer Systems, EuroSys '12, pages 211--224, New York, NY, USA, 2012. ACM.
6. Robert Bradford, Evangelos Kotsovinos, Anja Feldmann, and Harald Schiöberg. Live wide-area migration of virtual machines including local persistent state. In Proceedings of the 3rd international conference on Virtual execution environments, VEE '07, pages 169--179, New York, NY, USA, 2007. ACM.
7. Canonical Ltd. Ubuntu. http://www.ubuntu.com, 2013. Online; accessed February 2013.
8. Citrix Systems, Inc. Xen Hypervisor. http://www.xen.org/products/xenhyp.html, 2012. Online; accessed February 2013.
9. Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, and Andrew Warfield. Live migration of virtual machines. In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation - Volume 2, NSDI'05, pages 273--286, Berkeley, CA, USA, 2005. USENIX Association.
10. Brendan Cully, Geoffrey Lefebvre, Dutch Meyer, Mike Feeley, Norm Hutchinson, and Andrew Warfield. Remus: high availability via asynchronous virtual machine replication. In NSDI'08: Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, pages 161--174, Berkeley, CA, USA, 2008. USENIX Association.
11. Umesh Deshpande, Xiaoshuang Wang, and Kartik Gopalan. Live gang migration of virtual machines. In Proceedings of the 20th international symposium on High performance distributed computing, HPDC '11, pages 135--146, New York, NY, USA, 2011. ACM.
12. Irfan Habib. Virtualization with KVM. Linux Journal, 2008(166), February 2008.
13. Michael R. Hines and Kartik Gopalan. Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '09, pages 51--60, New York, NY, USA, 2009. ACM.
14. Takahiro Hirofuchi, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. Reactive consolidation of virtual machines enabled by postcopy live migration. In Proceedings of the 5th international workshop on Virtualization technologies in distributed computing, VTDC '11, pages 11--18, New York, NY, USA, 2011. ACM.
15. Wei Huang, Qi Gao, Jiuxing Liu, and Dhabaleswar K. Panda. High performance virtual machine migration with RDMA over modern interconnects. In Proceedings of the 2007 IEEE International Conference on Cluster Computing, CLUSTER '07, pages 11--20, Washington, DC, USA, 2007. IEEE Computer Society.
16. Hai Jin, Li Deng, Song Wu, Xuanhua Shi, and Xiaodong Pan. Live virtual machine migration with adaptive, memory compression. In Cluster Computing and Workshops, 2009. CLUSTER '09. IEEE International Conference on, pages 1--10, 31 2009-sept. 4 2009.
17. Jeffrey Katcher. PostMark: A New File System Benchmark. Technical Report Technical Report TR3022, Network Appliance, October 1997.
18. Haikun Liu, Hai Jin, Xiaofei Liao, Liting Hu, and Chen Yu. Live migration of virtual machine based on full system trace and replay. In Proceedings of the 18th ACM international symposium on High performance distributed computing, HPDC '09, pages 101--110, New York, NY, USA, 2009. ACM.
19. Raymond A. Lorie. Physical integrity in a large segmented database. ACM Transactions on Database Systems, 2(1):91--104, March 1977.
20. Michael Nelson, Beng-Hong Lim, and Greg Hutchins. Fast transparent migration for virtual machines. In Proceedings of the annual conference on USENIX Annual Technical Conference, ATEC '05, pages 25--25, Berkeley, CA, USA, 2005. USENIX Association.
21. Oracle. VirtualBox. https://www.virtualbox.org, 2012. Online; accessed February 2013.
22. Eunbyung Park, Bernhard Egger, and Jaejin Lee. Fast and space-efficient virtual machine checkpointing. In Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '11, pages 75--86, New York, NY, USA, 2011. ACM.
23. Tristan Richardson. The RFB protocol. http://www.realvnc.com/docs/rfbproto.pdf, 2010. Online; accessed February 2013.
24. Petter Svärd, Benoit Hudzia, Johan Tordsson, and Erik Elmroth. Evaluation of delta compression techniques for efficient live migration of large virtual machines. In Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '11, pages 111--120, New York, NY, USA, 2011. ACM.
25. The Document Foundation. LibreOffice. http://www.libreoffice.org, 2013. Online; accessed February 2013.
26. The Mozilla Foundation. Firefox. http://www.mozilla.org, 2013. Online; accessed February 2013.
27. Franco Travostino. Seamless live migration of virtual machines over the MAN/WAN. In Proceedings of the 2006 ACM/IEEE conference on Supercomputing, SC '06, New York, NY, USA, 2006. ACM.
28. VMware. VMware VMotion: Live migration of virtual machines without service interruption. http://www.vmware.com/files/pdf/VMware-VMotion-DS-EN.pdf, 2009. Online; accessed February 2013.

Index Terms

• Networks

  • Network protocols

• Software and its engineering

  • Software organization and properties

    • Extra-functional properties

      • Software fault tolerance

        • Checkpoint / restart