Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning

ABSTRACT

We present the design, implementation, and evaluation of post-copy based live migration for virtual machines (VMs) across a Gigabit LAN. Live migration is an indispensable feature in today's virtualization technologies. Post-copy migration defers the transfer of a VM's memory contents until after its processor state has been sent to the target host. This deferral is in contrast to the traditional pre-copy approach, which first copies the memory state over multiple iterations followed by a final transfer of the processor state. The post-copy strategy can provide a "win-win" by reducing total migration time closer to its equivalent time achieved by non-live VM migration. This is done while maintaining the liveness benefits of the pre-copy approach. We compare post-copy extensively against the traditional pre-copy approach on top of the Xen Hypervisor. Using a range of VM workloads we show improvements in several migration metrics including pages transferred, total migration time and network overhead. We facilitate the use of post-copy with adaptive pre-paging in order to eliminate all duplicate page transmissions. Our implementation is able to reduce the number of network-bound page faults to within 21% of the VM's working set for large workloads. Finally, we eliminate the transfer of free memory pages in both migration schemes through a dynamic self-ballooning (DSB) mechanism. DSB periodically releases free pages in a guest VM back to the hypervisor and significantly speeds up migration with negligible performance degradation.

REFERENCES

• 1: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 2: Robert Bradford , Evangelos Kotsovinos , Anja Feldmann , Harald Schiöberg, Live wide-area migration of virtual machines including local persistent state, Proceedings of the 3rd international conference on Virtual execution environments, June 13-15, 2007, San Diego, California, USA [doi>10.1145/1254810.1254834]
• 3: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 4: Brendan Cully , Geoffrey Lefebvre , Dutch Meyer , Mike Feeley , Norm Hutchinson , Andrew Warfield, Remus: high availability via asynchronous virtual machine replication, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, p.161-174, April 16-18, 2008, San Francisco, California
• 5: Peter J. Denning, The working set model for program behavior, Communications of the ACM, v.11 n.5, p.323-333, May 1968 [doi>10.1145/363095.363141]
• 6: Frederick Douglis, Transparent Process Migration in the Sprite Operating System, University of California at Berkeley, Berkeley, CA, 1990
• 7: Steven M. Hand, Self-paging in the Nemesis operating system, Proceedings of the third symposium on Operating systems design and implementation, p.73-86, February 1999, New Orleans, Louisiana, United States
• 8: HANSEN, J., AND HENRIKSEN, A. Nomadic operating systems. In Master's thesis, Dept. of Computer Science, University of Copenhagen, Denmark (2002).
• 9: Jacob Gorm Hansen , Eric Jul, Self-migration of operating systems, Proceedings of the 11th workshop on ACM SIGOPS European workshop, September 19-22, 2004, Leuven, Belgium [doi>10.1145/1133572.1133616]
• 10: Michael R. Hines , Kartik Gopalan, MemX: supporting large memory workloads in Xen virtual machines, Proceedings of the 3rd international workshop on Virtualization technology in distributed computing, p.1-8, November 12-12, 2007, Reno, Nevada [doi>10.1145/1408654.1408656]
• 11: HO, R. S., WANG, C.-L., AND LAU, F. C. Lightweight process migration and memory prefetching in OpenMosix. In Proc. of IPDPS (2008).
• 12: KERRIGHED. http://www.kerrighed.org.
• 13: KIVITY, A., KAMAY, Y., AND LAOR, D. kvm: the Linux Virtual Machine Monitor. 2007.
• 14: LAGAR-CAVILLA, H. A., WHITNEY, J., SCANNEL, A., RUMBLE, S., BRUDNO, M., DE LARA, E., AND SATYANARAYANAN, M. Impromptu clusters for near-interactive cloud-based services. Tech. rep., CSRG-578, University of Toronto, June 2008.
• 15: MAGENHEIMER, D. Add self-ballooning to balloon driver. Discussion on Xen Development mailing list and personal communication, April 2008.
• 16: Dejan S. Milojičić , Fred Douglis , Yves Paindaveine , Richard Wheeler , Songnian Zhou, Process migration, ACM Computing Surveys (CSUR), v.32 n.3, p.241-299, Sept. 2000 [doi>10.1145/367701.367728]
• 17: MOSIX. http://www.mosix.org.
• 18: Michael Nelson , Beng-Hong Lim , Greg Hutchins, Fast transparent migration for virtual machines, Proceedings of the annual conference on USENIX Annual Technical Conference, p.25-25, April 10-15, 2005, Anaheim, CA
• 19: NOACK, M. Comparative evaluation of process migration algorithms. Master's thesis, Dresden University of Technology -- Operating Systems Group, 2003.
• 20: OPENVZ. Container-based Virtualization for Linux, http://www.openvz.com/.
• 21: G. Oppenheimer , N. Weizer, Resource management for a medium scale time-sharing operating system, Communications of the ACM, v.11 n.5, p.313-322, May 1968 [doi>10.1145/363095.363140]
• 22: Steven Osman , Dinesh Subhraveti , Gong Su , Jason Nieh, The design and implementation of Zap: a system for migrating computing environments, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060323]
• 23: James S. Plank , Micah Beck , Gerry Kingsley , Kai Li, Libckpt: transparent checkpointing under Unix, Proceedings of the USENIX 1995 Technical Conference Proceedings on USENIX 1995 Technical Conference Proceedings, p.18-18, January 16-20, 1995, New Orleans, Louisiana
• 24: Michael Richmond , Michael Hitchens, A new process migration algorithm, ACM SIGOPS Operating Systems Review, v.31 n.1, p.31-42, Jan. 1997 [doi>10.1145/254784.254790]
• 25: E. T. Roush, Fast dynamic process migration, Proceedings of the 16th International Conference on Distributed Computing Systems (ICDCS '96), p.637, May 27-30, 1996
• 26: Constantine P. Sapuntzakis , Ramesh Chandra , Ben Pfaff , Jim Chow , Monica S. Lam , Mendel Rosenblum, Optimizing the migration of virtual computers, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060324]
• 27: Mahadev Satyanarayanan , Benjamin Gilbert , Matt Toups , Niraj Tolia , Ajay Surie , David R. O'Hallaron , Adam Wolbach , Jan Harkes , Adrian Perrig , David J. Farber , Michael A. Kozuch , Casey J. Helfrich , Partho Nath , H. Andres Lagar-Cavilla, Pervasive Personal Computing in an Internet Suspend/Resume System, IEEE Internet Computing, v.11 n.2, p.16-25, March 2007 [doi>10.1109/MIC.2007.46]
• 28: Brian Keith Schmidt , Monica S. Lam, Supporting ubiquitous computing with stateless consoles and computation caches, Stanford University, Stanford, CA, 2000
• 29: Georg Stellner, CoCheck: Checkpointing and Process Migration for MPI, Proceedings of the 10th International Parallel Processing Symposium, p.526-531, April 15-19, 1996
• 30: Douglas Thain , Todd Tannenbaum , Miron Livny, Distributed computing in practice: the Condor experience: Research Articles, Concurrency and Computation: Practice & Experience, v.17 n.2-4, p.323-356, February 2005 [doi>10.1002/cpe.v17:2/4]
• 31: TRIVEDI, K. An analysis of prepaging. Journal of Computing 22 (1979), 191--210.
• 32: K. S. Trivedi, On the Paging Performance of Array Algorithms, IEEE Transactions on Computers, v.26 n.10, p.938-947, October 1977 [doi>10.1109/TC.1977.1674729]
• 33: K. S. Trivedi, Prepaging and Applications to Array Algorithms, IEEE Transactions on Computers, v.25 n.9, p.915-921, September 1976 [doi>10.1109/TC.1976.1674716]
• 34: Carl A. Waldspurger, Memory resource management in VMware ESX server, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060307]
• 35: Andrew Whitaker , Richard S. Cox , Marianne Shaw , Steven D. Grible, Constructing services with interposable virtual hardware, Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation, p.13-13, March 29-31, 2004, San Francisco, California
• 36: Andrew Whitaker , Marianne Shaw , Steven D. Gribble, Scale and performance in the Denali isolation kernel, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060308]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.0 General

General Terms:
Experimentation, Performance

Keywords:
operating systems, post-copy, process migration, virtual machines, xen