Live migration of virtual machine based on full system trace and replay

ABSTRACT

Live migration of virtual machines (VM) across distinct physical hosts provides a significant new benefit for administrators of data centers and clusters. Previous migration schemes focused on transferring the runtime memory state of the VM. Those approaches employed memory pre-copy algorithm to synchronize the migrating VM states, which make VM live migration cost much network traffic and application downtime, especially for memory intensive workloads. This paper describes the design and implementation of a novel approach CR/TR-Motion that adopts checkpointing/recovery and trace/replay technology to provide fast, transparent VM migration. With execution trace logged on the source host, a synchronization algorithm is performed to orchestrate the running source and target VM until they get a consistent state. We also give a formalized characterization about the migration evaluation metrics and make a mathematical analysis about our algorithm. Our scheme can greatly reduce the migration downtime and network bandwidth consumption. Experimental measurements show that our approach can drastically reduce migration overheads compared with pre-copy algorithm: up to 72.4% on application observed downtime, up to 31.5% on total migration time and up to 95.9% on the data to synchronize the VM state, while the application performance overhead due to migration is less than 8.54% on average.

REFERENCES

• 1: http://www.eecs.umich.edu/virtual/software.html
• 2: http://www.tux.org/pub/tux/benchmarks/system/unixbench/
• 3: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 4: G. Bronevetsky, R. Fernandes, D. Marques, K. Pingali, and P. Stodghill. Recent Advances in Checkpoint/Recovery Systems. In Proceedings of 20th International Parallel and Distributed Processing Symposium (IPDPS'06), April 25-29, 2006
• 5: Kerstin Buchacker , Volkmar Sieh, Framework for Testing the Fault-Tolerance of Systems Including OS and Network Aspects, The 6th IEEE International Symposium on High-Assurance Systems Engineering: Special Topic: Impact of Networking, p.95-105, October 24-26, 2001
• 6: Robert Bradford , Evangelos Kotsovinos , Anja Feldmann , Harald Schiöberg, Live wide-area migration of virtual machines including local persistent state, Proceedings of the 3rd international conference on Virtual execution environments, June 13-15, 2007, San Diego, California, USA [doi>10.1145/1254810.1254834]
• 7: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 8: Brendan Cully , Geoffrey Lefebvre , Dutch Meyer , Mike Feeley , Norm Hutchinson , Andrew Warfield, Remus: high availability via asynchronous virtual machine replication, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, p.161-174, April 16-18, 2008, San Francisco, California
• 9: George W. Dunlap , Samuel T. King , Sukru Cinar , Murtaza A. Basrai , Peter M. Chen, ReVirt: enabling intrusion analysis through virtual-machine logging and replay, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060309]
• 10: Wei Huang , Qi Gao , Jiuxing Liu , Dhabaleswar K. Panda, High performance virtual machine migration with RDMA over modern interconnects, Proceedings of the 2007 IEEE International Conference on Cluster Computing, p.11-20, September 17-20, 2007 [doi>10.1109/CLUSTR.2007.4629212]
• 11: Samuel T. King , George W. Dunlap , Peter M. Chen, Debugging operating systems with time-traveling virtual machines, Proceedings of the annual conference on USENIX Annual Technical Conference, p.1-1, April 10-15, 2005, Anaheim, CA
• 12: S. T. King. Operating System Extensions to Support Host-Based Virtual Machines. Technical Report CSE-TR-465-02, University of Michigan, September 2002
• 13: Michael Kozuch , M. Satyanarayanan, Internet Suspend/Resume, Proceedings of the Fourth IEEE Workshop on Mobile Computing Systems and Applications, p.40, June 20-21, 2002
• 14: Sanjay Kumar , Karsten Schwan, Netchannel: a VMM-level mechanism for continuous, transparentdevice access during VM migration, Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, March 05-07, 2008, Seattle, WA, USA [doi>10.1145/1346256.1346261]
• 15: John R. Lange , Peter A. Dinda, Transparent network services via a virtual traffic layer for virtual machines, Proceedings of the 16th international symposium on High performance distributed computing, June 25-29, 2007, Monterey, California, USA [doi>10.1145/1272366.1272370]
• 16: Arun Babu Nagarajan , Frank Mueller , Christian Engelmann , Stephen L. Scott, Proactive fault tolerance for HPC with Xen virtualization, Proceedings of the 21st annual international conference on Supercomputing, June 17-21, 2007, Seattle, Washington [doi>10.1145/1274971.1274978]
• 17: Michael Nelson , Beng-Hong Lim , Greg Hutchins, Fast transparent migration for virtual machines, Proceedings of the annual conference on USENIX Annual Technical Conference, p.25-25, April 10-15, 2005, Anaheim, CA
• 18: Ripal Nathuji , Karsten Schwan, VirtualPower: coordinated power management in virtualized enterprise systems, Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, October 14-17, 2007, Stevenson, Washington, USA
• 19: Daniela A. S. de Oliveira , Jedidiah R. Crandall , Gary Wassermann , S. Felix Wu , Zhendong Su , Frederic T. Chong, ExecRecorder: VM-based full-system replay for attack analysis and system recovery, Proceedings of the 1st workshop on Architectural and system support for improving software dependability, p.66-71, October 21-21, 2006, San Jose, California [doi>10.1145/1181309.1181320]
• 20: C. Perkins, IP Encapsulation within IP, RFC Editor, 1996
• 21: Constantine P. Sapuntzakis , Ramesh Chandra , Ben Pfaff , Jim Chow , Monica S. Lam , Mendel Rosenblum, Optimizing the migration of virtual computers, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060324]
• 22: Ajay Surie , H. Andrés Lagar-Cavilla , Eyal de Lara , M. Satyanarayanan, Low-bandwidth VM migration via opportunistic replay, Proceedings of the 9th workshop on Mobile computing systems and applications, February 25-26, 2008, Napa Valley, California [doi>10.1145/1411759.1411779]
• 23: Franco Travostino , Paul Daspit , Leon Gommans , Chetan Jog , Cees de Laat , Joe Mambretti , Inder Monga , Bas van Oudenaarde , Satish Raghunath , Phil Yonghui Wang, Seamless live migration of virtual machines over the MAN/WAN, Future Generation Computer Systems, v.22 n.8, p.901-907, October 2006 [doi>10.1016/j.future.2006.03.007]
• 24: Andrew Whitaker , Richard S. Cox , Marianne Shaw , Steven D. Grible, Constructing services with interposable virtual hardware, Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation, p.13-13, March 29-31, 2004, San Francisco, California
• 25: B. Wellington. Secure DNS Dynamic Update, RFC 3007
• 26: M. Xu, V. Malyugin, J. Sheldon, G. Venkitachalam, and B. Weissman. ReTrace: Collecting Execution Trace with Virtual Machine Deterministic Replay. In Proceedings of the Third Annual Workshop on Modeling, Benchmarking and Simulation, June 10, 2007, California, USA
• 27: Ming Zhao , Renato J. Figueiredo, Experimental study of virtual machine migration in support of reservation of cluster resources, Proceedings of the 3rd international workshop on Virtualization technology in distributed computing, p.1-8, November 12-12, 2007, Reno, Nevada [doi>10.1145/1408654.1408659]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.7 Organization and Design
          Subjects: Distributed systems

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies

General Terms:
Algorithms, Design, Management, Measurement, Performance

Keywords:
checkpoint, live migration, replay, trace, virtual machine