Memory resource management in VMware ESX server

ABSTRACT

VMware ESX Server is a thin software layer designed to multiplex hardware resources efficiently among virtual machines running unmodified commodity operating systems. This paper introduces several novel ESX Server mechanisms and policies for managing memory. A ballooning technique reclaims the pages considered least valuable by the operating system running in a virtual machine. An idle memory tax achieves efficient memory utilization while maintaining performance isolation guarantees. Content-based page sharing and hot I/O page remapping exploit transparent page remapping to eliminate redundancy and reduce copying overheads. These techniques are combined to efficiently support virtual machine workloads that overcommit memory.

REFERENCES

• 1: Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau, Information and control in gray-box systems, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 2: Özalp Babaoglu , William Joy, Converting a swap-based system to do paging in an architecture lacking page-referenced bits, Proceedings of the eighth ACM symposium on Operating systems principles, p.78-86, December 14-16, 1981, Pacific Grove, California, United States
• 3: Edouard Bugnion , Scott Devine , Kinshuk Govil , Mendel Rosenblum, Disco: running commodity operating systems on scalable multiprocessors, ACM Transactions on Computer Systems (TOCS), v.15 n.4, p.412-447, Nov. 1997 [doi>10.1145/265924.265930]
• 4: Pei Cao, Edward W. Felten, and Kai Li. "Implementation and Performance of Application-Controlled File Caching," Proc. Symposium on Operating System Design and Implementation, November 1994.
• 5: Jeffrey S. Chase , Darrell C. Anderson , Prachi N. Thakar , Amin M. Vahdat , Ronald P. Doyle, Managing energy and server resources in hosting centers, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 6: R. J. Creasy. "The Origin of the VM/370 Time-Sharing System," IBM Journal of Research and Development, 25(5), September 1981.
• 7: Bryan Ford , Mike Hibler , Jay Lepreau , Patrick Tullmann , Godmar Back , Stephen Clawson, Microkernels meet recursive virtual machines, Proceedings of the second USENIX symposium on Operating systems design and implementation, p.137-151, October 29-November 01, 1996, Seattle, Washington, United States
• 8: Robert P. Goldberg. "Survey of Virtual Machine Research," IEEE Computer, 7(6), June 1974.
• 9: Kinshuk Govil , Dan Teodosiu , Yongqiang Huang , Mendel Rosenblum, Cellular Disco: resource management using virtual clusters on shared-memory multiprocessors, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.154-169, December 12-15, 1999, Charleston, South Carolina, United States
• 10: Peter H. Gum. "System/370 Extended Architecture: Facilities for Virtual Machines," IBM Journal of Research and Development', 27(6), November 1983.
• 11: Steven M. Hand, Self-paging in the Nemesis operating system, Proceedings of the third symposium on Operating systems design and implementation, p.73-86, February 1999, New Orleans, Louisiana, United States
• 12: Kieran Harty , David R. Cheriton, Application-controlled physical memory using external page-cache management, Proceedings of the fifth international conference on Architectural support for programming languages and operating systems, p.187-197, October 12-15, 1992, Boston, Massachusetts, United States
• 13: Intel Corporation. IA-32 Intel Architecture Software Developer's Manual. Volumes I, II, and III, 2001.
• 14: Bob Jenkins. "Algorithm Alley," Dr. Dobbs Journal, September 1997. Source code available from http://burtleburtle.net/bob/hash/
• 15: Ted Kaehler. "Virtual Memory for an Object-Oriented Language," Byte, August 1981.
• 16: Minkyong Kim , Brian Noble, Mobile network estimation, Proceedings of the 7th annual international conference on Mobile computing and networking, p.298-309, July 2001, Rome, Italy [doi>10.1145/381677.381705]
• 17: Jochen Liedtke , Hermann Haertig , Michael Hohmuth, OS-Controlled Cache Predictability for Real-Time Systems, Proceedings of the 3rd IEEE Real-Time Technology and Applications Symposium (RTAS '97), p.213, June 09-11, 1997
• 18: Marshall Kirk McKusick , Keith Bostic , Michael J. Karels , John S. Quarterman, The design and implementation of the 4.4BSD operating system, Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, 1996
• 19: Theodore H. Romer, Dennis Lee, Brian N. Bershad and J. Bradley Chen. "Dynamic Page Mapping Policies for Cache Conflict Resolution on Standard Hardware," Proc. Symposium on Operating System Design and Implementation, November 1994.
• 20: L. H. Seawright and R. A. McKinnon. "VM/370: A Study of Multiplicity and Usefulness," IBM Systems Journal, 18(1), 1979.
• 21: Timothy Sherwood , Brad Calder , Joel Emer, Reducing cache misses using hardware and software page placement, Proceedings of the 13th international conference on Supercomputing, p.155-164, June 20-25, 1999, Rhodes, Greece [doi>10.1145/305138.305189]
• 22: David A. Solomon , Mark Russinovich, Inside Microsoft Windows 2000, Microsoft Press, Redmond, WA, 2000
• 23: Jeremy Sugerman , Ganesh Venkitachalam , Beng-Hong Lim, Virtualizing I/O Devices on VMware Workstation's Hosted Virtual Machine Monitor, Proceedings of the General Track: 2002 USENIX Annual Technical Conference, p.1-14, June 25-30, 2001
• 24: David G. Sullivan , Robert Haas , Margo I. Seltzer, Tickets and Currencies Revisited: Extending Multi-Resource Lottery Scheduling, Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, p.148, March 28-30, 1999
• 25: David G. Sullivan and Margo I. Seltzer. "Isolation with Flexibility: A Resource Management Framework for Central Servers," Proc. Usenix Annual Technical Conference, June 2000.
• 26: Andrew S. Tanenbaum, Modern operating systems, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 27: R. Tremaine, P. Franaszek, J. Robinson, C. Schulz, T. Smith, M. Wazlowski, and P. Bland. "IBM Memory Expansion Technology (MXT)," IBM Journal of Research and Development', 45(2), March 2001.
• 28: Andrew Tridgell. "dbench" benchmark. Available from ftp://samba.org/pub/tridge/dbench/, September 2001.
• 29: VMware, Inc. VMware ESX Server User's Manual Version 1.5, Palo Alto, CA, April 2002.
• 30: Carl A. Waldspurger and William E. Weihl. "Lottery Scheduling: Flexible Proportional-Share Resource Management," Proc. Symposium on Operating System Design and Implementation, November 1994.
• 31: Carl A. Waldspurger, Lottery and stride scheduling: flexibile proportional-share resource management, Massachusetts Institute of Technology, Cambridge, MA, 1996
• 32: C. A. Waldspurger , W. E. Weihl, An object-oriented framework for modular resource management, Proceedings of the 5th International Workshop on Object Orientation in Operating Systems (IWOOOS '96), p.138, October 27-28, 1996

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.2 Storage Management
          Subjects: Distributed memories

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.7 Organization and Design
          Subjects: Distributed systems
      D.4.9 Systems Programs and Utilities
          Subjects: Window managers

General Terms:
Design, Management