MemX: supporting large memory workloads in Xen virtual machines

ABSTRACT

Modern grid computing and enterprise applications increasingly execute on clusters that rely upon virtual machines (VMs) to partition hardware resources and improve utilization efficiency. These applications tend to have memory and I/O intensive workloads, such as large databases, data mining, scientific workloads, and web services, which can strain the limited I/O and memory resources within a single VM. In this paper, we present our experiences in developing a fully transparent distributed system, called MemX, within the Xen VM environment that coordinates the use of cluster-wide memory resources to support large memory and I/O intensive workloads. Applications using MemX do not require specialized APIs, libraries, recompilation, relinking, or dataset pre-partitioning. We compare and contrast the different design choices in MemX and present preliminary performance evaluation using several resource-intensive benchmarks in both virtualized and non-virtualized Linux. Our evaluations show that large dataset applications and multiple concurrent VMs achieve significant speedups using MemX compared against virtualized local and iSCSI disks. As an added benefit, we also show that live Xen VMs using MemX can migrate seamlessly without disrupting any running applications.

REFERENCES

• 1: Cristiana Amza , Alan L. Cox , Sandhya Dwarkadas , Pete Keleher , Honghui Lu , Ramakrishnan Rajamony , Weimin Yu , Willy Zwaenepoel, TreadMarks: Shared Memory Computing on Networks of Workstations, Computer, v.29 n.2, p.18-28, February 1996 [doi>10.1109/2.485843]
• 2: The vMatrix: Server Switching, Proceedings of the 10th IEEE International Workshop on Future Trends of Distributed Computing Systems, p.110-118, May 26-28, 2004
• 3: Amnon Barak , Oren La'adan, The MOSIX multicomputer operating system for high performance cluster computing, Future Generation Computer Systems, v.13 n.4-5, p.361-372, March, 1998 [doi>10.1016/S0167-739X(97)00037-X]
• 4: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 5: Matthew Chapman , Gernot Heiser, Implementing transparent shared memory on clusters using virtual machines, Proceedings of the annual conference on USENIX Annual Technical Conference, p.23-23, April 10-15, 2005, Anaheim, CA
• 6: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 7: D. Comer and J. Griffoen. A new design for distributed systems: the remote memory model. Proc. of the USENIX 1991 Summer Technical Conference, pages 127--135, 1991.
• 8: Francisco Matias Cuenca-Acuna , Thu D. Nguyen, Cooperative Caching Middleware for Cluster-Based Servers, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing, p.303, August 07-09, 2001
• 9: Sandhya Dwarkadas , Robert Stets , Nikolaos Hardavellas , Leonidas I. Kontothanassis , Rishiyur S. Nikhil, Cashmere-VLM: Remote Memory Paging for Software Distributed Shared Memory, Proceedings of the 13th International Symposium on Parallel Processing and the 10th Symposium on Parallel and Distributed Processing, p.153-159, April 12-16, 1999
• 10: M. J. Feeley , W. E. Morgan , E. P. Pighin , A. R. Karlin , H. M. Levy , C. A. Thekkath, Implementing global memory management in a workstation cluster, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.201-212, December 03-06, 1995, Copper Mountain, Colorado, United States
• 11: Michail D. Flouris , Evangelos P. Markatos, The Network RamDisk: Using remote memory on heterogeneous NOWs, Cluster Computing, v.2 n.4, p.281-293, 1999 [doi>10.1023/A:1019051330479]
• 12: M. Hines, J. Wang, and K. Gopalan. Distributed Anemone: Transparent Low-Latency Access to Remote Memory in Commodity Clusters. In Proc. of the International Conference on High Performance Computing (HiPC), Dec. 2006.
• 13: L. Ibarria, P. Lindstrom, J. Rossignac, and A. Szymczak. Out-of-core compression and decompression of large N-dimensional scalar fields. In Proc. of Eurographics 2003, pages 343--348, September 2003.
• 14: Kerrighed. http://www.kerrighed.org.
• 15: Peter Lindstrom, Out-of-core construction and visualization of multiresolution surfaces, Proceedings of the 2003 symposium on Interactive 3D graphics, April 27-30, 2003, Monterey, California [doi>10.1145/641480.641500]
• 16: Evangelos P. Markatos , George Dramitinos, Implementation of a reliable remote memory pager, Proceedings of the 1996 annual conference on USENIX Annual Technical Conference, p.15-15, January 22-26, 1996, San Diego, CA
• 17: S. McCanne and S. Floyd. ns2: Network simulator, http://www.isi.edu/nsnam/ns/.
• 18: I. McDonald. Remote paging in a single address space operating system supporting quality of service. Tech. Report, Computing Science, University of Glasgow, 1999.
• 19: Dejan S. Milojičić , Fred Douglis , Yves Paindaveine , Richard Wheeler , Songnian Zhou, Process migration, ACM Computing Surveys (CSUR), v.32 n.3, p.241-299, Sept. 2000 [doi>10.1145/367701.367728]
• 20: MySQL AB. The MySQL Benchmark Suite http://dev.mysql.com/doc/refman/5.0/en/mysql-benchmarks.html.
• 21: NAS Parallel Benchmarks. http://www.nas.nasa.gov/Resources/Software/npb.html.
• 22: POV-Ray. The persistence of vision raytracer, http://povray.org/.
• 23: The iSCSI Enterprise Target Project. http://iscsitarget.sourceforge.net/.
• 24: Theodore M. Wong , John Wilkes, My Cache or Yours? Making Storage More Exclusive, Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference, p.161-175, June 10-15, 2002