Disaggregated memory for expansion and sharing in blade servers

ABSTRACT

Analysis of technology and application trends reveals a growing imbalance in the peak compute-to-memory-capacity ratio for future servers. At the same time, the fraction contributed by memory systems to total datacenter costs and power consumption during typical usage is increasing. In response to these trends, this paper re-examines traditional compute-memory co-location on a single system and details the design of a new general-purpose architectural building block-a memory blade-that allows memory to be "disaggregated" across a system ensemble. This remote memory blade can be used for memory capacity expansion to improve performance and for sharing memory across servers to reduce provisioning and power costs. We use this memory blade building block to propose two new system architecture solutions-(1) page-swapped remote memory at the virtualization layer, and (2) block-access remote memory with support in the coherence hardware-that enable transparent memory expansion and sharing on commodity-based systems. Using simulations of a mix of enterprise benchmarks supplemented with traces from live datacenters, we demonstrate that memory disaggregation can provide substantial performance benefits (on average 10X) in memory constrained environments, while the sharing enabled by our solutions can improve performance-per-dollar by up to 57% when optimizing memory provisioning across multiple servers.

REFERENCES

• 1: K. Asanovic et al. The Landscape of Parallel Computing Research: A View from Berkeley. UC Berkeley EECS Tech Report UCB/EECS-2006-183, Dec. 2006.
• 2: VMWare Performance Team Blogs. Ten Reasons Why Oracle Databases Run Best on VMWare "Scale up with Large Memory." http://tinyurl.com/cudjuy
• 3: J. Larus. Spending Moore's Dividend. Microsoft Tech Report MSR-TR-2008-69, May 2008
• 4: SIA. International Technology Roadmap for Semiconductors 2007 Edition, 2007.
• 5: HP. Memory technology evolution: an overview of system memory technologies. http://tinyurl.com/ctfjs2
• 6: Alvin R. Lebeck , Xiaobo Fan , Heng Zeng , Carla Ellis, Power aware page allocation, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.105-116, November 2000, Cambridge, Massachusetts, United States
• 7: V. Pandey, W. Jiang, Y. Zhou and R. Bianchini. DMA-Aware Memory Energy Conservation. In Proc. of the 12th Int. Sym. on High-Performance Computer Architecture (HPCA-12), 2006
• 8: Kevin Lim , Parthasarathy Ranganathan , Jichuan Chang , Chandrakant Patel , Trevor Mudge , Steven Reinhardt, Understanding and Designing New Server Architectures for Emerging Warehouse-Computing Environments, Proceedings of the 35th International Symposium on Computer Architecture, p.315-326, June 21-25, 2008
• 9: Parthasarathy Ranganathan , Norman Jouppi, Enterprise IT Trends and Implications for Architecture Research, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.253-256, February 12-16, 2005 [doi>10.1109/HPCA.2005.14]
• 10: http://apotheca.hpl.hp.com/pub/datasets/animation-bear/
• 11: Luiz André Barroso , Jeffrey Dean , Urs Hölzle, Web Search for a Planet: The Google Cluster Architecture, IEEE Micro, v.23 n.2, p.22-28, March 2003 [doi>10.1109/MM.2003.1196112]
• 12: E. Felten and J. Zahorjan. Issues in the implementation of a remote memory paging system. University of Washington CSE TR 91-03-09, March 1991.
• 13: M. J. Feeley , W. E. Morgan , E. P. Pighin , A. R. Karlin , H. M. Levy , C. A. Thekkath, Implementing global memory management in a workstation cluster, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.201-212, December 03-06, 1995, Copper Mountain, Colorado, United States
• 14: Michail D. Flouris , Evangelos P. Markatos, The Network RamDisk: Using remote memory on heterogeneous NOWs, Cluster Computing, v.2 n.4, p.281-293, 1999 [doi>10.1023/A:1019051330479]
• 15: Michael D. Dahlin , Randolph Y. Wang , Thomas E. Anderson , David A. Patterson, Cooperative caching: using remote client memory to improve file system performance, Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation, p.19-es, November 14-17, 1994, Monterey, California
• 16: M. Hines, L. Lewandowski and K. Gopalan. Anemone: Adaptive Network Memory Engine. Florida State University TR-050128, 2005.
• 17: L. Iftode. K. Li and K. Peterson. Memory servers for multicomputers. IEEE Spring COMPCON '93, 1993.
• 18: Samir Koussih , Anurag Acharya , Sanjeev Setia, Dodo: A User-Level System for Exploiting Idle Memory in Workstation Clusters, Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing, p.46, August 03-06, 1999
• 19: Anant Agarwal , Ricardo Bianchini , David Chaiken , Kirk L. Johnson , David Kranz , John Kubiatowicz , Beng-Hong Lim , Kenneth Mackenzie , Donald Yeung, The MIT Alewife machine: architecture and performance, Proceedings of the 22nd annual international symposium on Computer architecture, p.2-13, June 22-24, 1995, S. Margherita Ligure, Italy
• 20: Daniel Lenoski , James Laudon , Kourosh Gharachorloo , Wolf-Dietrich Weber , Anoop Gupta , John Hennessy , Mark Horowitz , Monica S. Lam, The Stanford Dash Multiprocessor, Computer, v.25 n.3, p.63-79, March 1992 [doi>10.1109/2.121510]
• 21: Erik Hagersten , Michael Koster, WildFire: A Scalable Path for SMPs, Proceedings of the 5th International Symposium on High Performance Computer Architecture, p.172, January 09-12, 1999
• 22: James Laudon , Daniel Lenoski, The SGI Origin: a ccNUMA highly scalable server, Proceedings of the 24th annual international symposium on Computer architecture, p.241-251, June 01-04, 1997, Denver, Colorado, United States
• 23: William J. Bolosky , Michael L. Scott , Robert P. Fitzgerald , Robert J. Fowler , Alan L. Cox, NUMA policies and their relation to memory architecture, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.212-221, April 08-11, 1991, Santa Clara, California, United States
• 24: Kai Li , Paul Hudak, Memory coherence in shared virtual memory systems, ACM Transactions on Computer Systems (TOCS), v.7 n.4, p.321-359, Nov. 1989 [doi>10.1145/75104.75105]
• 25: Daniel J. Scales , Kourosh Gharachorloo , Chandramohan A. Thekkath, Shasta: a low overhead, software-only approach for supporting fine-grain shared memory, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.174-185, October 01-04, 1996, Cambridge, Massachusetts, United States
• 26: Cristiana Amza , Alan L. Cox , Sandhya Dwarkadas , Pete Keleher , Honghui Lu , Ramakrishnan Rajamony , Weimin Yu , Willy Zwaenepoel, TreadMarks: Shared Memory Computing on Networks of Workstations, Computer, v.29 n.2, p.18-28, February 1996 [doi>10.1109/2.485843]
• 27: Ioannis Schoinas , Babak Falsafi , Alvin R. Lebeck , Steven K. Reinhardt , James R. Larus , David A. Wood, Fine-grain access control for distributed shared memory, Proceedings of the sixth international conference on Architectural support for programming languages and operating systems, p.297-306, October 05-07, 1994, San Jose, California, United States
• 28: K Gharachorloo. The Plight of Software Distributed Shared Memory. Invited talk at 1st Workshop on Software Distributed Shared Memory (WSDSM '99), 1999.
• 29: ScaleMP. The Versatile SMP" (vSMP) Architecture and Solutions Based on vSMP Foundation". White paper at http://www.scalemp.com/prod/technology/how-does-it-work/
• 30: F. Douglis. The compression cache: using online compression to extend physical memory. In Proc. of 1993 Winter USENIX Conference, 1993.
• 31: Magnus Ekman , Per Stenstrom, A Robust Main-Memory Compression Scheme, Proceedings of the 32nd annual international symposium on Computer Architecture, p.74-85, June 04-08, 2005
• 32: Virident. Virident's GreenGateway" technology and Spansion® EcoRAM. http://www.virident.com/solutions.php
• 33: Texas Memory Systems. TMS RamSan-440 Details. http://www.superssd.com/products/ramsan-440/
• 34: Intel. Intel Fully Buffered DIMM Specification Addendum. http://www.intel.com/technology/memory/FBDIMM/spec/Intel_FBD_Spec_Addendum_rev_p9.pdf
• 35: Taeho Kgil , Shaun D'Souza , Ali Saidi , Nathan Binkert , Ronald Dreslinski , Trevor Mudge , Steven Reinhardt , Krisztian Flautner, PicoServer: using 3D stacking technology to enable a compact energy efficient chip multiprocessor, Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, October 21-25, 2006, San Jose, California, USA
• 36: Magnus Ekman , Per Stenstrom, A Cost-Effective Main Memory Organization for Future Servers, Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Papers, p.45.1, April 04-08, 2005 [doi>10.1109/IPDPS.2005.12]
• 37: Carl A. Waldspurger, Memory resource management in VMware ESX server, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060307]
• 38: D. Ye, A . Pavuluri, C. Waldspurger, B. Tsang, B. Rychlik and S. Woo. Prototyping a Hybrid Main Memory Using a Virtual Machine Monitor. In Proc. of the 26th Int. Conf. on Computer Design (ICCD), 2008.
• 39: Eduardo Argollo , Ayose Falcón , Paolo Faraboschi , Matteo Monchiero , Daniel Ortega, COTSon: infrastructure for full system simulation, ACM SIGOPS Operating Systems Review, v.43 n.1, January 2009 [doi>10.1145/1496909.1496921]
• 40: Jose Renato Santos , Yoshio Turner , G. Janakiraman , Ian Pratt, Bridging the gap between software and hardware techniques for I/O virtualization, USENIX 2008 Annual Technical Conference on Annual Technical Conference, p.29-42, June 22-27, 2008, Boston, Massachusetts
• 41: J. Rolia, A. Andrzejak and M. Arlitt. Automating Enterprise Application Placement in Resource Utilities. 14th IFIP/IEEE Int. Workshop on Distributed Systems: Operations and Management, DSOM 2003.
• 42: R. Bryant and J. Hawkes. Linux® Scalability for Large NUMA Systems. In Proc. of Ottowa Linux Symposium 2003, July 2003.
• 43: Taeho Kgil , David Roberts , Trevor Mudge, Improving NAND Flash Based Disk Caches, Proceedings of the 35th International Symposium on Computer Architecture, p.327-338, June 21-25, 2008

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.0 GENERAL
      Subjects: System architectures

Additional Classification:
  B. Hardware
  B.3 MEMORY STRUCTURES
      B.3.2 Design Styles
          Subjects: Virtual memory; Primary memory

General Terms:
Design, Management, Performance

Keywords:
disaggregated memory, memory blades, memory capacity expansion, power and cost efficiencies