Decoupled DIMM: building high-bandwidth memory system using low-speed DRAM devices

ABSTRACT

The widespread use of multicore processors has dramatically increased the demands on high bandwidth and large capacity from memory systems. In a conventional DDR2/DDR3 DRAM memory system, the memory bus and DRAM devices run at the same data rate. To improve memory bandwidth, we propose a new memory system design called decoupled DIMM that allows the memory bus to operate at a data rate much higher than that of the DRAM devices. In the design, a synchronization buffer is added to relay data between the slow DRAM devices and the fast memory bus; and memory access scheduling is revised to avoid access conflicts on memory ranks. The design not only improves memory bandwidth beyond what can be supported by current memory devices, but also improves reliability, power efficiency, and cost effectiveness by using relatively slow memory devices. The idea of decoupling, precisely the decoupling of bandwidth match between memory bus and a single rank of devices, can also be applied to other types of memory systems including FB-DIMM.

Our experimental results show that a decoupled DIMM system of 2667MT/s bus data rate and 1333MT/s device data rate improves the performance of memory-intensive workloads by 51% on average over a conventional memory system of 1333MT/s data rate. Alternatively, a decoupled DIMM system of 1600MT/s bus data rate and 800MT/s device data rate incurs only 8% performance loss when compared with a conventional system of 1600MT/s data rate, with 16% reduction on the memory power consumption and 9% saving on memory energy.

REFERENCES

• 1: Nathan L. Binkert , Ronald G. Dreslinski , Lisa R. Hsu , Kevin T. Lim , Ali G. Saidi , Steven K. Reinhardt, The M5 Simulator: Modeling Networked Systems, IEEE Micro, v.26 n.4, p.52-60, July 2006 [doi>10.1109/MM.2006.82]
• 2: Doug Burger , James R. Goodman , Alain Kägi, Memory bandwidth limitations of future microprocessors, Proceedings of the 23rd annual international symposium on Computer architecture, p.78-89, May 22-24, 1996, Philadelphia, Pennsylvania, United States
• 3: Vinodh Cuppu , Bruce Jacob, Concurrency, latency, or system overhead: which has the largest impact on uniprocessor DRAM-system performance?, Proceedings of the 28th annual international symposium on Computer architecture, p.62-71, June 30-July 04, 2001, Göteborg, Sweden
• 4: Vinodh Cuppu , Bruce Jacob , Brian Davis , Trevor Mudge, A performance comparison of contemporary DRAM architectures, Proceedings of the 26th annual international symposium on Computer architecture, p.222-233, May 01-04, 1999, Atlanta, Georgia, United States
• 5: V. Delaluz , M. Kandemir , N. Vijaykrishnan , A. Sivasubramaniam , M. J. Irwin, DRAM Energy Management Using Sof ware and Hardware Directed Power Mode Control, Proceedings of the 7th International Symposium on High-Performance Computer Architecture, p.159, January 20-24, 2001
• 6: Bruno Diniz , Dorgival Guedes , Wagner Meira, Jr. , Ricardo Bianchini, Limiting the power consumption of main memory, Proceedings of the 34th annual international symposium on Computer architecture, June 09-13, 2007, San Diego, California, USA
• 7: Brinda Ganesh , Aamer Jaleel , David Wang , Bruce Jacob, Fully-Buffered DIMM Memory Architectures: Understanding Mechanisms, Overheads and Scaling, Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, p.109-120, February 10-14, 2007 [doi>10.1109/HPCA.2007.346190]
• 8: Mrinmoy Ghosh , Hsien-Hsin S. Lee, Smart Refresh: An Enhanced Memory Controller Design for Reducing Energy in Conventional and 3D Die-Stacked DRAMs, Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, p.134-145, December 01-05, 2007 [doi>10.1109/MICRO.2007.38]
• 9: Ibrahim Hur , Calvin Lin, Adaptive History-Based Memory Schedulers, Proceedings of the 37th annual IEEE/ACM International Symposium on Microarchitecture, p.343-354, December 04-08, 2004, Portland, Oregon [doi>10.1109/MICRO.2004.4]
• 10: I. Hur and C. Lin. A comprehensive approach to DRAM power management. In Proceedings of the 13th International Symposium on High-Performance Computer Architecture, pages 305--316, 2008.
• 11: Alvin R. Lebeck , Xiaobo Fan , Heng Zeng , Carla Ellis, Power aware page allocation, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.105-116, November 2000, Cambridge, Massachusetts, United States
• 12: Wi-fen Lin, Reducing DRAM Latencies with an Integrated Memory Hierarchy Design, Proceedings of the 7th International Symposium on High-Performance Computer Architecture, p.301, January 20-24, 2001
• 13: K. Luo, J. Gummaraju, and M. Franklin. Balancing throughput and fairness in SMT processors. In IEEE International Symposium on Performance Analysis of Systems and Software, pages 164--171, 2001.
• 14: S. A. Mckee , W. A. Wulf, Access ordering and memory-conscious cache utilization, Proceedings of the 1st IEEE Symposium on High-Performance Computer Architecture, p.253, January 22-25, 1995
• 15: MemoryStore.com. Memory module price. http://www.memorystore.com/config/_generic.asp?cboLevel1=71.
• 16: MetaRAM, Inc. MetaRAM product brief. http://www.metaram.com/pdf/briefs/MetaRAM_DDR3_PB.pdf.
• 17: Micron Technology, Inc. DDR3 SDRAM system-power calculator. http://download.micron.com/downloads/misc/ddr3_power_calc.xls.
• 18: Micron Technology, Inc. MT41J128M8BY-187E. http://download.micron.com/pdf/datasheets/dram/ddr3/1Gb%20DDR3%20SDRAM.pdf.
• 19: Micron Technology, Inc. HTF18C64-128-256x72D. http://download.micron.com/pdf/datasheets/modules/ddr2/HTF18C64_128_256x72D.pdf, 2007.
• 20: Micron Technology, Inc. TN-41-01: Calculating memory system power for DDR3. http://download.micron.com/pdf/technotes/ddr3/TN41_01DDR3%20Power.pdf, 2007.
• 21: Onur Mutlu , Thomas Moscibroda, Stall-Time Fair Memory Access Scheduling for Chip Multiprocessors, Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture, p.146-160, December 01-05, 2007 [doi>10.1109/MICRO.2007.40]
• 22: Onur Mutlu , Thomas Moscibroda, Parallelism-Aware Batch Scheduling: Enhancing both Performance and Fairness of Shared DRAM Systems, Proceedings of the 35th International Symposium on Computer Architecture, p.63-74, June 21-25, 2008
• 23: Kyle J. Nesbit , Nidhi Aggarwal , James Laudon , James E. Smith, Fair Queuing Memory Systems, Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture, p.208-222, December 09-13, 2006 [doi>10.1109/MICRO.2006.24]
• 24: Scott Rixner , William J. Dally , Ujval J. Kapasi , Peter Mattson , John D. Owens, Memory access scheduling, Proceedings of the 27th annual international symposium on Computer architecture, p.128-138, June 2000, Vancouver, British Columbia, Canada
• 25: Jun Shao , Brian T. Davis, A Burst Scheduling Access Reordering Mechanism, Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, p.285-294, February 10-14, 2007 [doi>10.1109/HPCA.2007.346206]
• 26: Timothy Sherwood , Erez Perelman , Greg Hamerly , Brad Calder, Automatically characterizing large scale program behavior, Proceedings of the 10th international conference on Architectural support for programming languages and operating systems, October 05-09, 2002, San Jose, California
• 27: Allan Snavely , Dean M. Tullsen , Geoff Voelker, Symbiotic jobscheduling with priorities for a simultaneous multithreading processor, Proceedings of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 15-19, 2002, Marina Del Rey, California
• 28: J. Vera, F. J. Cazorla, A. Pajuelo, O. J. Santana, E. Fernandez, and M. Valero. A novel evaluation methodology to obtain fair measurements in multithreaded architectures. In Workshop on Modeling Benchmarking and Simulation, 2006.
• 29: P. Vogt and J. Haas. Fully-Buffered DIMM technology moves enterprise platforms to the next level. http://www.intel.com/technology/magazine/computing/fully-buffered-dimm-0305.htm, 2005.
• 30: David Tawei Wang , Bruce L. Jacob, Modern dram memory systems: performance analysis and scheduling algorithm, University of Maryland at College Park, College Park, MD, 2005
• 31: F. A. Ware and C. Hampel. Improving power and data efficiency with threaded memory modules. In Proceedings of the 24th International Conference on Computer Design, pages 417--424, 2006.
• 32: Zhao Zhang , Zhichun Zhu , Xiaodong Zhang, A permutation-based page interleaving scheme to reduce row-buffer conflicts and exploit data locality, Proceedings of the 33rd annual ACM/IEEE international symposium on Microarchitecture, p.32-41, December 2000, Monterey, California, United States [doi>10.1145/360128.360134]
• 33: Hongzhong Zheng , Jiang Lin , Zhao Zhang , Eugene Gorbatov , Howard David , Zhichun Zhu, Mini-rank: Adaptive DRAM architecture for improving memory power efficiency, Proceedings of the 2008 41st IEEE/ACM International Symposium on Microarchitecture, p.210-221, November 08-12, 2008 [doi>10.1109/MICRO.2008.4771792]
• 34: Zhichun Zhu , Zhao Zhang, A Performance Comparison of DRAM Memory System Optimizations for SMT Processors, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.213-224, February 12-16, 2005 [doi>10.1109/HPCA.2005.2]

INDEX TERMS

Primary Classification:
  B. Hardware
  B.3 MEMORY STRUCTURES
      B.3.2 Design Styles
          Subjects: Primary memory

General Terms:
Design, Performance

Keywords:
DRAM memories, bandwidth decoupling, decoupled DIMM