VM-based shared memory on low-latency, remote-memory-access networks

ABSTRACT

Recent technological advances have produced network interfaces that provide users with very low-latency access to the memory of remote machines. We examine the impact of such networks on the implementation and performance of software DSM. Specifically, we compare two DSM systems---Cashmere and TreadMarks---on a 32-processor DEC Alpha cluster connected by a Memory Channel network.Both Cashmere and TreadMarks use virtual memory to maintain coherence on pages, and both use lazy, multi-writer release consistency. The systems differ dramatically, however, in the mechanisms used to track sharing information and to collect and merge concurrent updates to a page, with the result that Cashmere communicates much more frequently, and at a much finer grain.Our principal conclusion is that low-latency networks make DSM based on fine-grain communication competitive with more coarse-grain approaches, but that further hardware improvements will be needed before such systems can provide consistently superior performance. In our experiments, Cashmere scales slightly better than TreadMarks for applications with false sharing. At the same time, it is severely constrained by limitations of the current Memory Channel hardware. In general, performance is better for TreadMarks.

REFERENCES

• 1: Cristiana Amza , Alan L. Cox , Sandhya Dwarkadas , Pete Keleher , Honghui Lu , Ramakrishnan Rajamony , Weimin Yu , Willy Zwaenepoel, TreadMarks: Shared Memory Computing on Networks of Workstations, Computer, v.29 n.2, p.18-28, February 1996 [doi>10.1109/2.485843]
• 2: M. A. Blumrich , K. Li , R. Alpert , C. Dubnicki , E. W. Felten , J. Sandberg, Virtual memory mapped network interface for the SHRIMP multicomputer, Proceedings of the 21ST annual international symposium on Computer architecture, p.142-153, April 18-21, 1994, Chicago, Illinois, United States
• 3: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 4: William J. Bolosky , Michael L. Scott , Robert P. Fitzgerald , Robert J. Fowler , Alan L. Cox, NUMA policies and their relation to memory architecture, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.212-221, April 08-11, 1991, Santa Clara, California, United States
• 5: Greg Buzzard , David Jacobson , Milon Mackey , Scott Marovich , John Wilkes, An implementation of the Hamlyn sender-managed interface architecture, Proceedings of the second USENIX symposium on Operating systems design and implementation, p.245-259, October 29-November 01, 1996, Seattle, Washington, United States
• 6: J. Chase , F. Amador , E. Lazowska , H. Levy , R. Littlefield, The Amber system: parallel programming on a network of multiprocessors, Proceedings of the twelfth ACM symposium on Operating systems principles, p.147-158, November 1989
• 7: A. Krishnamurthy , D. E. Culler , A. Dusseau , S. C. Goldstein , S. Lumetta , T. von Eicken , K. Yelick, Parallel programming in Split-C, Proceedings of the 1993 ACM/IEEE conference on Supercomputing, p.262-273, December 1993, Portland, Oregon, United States [doi>10.1145/169627.169724]
• 8: $. Dwarkadas, A. A. Sehaffer, R. W. Cottingham Jr., A, L, Cox, P. Keleher, and W. Zwaenepoel. Parallelization of General Linkage Analysis Problems. Human Heredity, 44:127-141,1994.
• 9: Sandhya Dwarkadas , Alan L. Cox , Willy Zwaenepoel, An integrated compile-time/run-time software distributed shared memory system, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.186-197, October 01-04, 1996, Cambridge, Massachusetts, United States
• 10: Andrew Erlichson , Neal Nuckolls , Greg Chesson , John Hennessy, SoftFLASH: analyzing the performance of clustered distributed virtual shared memory, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.210-220, October 01-04, 1996, Cambridge, Massachusetts, United States
• 11: M.J. Feeley, J. S. Chase, V. R. Narasayya, and H. M. Levy. Integrating Coherency and Recovery in Distributed Systems. In Proc. of the 1st Symp. on Operating Systems Design and Implementation, Nov. 1994.
• 12: Richard B. Gillett, Memory Channel Network for PCI, IEEE Micro, v.16 n.1, p.12-18, February 1996 [doi>10.1109/40.482307]
• 13: Davib B. Gustavson, The Scalable Coherent Interface and Related Standards Projects, IEEE Micro, v.12 n.1, p.10-22, January 1992 [doi>10.1109/40.124376]
• 14: Liviu Iftode , Cezary Dubnicki , Edward W. Felten , Kai Li, Improving Release-Consistent Shared Virtual Memory using Automatic Update, Proceedings of the 2nd IEEE Symposium on High-Performance Computer Architecture, p.14, February 03-07, 1996
• 15: K. L. Johnson , M. F. Kaashoek , D. A. Wallach, CRL: high-performance all-software distributed shared memory, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.213-226, December 03-06, 1995, Copper Mountain, Colorado, United States
• 16: Pete Keleher , Alan L. Cox , Willy Zwaenepoel, Lazy release consistency for software distributed shared memory, Proceedings of the 19th annual international symposium on Computer architecture, p.13-21, May 19-21, 1992, Queensland, Australia
• 17: Leonidas I. Kontothanassis , Michael L. Scott, High performance software coherence for current and future architectures, Journal of Parallel and Distributed Computing, v.29 n.2, p.179-195, Sept. 1995 [doi>10.1006/jpdc.1995.1116]
• 18: L. I. Kontothanassis , M. L. Scott, Using memory-mapped network interfaces to improve the performance of distributed shared memory, Proceedings of the 2nd IEEE Symposium on High-Performance Computer Architecture, p.166, February 03-07, 1996
• 19: Richard P. Larowe, Jr. , Carla Schlatter Ellis, Experimental comparison of memory management policies for NUMA multiprocessors, ACM Transactions on Computer Systems (TOCS), v.9 n.4, p.319-363, Nov. 1991 [doi>10.1145/118544.118546]
• 20: Daniel Lenoski , James Laudon , Kourosh Gharachorloo , Anoop Gupta , John Hennessy, The directory-based cache coherence protocol for the DASH multiprocessor, Proceedings of the 17th annual international symposium on Computer Architecture, p.148-159, May 28-31, 1990, Seattle, Washington, United States
• 21: Kai Li , Paul Hudak, Memory coherence in shared virtual memory systems, ACM Transactions on Computer Systems (TOCS), v.7 n.4, p.321-359, Nov. 1989 [doi>10.1145/75104.75105]
• 22: O. Lysne, S. Gjessing, and K. Loehsen. Running the SCI Protocol over HIC Networks. In 2nd Intl. Workshop on SCl-based Low-cos~igh-performance Computing (SCIzzL-2), Mar. 1995.
• 23: Michael Marchetti , Leonidas I. Kontothanassis , Ricardo Bianchini , Michael L. Scott, Using simple page placement policies to reduce the cost of cache fills in coherent shared-memory systems, Proceedings of the 9th International Symposium on Parallel Processing, p.480-485, April 25-28, 1995
• 24: Rishiyur S. Nikhil, Cid: A Parallel, "Shared-Memory" C for Distributed-Memory Machines, Proceedings of the 7th International Workshop on Languages and Compilers for Parallel Computing, p.376-390, August 08-10, 1994
• 25: Bill Nitzberg , Virginia Lo, Distributed Shared Memory: A Survey of Issues and Algorithms, Computer, v.24 n.8, p.52-60, August 1991 [doi>10.1109/2.84877]
• 26: K. Petersen and K. Li. Cache Coherence for Shared Memory Multiprocessors Based on Virtual Memory Support. In Proc. of the 7th Intl. Parallel Processing Symp., Apr. 1993.
• 27: Harjinder S. Sandhu , Benjamin Gamsa , Songnian Zhou, The shared regions approach to software cache coherence on multiprocessors, Proceedings of the fourth ACM SIGPLAN symposium on Principles and practice of parallel programming, p.229-238, May 19-22, 1993, San Diego, California, United States
• 28: Daniel J. Scales , Kourosh Gharachorloo , Chandramohan A. Thekkath, Shasta: a low overhead, software-only approach for supporting fine-grain shared memory, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.174-185, October 01-04, 1996, Cambridge, Massachusetts, United States
• 29: Ioannis Schoinas , Babak Falsafi , Alvin R. Lebeck , Steven K. Reinhardt , James R. Larus , David A. Wood, Fine-grain access control for distributed shared memory, Proceedings of the sixth international conference on Architectural support for programming languages and operating systems, p.297-306, October 05-07, 1994, San Jose, California, United States
• 30: Jaswinder Pal Singh , Wolf-Dietrich Weber , Anoop Gupta, SPLASH: Stanford parallel applications for shared-memory, ACM SIGARCH Computer Architecture News, v.20 n.1, p.5-44, March 1992 [doi>10.1145/130823.130824]
• 31: Andrew S. Tanenbaum , M. Frans Kaashoek , Henri E. Bal, Parallel Programming Using Shared Objects and Broadcasting, Computer, v.25 n.8, p.10-19, August 1992 [doi>10.1109/2.153276]
• 32: Steven Cameron Woo , Moriyoshi Ohara , Evan Torrie , Jaswinder Pal Singh , Anoop Gupta, The SPLASH-2 programs: characterization and methodological considerations, Proceedings of the 22nd annual international symposium on Computer architecture, p.24-36, June 22-24, 1995, S. Margherita Ligure, Italy
• 33: Donald Yeung , John Kubiatowicz , Anant Agarwal, MGS: a multigrain shared memory system, Proceedings of the 23rd annual international symposium on Computer architecture, p.44-55, May 22-24, 1996, Philadelphia, Pennsylvania, United States
• 34: M.J. Zekauskas,W. A. Sawdon, and B. N. Bershad. Software Write Detection for Distributed Shared Memory. In Proc. of the 1st Symp. on Operating Systems Design and Implementation, Nov. 1994.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS

Additional Classification:
  C. Computer Systems Organization
  C.0 GENERAL
      Subjects: Hardware/software interfaces

General Terms:
Design, Measurement, Performance, Theory