Ultra-high performance communication with MPI and the Sun fire™ link interconnect

ABSTRACT

We present a new low-latency system area network that provides the ultra-high bandwidth needed to fuse a collection of large SMP servers into a capability cluster. The network adapter exports a remote shared memory (RSM) model that supports low latency kernel bypass messaging. The Sun™ MPI library uses the RSM interface to implement a highly efficient memory-to-memory messaging protocol in which the library directly manages buffers and data structures in remote memory. This allows flexible allocation of buffer space to active connections, while avoiding resource contention that could otherwise increase latencies. We discuss the characteristics of the interconnect, describe the MPI protocols, and measure the performance of a number of MPI benchmarks. Our results include MPI inter-node bandwidths of almost 3 Gigabytes per second and MPI ping-pong latencies as low as 3.7 microseconds.

REFERENCES

• 1: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 2: Scott Pakin , Mario Lauria , Andrew Chien, High performance messaging on workstations: Illinois fast messages (FM) for Myrinet, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.55-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224360]
• 3: Donald J. Becker, Thomas Sterling, Daniel Savarese, John E. Dorband, Udaya A. Ranawak, Charles V. Packer. BEOWULF: A Parallel Workstation for Scientific Computation. Proceedings of the International Conference on Parallel Processing '95, August 1995.
• 4: Thomas E. Anderson , David E. Culler , David A. Patterson , and the NOW team, A Case for NOW (Networks of Workstations), IEEE Micro, v.15 n.1, p.54-64, February 1995 [doi>10.1109/40.342018]
• 5: Steven S. Lumetta , Alan M. Mainwaring , David E. Culler, Multi-protocol active messages on a cluster of SMP's, Proceedings of the 1997 ACM/IEEE conference on Supercomputing (CDROM), p.1-22, November 15-21, 1997, San Jose, CA [doi>10.1145/509593.509596]
• 6: Steven S. Lumetta and David E. Culler. Managing Concurrent Access for Shared Memory Active Messages. IPPS/SPDP '98, Orlando.
• 7: Lisa Noordergraaf , Robert Zak, SMP system interconnect instrumentation for performance analysis, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-9, November 16, 2002, Baltimore, Maryland
• 8: Sun Microsystems. librsm(3LIB) man page. http://docs.sun.com.
• 9: IEEE Standard for Scalable Coherent Interface (SCI), IEEE Std 1596--1992.
• 10: Richard B. Gillett, Memory Channel Network for PCI, IEEE Micro, v.16 n.1, p.12-18, February 1996 [doi>10.1109/40.482307]
• 11: Fabrizio Petrini , Adolfy Hoisie , Wu-chun Feng , Richard Graham, Performance Evaluation of the Quadrics Interconnection Network, Proceedings of the 15th International Parallel & Distributed Processing Symposium, p.165, April 23-27, 2001
• 12: InfiniBand Trade Association. InfiniBand Architecture Specification Volume 1, Release 1.0.a, June 2001. http://www.infinibandta.org
• 13: Steve Sistare , Rolf vandeVaart , Eugene Loh, Optimization of MPI collectives on clusters of large-scale SMP's, Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM), p.23-es, November 14-19, 1999, Portland, Oregon, United States [doi>10.1145/331532.331555]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.1 Large and Medium ("Mainframe") Computers
          Subjects: Super (very large) computers

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Measurement techniques

  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Monitors
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Monitors

  G. Mathematics of Computing
  G.2 DISCRETE MATHEMATICS
      G.2.2 Graph Theory
          Subjects: Network problems

General Terms:
Experimentation, Measurement, Performance

Keywords:
MPI, SAN, interconnects, kernel bypass, performance evaluation, remote shared memory