High performance RDMA-based MPI implementation over InfiniBand

ABSTRACT

Although InfiniBand Architecture is relatively new in the high performance computing area, it offers many features which help us to improve the performance of communication subsystems. One of these features is Remote Direct Memory Access (RDMA) operations. In this paper, we propose a new design of MPI over InfiniBand which brings the benefit of RDMA to not only large messages, but also small and control messages. We also achieve better scalability by exploiting application communication pattern and combining send/receive operations with RDMA operations. Our RDMA-based MPI implementation currently delivers a latency of 6.8 microseconds for small messages and a peak bandwidth of 871 Million Bytes (831 Mega Bytes) per second. Performance evaluation at the MPI level shows that for small messages, our RDMA-based design can reduce the latency by 24%, increase the bandwidth by over 104%, and reduce the host overhead by up to 22%. For large messages, we improve performance by reducing the time for transferring control messages. We have also shown that our new design is beneficial to MPI collective communication and NAS Parallel Benchmarks.

REFERENCES

• 1: Francisco José Alfaro , José J. Sánchez , José Duato , Chita R. Das, A Strategy to Compute the InfiniBand Arbitration Tables, Proceedings of the 16th International Parallel and Distributed Processing Symposium, p.46, April 15-19, 2002
• 2: Mohammad Banikazemi , Rama K. Govindaraju , Robert Blackmore , Dhabaleswar K. Panda, MPI-LAPI: An Efficient Implementation of MPI for IBM RS/6000 SP Systems, IEEE Transactions on Parallel and Distributed Systems, v.12 n.10, p.1081-1093, October 2001 [doi>10.1109/71.963419]
• 3: Srinath Rao, User-Level Communication in Cluster-Based Servers, Proceedings of the 8th International Symposium on High-Performance Computer Architecture, p.275, February 02-06, 2002
• 4: David Culler , Richard Karp , David Patterson , Abhijit Sahay , Klaus Erik Schauser , Eunice Santos , Ramesh Subramonian , Thorsten von Eicken, LogP: towards a realistic model of parallel computation, Proceedings of the fourth ACM SIGPLAN symposium on Principles and practice of parallel programming, p.1-12, May 19-22, 1993, San Diego, California, United States
• 5: R. Dimitrov and A. Skjellum. An Efficient MPI Implementation for Virtual Interface (VI) Architecture-Enabled Cluster Computing. http://www.mpi-softtech.com/publications/, 1998.
• 6: Dave Dunning , Greg Regnier , Gary McAlpine , Don Cameron , Bill Shubert , Frank Berry , Anne Marie Merritt , Ed Gronke , Chris Dodd, The Virtual Interface Architecture, IEEE Micro, v.18 n.2, p.66-76, March 1998 [doi>10.1109/40.671404]
• 7: William Gropp , Ewing Lusk , Nathan Doss , Anthony Skjellum, A high-performance, portable implementation of the MPI message passing interface standard, Parallel Computing, v.22 n.6, p.789-828, Sept. 1996 [doi>10.1016/0167-8191(96)00024-5]
• 8: Pin-down Cache: A Virtual Memory Management Technique for Zero-copy Communication, Proceedings of the 12th. International Parallel Processing Symposium on International Parallel Processing Symposium, p.308, March 30-April 03, 1998
• 9: Parry Husbands , James C. Hoe, MPI-StarT: delivering network performance to numerical applications, Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), p.1-15, November 07-13, 1998, San Jose, CA
• 10: InfiniBand Trade Association. InfiniBand Architecture Specification, Release 1.0, October 24, 2000.
• 11: Lawrence Livermore National Laboratory. MVICH: MPI for Virtual Interface Architecture, August, 2001.
• 12: J. Liu, J. Wu, S. P. Kinis, D. Buntins, W. Yu, B. Chandrasekaran, R. Noronha, P. Wyckoff, and D. K. Panda. MPI over InfiniBand: Early Experiences. Technical Report, OSU-CISRC-10/02-TR25, Computer and Information Science, the Ohio State University, January, 2003.
• 13: Kostas Magoutis , Salimah Addetia , Alexandra Fedorova , Margo I. Seltzer , Jeffrey S. Chase , Andrew J. Gallatin , Richard Kisley , Rajiv Wickremesinghe , Eran Gabber, Structure and Performance of the Direct Access File System, Proceedings of the General Track: 2002 USENIX Annual Technical Conference, p.1-14, June 10-15, 2002
• 14: Richard P. Martin , Amin M. Vahdat , David E. Culler , Thomas E. Anderson, Effects of communication latency, overhead, and bandwidth in a cluster architecture, Proceedings of the 24th annual international symposium on Computer architecture, p.85-97, June 01-04, 1997, Denver, Colorado, United States
• 15: Mellanox Technologies. Mellanox InfiniBand InfiniHost Adapters, July, 2002.
• 16: NASA. NAS Parallel Benchmarks.
• 17: Pallas. Pallas MPI Benchmarks. http://www.pallas.com/e/products/pmb/.
• 18: Rinku Gupta , Pavan Balaji , Dhabaleswar K. Panda , Jarek Nieplocha, Efficient Collective Operations Using Remote Memory Operations on VIA-Based Clusters, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, p.46.2, April 22-26, 2003
• 19: Steven J. Sistare , Christopher J. Jackson, Ultra-high performance communication with MPI and the Sun fire™ link interconnect, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-15, November 16, 2002, Baltimore, Maryland
• 20: Jeffrey S. Vetter , Frank Mueller, Communication Characteristics of Large-Scale Scientific Applications for Contemporary Cluster Architectures, Proceedings of the 16th International Parallel and Distributed Processing Symposium, p.96, April 15-19, 2002
• 21: Jiesheng Wu , Jiuxing Liu , Pete Wyckoff , Dhabaleswar Panda, Impact of On-Demand Connection Management in MPI over VIA, Proceedings of the IEEE International Conference on Cluster Computing, p.152, September 23-26, 2002
• 22: Yuanyuan Zhou , Angelos Bilas , Suresh Jagannathan , Cezary Dubnicki , James F. Philbin , Kai Li, Experiences with VI communication for database storage, Proceedings of the 29th annual international symposium on Computer architecture, p.257, May 25-29, 2002, Anchorage, Alaska

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.4 Parallel Architectures

General Terms:
Design, Performance

Keywords:
InfiniBand, MPI, cluster computing, high performance computing