Design and implementation of FMPL, a fast message-passing library for remote memory operations

ABSTRACT

A fast message-passing library FMPL has been designed and developed to maximize communication performance by utilizing general architectural communication support such as remote memory operations, as well as to maximize total performance by eliminating dynamic communication overhead and overlapping communication and computation. FMPL provides a low-cost general-purpose point-to-point communication and collective communication such as broadcast, barrier synchronization and reduction. On a Hitachi SR8000, FMPL achieves an 8-byte latency of 12.8μsec., while MPI achieves 20μsec. FMPL is designed for building more highly functional message-passing libraries like BLACS as well as applications that need maximum performance.

REFERENCES

• 1: Arvind , Rishiyur S. Nikhil , Keshav K. Pingali, I-structures: data structures for parallel computing, ACM Transactions on Programming Languages and Systems (TOPLAS), v.11 n.4, p.598-632, Oct. 1989 [doi>10.1145/69558.69562]
• 2: Jack J. Dongarra , L. S. Blackford , J. Choi , A. Cleary , E. D'Azeuedo , J. Demmel , I. Dhillon , S. Hammarling , G. Henry , A. Petitet , K. Stanley , D. Walker , R. C. Whaley, ScaLAPACK user's guide, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1997
• 3: Taisuke Boku , Ken'ichi Itakura , Hiroshi Nakamura , Kisaburo Nakazawa, CP-PACS: a massively parallel processor for large scale scientific calculations, Proceedings of the 11th international conference on Supercomputing, p.108-115, July 07-11, 1997, Vienna, Austria [doi>10.1145/263580.263608]
• 4: Jack J. Dongarra , R. C Whaley, LAPACK Working Note 94: A User''s Guide to the BLACS v1.0, University of Tennessee, Knoxville, TN, 1995
• 5: William Gropp , Ewing Lusk , Nathan Doss , Anthony Skjellum, A high-performance, portable implementation of the MPI message passing interface standard, Parallel Computing, v.22 n.6, p.789-828, Sept. 1996 [doi>10.1016/0167-8191(96)00024-5]
• 6: Yuetsu Kodama , Hirohumi Sakane , Mitsuhisa Sato , Hayato Yamana , Shuichi Sakai , Yoshinori Yamaguchi, The EM-X parallel computer: architecture and basic performance, Proceedings of the 22nd annual international symposium on Computer architecture, p.14-23, June 22-24, 1995, S. Margherita Ligure, Italy
• 7: Mario Lauria , Andrew Chien, MPI-FM: high performance MPI on workstation clusters, Journal of Parallel and Distributed Computing, v.40 n.1, p.4-18, Jan. 10, 1997 [doi>10.1006/jpdc.1996.1264]
• 8: Takashi Matsumoto , Kei Hiraki, MBCF: a protected and virtualized high-speed user-level memory-based communication facility, Proceedings of the 12th international conference on Supercomputing, p.259-266, July 1998, Melbourne, Australia [doi>10.1145/277830.277889]
• 9: Kenji Morimoto , Takashi Matsumoto , Kei Hiraki, Implementing MPI with the Memory-Based Communication Facilities on the SSS-CORE Operating System, Proceedings of the 5th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, p.223-230, September 07-09, 1998
• 10: Hiroshi Nakamura , Taisuke Boku , Hideo Wada , Hiromitsu Imori , Ikuo Nakata , Yasuhiro Inagami , Kisaburo Nakazawa , Yoshiyuki Yamashita, A scalar architecture for pseudo vector processing based on slide-windowed registers, Proceedings of the 7th international conference on Supercomputing, p.298-307, July 19-23, 1993, Tokyo, Japan [doi>10.1145/165939.165998]
• 11: netlib. http://www.netlib.org/, http://phase.hpcc.gr.jp/mirrors/netlib/.
• 12: Francis O'Carroll , Hiroshi Tezuka , Atsushi Hori , Yutaka Ishikawa, The design and implementation of zero copy MPI using commodity hardware with a high performance network, Proceedings of the 12th international conference on Supercomputing, p.243-250, July 1998, Melbourne, Australia [doi>10.1145/277830.277883]
• 13: Scott Pakin , Mario Lauria , Andrew Chien, High performance messaging on workstations: Illinois fast messages (FM) for Myrinet, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.55-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224360]
• 14: D. Sitsky and K. Hayashi. Implementing MPI for the Fujitsu AP1000/AP1000+ using polling, interrupts and remote copying. In Proceedings of Joint Symposium on Parallel Processing 1996 (JSPP'96), pages 177-184, June 1996.
• 15: An MPI Library which uses Polling, Interrupts and Remote Copying for the Fujitsu AP1000+, Proceedings of the 1996 International Symposium on Parallel Architectures, Algorithms and Networks, p.43, June 12-14, 1996
• 16: Shinji Sumimoto , Hiroshi Tezuka , Atsushi Hori , Hiroshi Harada , Toshiyuki Takahashi , Yutaka Ishikawa, The design and evaluation of high performance communication using a Gigabit Ethernet, Proceedings of the 13th international conference on Supercomputing, p.260-267, June 20-25, 1999, Rhodes, Greece [doi>10.1145/305138.305202]
• 17: Toshiyuki Takahashi , Shinji Sumimoto , Atsushi Hori , Hiroshi Harada , Yutaka Ishikawa, PM2: a high performance communication middleware for heterogeneous network environments, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.16-es, November 04-10, 2000, Dallas, Texas, United States
• 18: Osamu Tatebe , Yuetsu Kodama , Satoshi Sekiguchi , Yoshinori Yamaguchi, Highly efficient implementation of MPI point-to-point communication using remote memory operations, Proceedings of the 12th international conference on Supercomputing, p.267-273, July 1998, Melbourne, Australia [doi>10.1145/277830.277890]
• 19: Hiroshi Tezuka , Atsushi Hori , Yutaka Ishikawa , Mitsuhisa Sato, PM: An Operating System Coordinated High Performance Communication Library, Proceedings of the International Conference and Exhibition on High-Performance Computing and Networking, p.708-717, April 28-30, 1997
• 20: Thorsten von Eicken , David E. Culler , Seth Copen Goldstein , Klaus Erik Schauser, Active messages: a mechanism for integrated communication and computation, Proceedings of the 19th annual international symposium on Computer architecture, p.256-266, May 19-21, 1992, Queensland, Australia
• 21: F. C. Wong and D. E. Culler. Message Passing Interface Implementation on Active Messages.http://now.CS.Berkeley.EDU/Fastcomm/MPI/.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: Software libraries

Additional Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.2 Multiple Data Stream Architectures (Multiprocessors)
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Performance attributes

General Terms:
Algorithms, Design, Performance

Keywords:
BLACS, BLACS-FMPL, FMPL, hardware barrier synchronization, hardware broadcast, hitachi SR8000, message-passing library, remote memory operations