A scalable communication layer for multi-dimensional hyper crossbar network using multiple gigabit ethernet

ABSTRACT

This paper proposes a scalable communication layer for a multi-dimensional hyper crossbar network using multiple Gigabit Ethernet for the PACS-CS system which consists of 2560 single-processor nodes and a 16 x 16 x 10 three dimensional hyper-crossbar network (3D-HXB). To realize a high performance communication layer using multiple existing Ethernet networks, the host processor usage for the communication processing must be reduced to less than the appropriate packet processing time which is calculated from a message size and a target communication bandwidth. To overcome this problem, we have developed the PM/Ethernet-HXB communication facility. PM/Ethernet-HXB realizes communication protocol processing without exclusion even for Zero-copy communication between the communication buffers of nodes. We have implemented the PM/Ethernet-HXB on SCore cluster system software, and evaluated its communication and application performance. PM/Ethernet-HXB achieves a unidirectional communication bandwidth of 1065 MB/s using nine Gigabit Ethernet links on a single dimension network. It also realizes a unidirectional communication bandwidth of 741 MB/s (98.8% of the theoretical performance) and a bidirectional bandwidth of 1401 MB/s (93.4% of the theoretical performance) on the three dimensional connections (3D-HXB: a total of six Ethernet links). The results of MPI communication bandwidth are a unidirectional communication bandwidth of 960 MB/s and a bidirectional bandwidth of 1008 MB/s using eight links on a single dimension network. These results show that PM/Ethernet-HXB realizes a comparative performance using multiple Gigabit Ethernet networks to dedicated cluster networks such as InfiniBand 4x (1000 MB/s). The speedups of IS and CG Class C NAS parallel benchmarks are scalable up to using four links on eight node cluster, and performance degradation between 3D-HXB (2 x 2 x 2) and 1-dimensional network is small.

REFERENCES

• 1: PACS-CS Project: http://www.ccs.tsukuba.ac.jp/PACS-CS/.
• 2: InfiniBand Trade Association: http://www.infinibandta.org/.
• 3: SCore Cluster System Software: http://www.pccluster.org/.
• 4: The Local Area Memory Port, Local Area MultiProcessor, Scalable Coherent Interface, and Serial Express Users, Developers, and Manufacturers Association: http://www.scizzl.com/.
• 5: YAMPII Yet Another MPI Implementation: http://www.il.is.s.u-tokyo.ac.jp/yampii/index.html.
• 6: ELAN(QSNET): http://www.quadrics.com/.
• 7: 10 Gigabit Ethernet Alliance: http://www.10gea.org/.
• 8: Myrinet 10G: http://www.myri.com/.
• 9: A. Basu, V. Buch, W. Vogels, and T. von Eicken. U-Net: A User-Level Network Interface for Parallel and Distributed Computing. In Proceedings of the Third International Symposium on High Performance Computer Architecture (HPCA), February 1997.
• 10: Taisuke Boku , Ken'ichi Itakura , Hiroshi Nakamura , Kisaburo Nakazawa, CP-PACS: a massively parallel processor for large scale scientific calculations, Proceedings of the 11th international conference on Supercomputing, p.108-115, July 07-11, 1997, Vienna, Austria [doi>10.1145/263580.263608]
• 11: M. Boosten, R. W. Dobinson, and P. D. V. van der Stok. Mesh: Messageing and scheduling for fine-grain parallel processing on commodity plattforms. In Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, PDPTA 1999, June 28 - Junlly 1, 1999, Las Vegas, Nevada, USA, pages 1716--1722. CSREA Press, 1999.
• 12: Charles L. Seitz , Nanette J. Boden , Jakov Seizovic , Wen-King Su, The design of the Caltech Mosaic C multicomputer, Proceeding of the 1993 symposium on Research on integrated systems, p.1-22, February 1993, Seattle, Washington, United States
• 13: Jie Chen , William Watson III , Robert Edwards , Weizhen Mao, Message Passing for Linux Clusters with Gigabit Ethernet Mesh Connections, Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 9, p.214.1, April 04-08, 2005 [doi>10.1109/IPDPS.2005.287]
• 14: G. Chiola , G. Ciaccio, Efficient parallel processing on low-cost clusters with GAMMA active ports, Parallel Computing, v.26 n.2-3, p.333-354, Feb. 2000 [doi>10.1016/S0167-8191(99)00108-8]
• 15: C. Csanady and P. Wyckoff. Bobnet: Highperformance message passing for commodity networking components, 1998.
• 16: John A. Ronciak, Jesse Brandeburg, and Ganesh Vnekatesan. Page-Flip Technology for use within the Linux Networking Stack. In Proceedings of the Linux Symposium 2004, http://www.linuxsymposium.org/proceedings/reprints/Reprint-Ronciak-OLS2004.pdf, July 2004.
• 17: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 18: PACS-CS Collaboration : S. Aoki, K. Ishikawa, T. Ishikawa, N. Ishizuka, K. Kanaya, Y. Kuramashi, M. Okawa, K. Sasaki, Y. Taniguchi, N. Tsutsui, A. Ukawa, and T. Yoshie. The PACS-CS Project. In PoS: PROCEEDINGS of SCIENCE, volume LAT2005, page 111, Oct. 2005.
• 19: Piyush Shivam , Pete Wyckoff , Dhabaleswar Panda, EMP: zero-copy OS-bypass NIC-driven gigabit ethernet message passing, Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM), p.57-57, November 10-16, 2001, Denver, Colorado [doi>10.1145/582034.582091]
• 20: S. Sumimoto. Ph.D Dissertation presented to Keio University, 2000, http://www.pccluster.org/score/dist/score/papers/sumimoto00.pdf.
• 21: Shinji Sumimoto , Kouichi Kumon, PM/Ehernet-kRMA: A High Performance Remote Memory Access Facility Using Multiple Gigabit Ethernet Cards, Proceedings of the 3st International Symposium on Cluster Computing and the Grid, p.326, May 12-15, 2003
• 22: Shinji Sumimoto , Hiroshi Tezuka , Atsushi Hori , Hiroshi Harada , Toshiyuki Takahashi , Yutaka Ishikawa, The design and evaluation of high performance communication using a Gigabit Ethernet, Proceedings of the 13th international conference on Supercomputing, p.260-267, June 20-25, 1999, Rhodes, Greece [doi>10.1145/305138.305202]
• 23: Shinji Sumimoto , Hiroshi Tezuka , Atsushi Hori , Hiroshi Harada , Toshiyuki Takahashi , Yutaka Ishikawa, High Performance Communication using a Commodity Network for Cluster Systems, Proceedings of the Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC'00), p.139, August 01-04, 2000
• 24: T. Sterling , D. Savarese , D. J. Becker , B. Fryxell , K. Olson, Communication overhead for space science applications on the Beowulf parallel workstation, Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing - HPDC '95, p.23, August 02-04, 1995
• 25: Toshiyuki Takahashi , Shinji Sumimoto , Atsushi Hori , Hiroshi Harada , Yutaka Ishikawa, PM2: a high performance communication middleware for heterogeneous network environments, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.16-es, November 04-10, 2000, Dallas, Texas, United States

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design

          Nouns: Ethernet

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.3 Interconnections (subsystems)

  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Network topology
      C.2.4 Distributed Systems
      C.2.5 Local and Wide-Area Networks
          Subjects: Ethernet (e.g., CSMA/CD)

General Terms:
Design, Performance, Theory