High performance messaging on workstations: Illinois fast messages (FM) for Myrinet

ABSTRACT

The Convex SPP-1000 is the first commercial implementation of a new generation of scalable shared memory parallel computers with full cache coherence. It employs a hierarchical structure of processing communication and memory name-space management resources to provide a scalableNUMA environment. Ensembles of 8 HP PA-RISC7100 microprocessorsemploy an internal cross-bar switch and directory based cache coherence scheme to provide a tightly coupled SMP.Up to 16 processing ensembles are interconnected by a 4 ring network incorporating a full hardware implementation of the SCI protocol for a full system configuration of 128 processors. This paper presents the findings of a set of empirical studies using both synthetic test codes and full applications for the Earth and space sciences to characterize the performance properties of this new architecture. It is shown that overhead and latencies of global primitive mechanisms, while low in absolute time, are significantly more costly than similar functions local to an individual processor ensemble.

REFERENCES

• 1: The Generic Active Message Interface Specification. Available from http://now.cs.berkeley.edu/Papers/Papers/gam spec.ps, 1994.
• 2: Robert Alverson , David Callahan , Daniel Cummings , Brian Koblenz , Allan Porterfield , Burton Smith, The Tera computer system, Proceedings of the 4th international conference on Supercomputing, p.1-6, June 11-15, 1990, Amsterdam, The Netherlands
• 3: Thomas E. Anderson , David E. Culler , David A. Patterson , and the NOW team, A Case for NOW (Networks of Workstations), IEEE Micro, v.15 n.1, p.54-64, February 1995 [doi>10.1109/40.342018]
• 4: Terry M. Anderson , Robert S. Cornelius, High-performance switching with fibre channel, Proceedings of the thirty-seventh international conference on COMPCON, p.261-264, January 1992, San Francisco, California, United States
• 5: G. Armitage and K. Adams. How inefficient is IP over ATM anyway? IEEE Network, Jan/Feb 1995.
• 6: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 7: CCITT, SG XVIII, Report R34. Draft Recommendation I.150: B-ISDN ATM functional characteristics, June 1990.
• 8: Andrew A. Chien, Vijay Karamcheti, John Plevyak, and Xingbin Zhang. Concurrent aggregates language report 2.0. Available via anonymous ftp from cs.uiuc.edu in /pub/csag or from http://www-csag.cs.uiuc.edu/, September 1993.
• 9: D. Clark, V. Jacobson, J Romkey, and H. Salwen. An analysis of TCP processing overhead. IEEE Communication Magazine, 27(6):23-29, June 1989.
• 10: Douglas E. Comer, Internetworking with TCP/IP (2nd ed.), vol. I, Prentice-Hall, Inc., Upper Saddle River, NJ, 1991
• 11: Cray Research, Inc. Cray T3D System Architecture Overview, March 1993.
• 12: Peter Druschel , Larry L. Peterson, Fbufs: a high-bandwidth cross-domain transfer facility, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.189-202, December 05-08, 1993, Asheville, North Carolina, United States
• 13: Fiber-distributed data interface (FDDI)-Token ring media access control (MAC). American National Standard for Information Systems ANSI X3.139-1987, July 1987. American National Standards Institute.
• 14: Message P Forum, MPI: A Message-Passing Interface Standard, University of Tennessee, Knoxville, TN, 1994
• 15: H. Franke, C. E. Wu, M Riviere, P Pattnik, and M Snir. MPI programming environment for IBM SP1/SP2. In Proceedings of the International Symposium on Computer Architecture, 1995.
• 16: F. Hady, R. Minnich, and D. Burns. The Memory Integrated Network Interface. In Proceedings of the IEEE Symposium on Hot Interconnects, 1994.
• 17: Mark Henderson, Bill Nickless, and Rick Stevens. A scalable highperformance I/O system. In Proceedings of the Scalable High- Performance Computing Conference, pages 79-86, 1994.
• 18: James Hoe and A. Boughton. Network substrate for parallel processing on a workstation cluster. In Proceedings of the IEEE Symposium on Hot Interconnects, 1994.
• 19: H. Houh, J. Adam, M. Ismert, C. Lindblad, and D. Tennenhouse. The VuNet desk area network: Architecture, implementation and experience. IEEE Journal of Selected Areas in Communications, 1995.
• 20: IBM 9076 Scalable POWERparallel 1: General information. IBM brochure GH26-7219-00, February 1993. Available from http://ibm.tc.cornell.edu/ibm/pps/sp2/index.html .
• 21: Intel Corporation. Paragon XP/S Product Overview, 1991.
• 22: Vijay Karamcheti , Andrew A. Chien, A comparison of architectural support for messaging in the TMC CM-5 and the Cray T3D, Proceedings of the 22nd annual international symposium on Computer architecture, p.298-307, June 22-24, 1995, S. Margherita Ligure, Italy
• 23: Vijay Karamcheti and Andrew A. Chien. FM-fast messaging on the Cray T3D. Available from http://www-csag.cs.uiuc.edu/papers/t3d-fmmanual. ps, February 1995.
• 24: M. Liu, J. Hsieh, D. Hu, J. Thomas, and J. MacDonald. Distributed network computing over Local ATM Networks. In Supercomputing '94, 1995.
• 25: R. Martin. HPAM: An Active Message layer for a network of HP workstation. In Proceedings of the IEEE Symposium on Hot Interconnects, 1994. Available from ftp://ftp.cs.berkeley.edu/ucb/CASTLE/Active Messages/hotipaper.ps.
• 26: Meiko World Incorporated. Meiko Computing Surface Communications Processor Overview, 1993.
• 27: V. S. Sunderam, PVM: a framework for parallel distributed computing, Concurrency: Practice and Experience, v.2 n.4, p.315-339, Dec. 1990 [doi>10.1002/cpe.4330020404]
• 28: A. S. Tanenbaum, Computer networks, Prentice-Hall, Inc., Upper Saddle River, NJ, 1988
• 29: Thinking Machines Corporation, 245 First Street, Cambridge, MA 02154-1264. The Connection Machine CM-5 Technical Summary, October 1991.
• 30: Thorsten von Eicken , Anindya Basu , Vineet Buch, Low-Latency Communication Over ATM Networks Using Active Messages, IEEE Micro, v.15 n.1, p.46-53, February 1995 [doi>10.1109/40.342017]
• 31: Thorsten von Eicken , David E. Culler , Seth Copen Goldstein , Klaus Erik Schauser, Active messages: a mechanism for integrated communication and computation, Proceedings of the 19th annual international symposium on Computer architecture, p.256-266, May 19-21, 1992, Queensland, Australia

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.0 General
          Subjects: Data communications
      C.2.1 Network Architecture and Design
          Subjects: Packet-switching networks
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.1 Large and Medium ("Mainframe") Computers
          Subjects: Super (very large) computers
      C.5.3 Microcomputers
          Subjects: Workstations

General Terms:
Measurement, Performance