Evaluating design alternatives for reliable communication on high-speed networks

ABSTRACT

We systematically evaluate the performance of five implementations of a single, user-level communication interface. Each implementation makes different architectural assumptions about the reliability of the network hardware and the capabilities of the network interface. The implementations differ accordingly in their division of protocol tasks between host software, network-interface firmware, and network hardware. Using microbenchmarks, parallel-programming systems, and parallel applications, we assess the performance impact of different protocol decompositions. We show how moving protocol tasks to a relatively slow network interface yields both performance advantages and disadvantages, depending on the characteristics of the application and the underlying parallel-programming system. In particular, we show that a communication system that assumes highly reliable network hardware and that uses network-interface support to process multicast traffic performs best for all applications.

REFERENCES

• 1: Soichiro Araki , Angelos Bilas , Cezary Dubnicki , Jan Edler , Koichi Konishi , James Philbin, User-space communication: a quantitative study, Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), p.1-16, November 07-13, 1998, San Jose, CA
• 2: Mohit Aron , Peter Druschel, Soft timers: efficient microsecond software timer support for network processing, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.232-246, December 12-15, 1999, Charleston, South Carolina, United States
• 3: Henri E. Bal , Raoul Bhoedjang , Rutger Hofman , Ceriel Jacobs , Koen Langendoen , Tim Rühl , M. Frans Kaashoek, Performance evaluation of the Orca shared-object system, ACM Transactions on Computer Systems (TOCS), v.16 n.1, p.1-40, Feb. 1998 [doi>10.1145/273011.273014]
• 4: R. Bhoedjang. Communication Architectures for Parallel- Programming Systems. PhD thesis, Dept. of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands, June 2000.
• 5: Raoul Bhoedjang , Tim Rühl , Henri E. Bal, Efficient Multicast on Myrinet using Link-Level Flow Control, Proceedings of the 1998 International Conference on Parallel Processing, p.381, August 10-14, 1998
• 6: Raoul A. F. Bhoedjang , Tim Rühl , Henri E. Bal, User-Level Network Interface Protocols, Computer, v.31 n.11, p.53-60, November 1998 [doi>10.1109/2.730737]
• 7: Angelos Bilas , Dongming Jiang , Yuanyuan Zhou , Jaswinder Pal Singh, Limits to the Performance of Software Shared Memory: A Layered Approach, Proceedings of the 5th International Symposium on High Performance Computer Architecture, p.193, January 09-12, 1999
• 8: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 9: B. Chun, A. Mainwaring, and D. Culler. Virtual Network Transport Protocols for Myrinet. In Hot Interconnects'97, Stanford, CA, Apr. 1997.
• 10: David E. Culler , Lok Tin Liu , Richard P. Martin , Chad O. Yoshikawa, Assessing Fast Network Interfaces, IEEE Micro, v.16 n.1, p.35-43, February 1996 [doi>10.1109/40.482310]
• 11: C. Dubnicki, A. Bilas, Y. Chen, S. Damianakis, and K. Li. VMMC-2: Efficient Support for Reliable, Connection- Oriented Communication. In Hot Interconnects'97, Stanford, CA, Apr. 1997.
• 12: Mario Gerla , Prasasth Palnati , Simon Walton, Multicasting protocols for high-speed, wormhole-routing local area networks, Conference proceedings on Applications, technologies, architectures, and protocols for computer communications, p.184-193, August 28-30, 1996, Palo Alto, California, United States
• 13: William Gropp , Ewing Lusk , Nathan Doss , Anthony Skjellum, A high-performance, portable implementation of the MPI message passing interface standard, Parallel Computing, v.22 n.6, p.789-828, Sept. 1996 [doi>10.1016/0167-8191(96)00024-5]
• 14: Y. Huang and P. McKinley. Efficient Collective Operations with ATM Network Interface Support. In Proc. of the Int. Conf. on Parallel Processing, pp. 34--43, Bloomingdale, IL, Aug. 1996.
• 15: K. L. Johnson , M. F. Kaashoek , D. A. Wallach, CRL: high-performance all-software distributed shared memory, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.213-226, December 03-06, 1995, Copper Mountain, Colorado, United States
• 16: Vijay Karamcheti , Andrew A. Chien, Software overhead in messaging layers: where does the time go?, Proceedings of the sixth international conference on Architectural support for programming languages and operating systems, p.51-60, October 05-07, 1994, San Jose, California, United States
• 17: Ram Kesavan , Dhabaleswar K. Panda, Optimal Multicast with Packetization and Network Interface Support, Proceedings of the international Conference on Parallel Processing, p.370-377, August 11-15, 1997
• 18: Arvind Krishnamurthy , Klaus E. Schauser , Chris J. Scheiman , Randolph Y. Wang , David E. Culler , Katherine Yelick, Evaluation of architectural support for global address-based communication in large-scale parallel machines, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.37-48, October 01-04, 1996, Cambridge, Massachusetts, United States
• 19: Olivier Maquelin , Guang R. Gao , Herbert H. J. Hum , Kevin B. Theobald , Xin-Min Tian, Polling watchdog: combining polling and interrupts for efficient message handling, Proceedings of the 23rd annual international symposium on Computer architecture, p.179-188, May 22-24, 1996, Philadelphia, Pennsylvania, United States
• 20: Richard P. Martin , Amin M. Vahdat , David E. Culler , Thomas E. Anderson, Effects of communication latency, overhead, and bandwidth in a cluster architecture, Proceedings of the 24th annual international symposium on Computer architecture, p.85-97, June 01-04, 1997, Denver, Colorado, United States
• 21: D. Mosberger and L. Peterson. Careful Protocols or How to Use Highly Reliable Networks. In Proc. of the Fourth Workshop on Workstation Operating Systems, pp. 80-84, Napa, CA, Oct. 1993.
• 22: Scott Pakin , Mario Lauria , Andrew Chien, High performance messaging on workstations: Illinois fast messages (FM) for Myrinet, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.55-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224360]
• 23: John W. Romein , Aske Plat , Henri E. Bal , Jonathan Schaeffer, Transposition table driven work scheduling in distributed search, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.725-731, July 18-22, 1999, Orlando, Florida, United States
• 24: Hiroshi Tezuka , Atsushi Hori , Yutaka Ishikawa , Mitsuhisa Sato, PM: An Operating System Coordinated High Performance Communication Library, Proceedings of the International Conference and Exhibition on High-Performance Computing and Networking, p.708-717, April 28-30, 1997
• 25: (R) Efficient Reliable Multicast on MYRINET, Proceedings of the Proceedings of the 1996 International Conference on Parallel Processing, p.156, August 12-16, 1996
• 26: T. von Eicken , A. Basu , V. Buch , W. Vogels, U-Net: a user-level network interface for parallel and distributed computing (includes URL), Proceedings of the fifteenth ACM symposium on Operating systems principles, p.40-53, December 03-06, 1995, Copper Mountain, Colorado, United States

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies