Limits to low-latency communication on high-speed networks

ABSTRACT

The throughput of local area networks is rapidly increasing. For example, the bandwidth of new ATM networks and FDDI token rings is an order of magnitude greater than that of Ethernets. Other network technologies promise a bandwidth increase of yet another order of magnitude in several years. However, in distributed systems, lowered latency rather than increased throughput is often of primary concern. This paper examines the system-level effects of newer high-speed network technologies on low-latency, cross-machine communications. To evaluate a number of influences, both hardware and software, we designed and implemented a new remote procedure call system targeted at providing low latency. We then ported this system to several hardware platforms (DECstation and SPARCstation) with several different networks and controllers (ATM, FDDI, and Ethernet). Comparing these systems allows us to explore the performance impact of alternative designs in the communication system with respect to achieving low latency, e.g., the network, the network controller, the hose architecture and cache system, and the kernel and user-level runtime software. Our RPC system, which achieves substantially reduced call times (170 &mgr;seconds on an ATM network using DECstation 5000/200 hosts), allows us to isolate those components of next-generation networks and controllers that still stand in the way of low-latency communication. We demonstrate that new-generation processor technology and software design can reduce small-packet RPC times to near network-imposed limits, making network and controller design more crucial than ever to achieving truly low-latency communication.

REFERENCES

• 1: ADVANCED MICRO DEVICES. Am7990 Local Area Network Controller for Ethernet (LANCE). Advanced Micro Devices, Sunnyvale, Calif., 1986.
• 2: Ross, F.E. FDDI A tutorial. IEEE Commun. Mag. 24, 5 (May 1986), 10-17.
• 3: Thomas E. Anderson , Henry M. Levy , Brian N. Bershad , Edward D. Lazowska, The interaction of architecture and operating system design, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.108-120, April 08-11, 1991, Santa Clara, California, United States
• 4: Brian N. Bershad , Thomas E. Anderson , Edward D. Lazowska , Henry M. Levy, Lightweight remote procedure call, ACM Transactions on Computer Systems (TOCS), v.8 n.1, p.37-55, Feb. 1990 [doi>10.1145/77648.77650]
• 5: Andrew D. Birrell , Bruce Jay Nelson, Implementing remote procedure calls, ACM Transactions on Computer Systems (TOCS), v.2 n.1, p.39-59, February 1984 [doi>10.1145/2080.357392]
• 6: D. L. Black , R. F. Rashid , D. B. Golub , C. R. Hill, Translation lookaside buffer consistency: a software approach, Proceedings of the third international conference on Architectural support for programming languages and operating systems, p.113-122, April 03-06, 1989, Boston, Massachusetts, United States
• 7: J. Chase , F. Amador , E. Lazowska , H. Levy , R. Littlefield, The Amber system: parallel programming on a network of multiprocessors, Proceedings of the twelfth ACM symposium on Operating systems principles, p.147-158, November 1989
• 8: CHERITON, D. R The V kernel: A software base for distributed systems. IEEE Softw. 1, 2 (Apr. 1984), 19-42.
• 9: D. D. Clark , D. L. Tennenhouse, Architectural considerations for a new generation of protocols, Proceedings of the ACM symposium on Communications architectures & protocols, p.200-208, September 26-28, 1990, Philadelphia, Pennsylvania, United States
• 10: CLARK, D. D., JACOBSON, V., ROMKEY, J., AND SALWEN, H. An analysis of TCP processing overhead. IEEE Commun. Mag. 27, 6 (June 1989), 23-36.
• 11: COMER, D., AND GRIFFIOEN, J. A new design for distributed systems: The remote memory model. In Proceedings of the Summer 1990 USENIX Conference (June 1990), 127-135.
• 12: Bruce S. Davie, A host-network interface architecture for ATM, Proceedings of the conference on Communications architecture & protocols, p.307-315, September 03-06, 1991, Zurich, Switzerland
• 13: DIGITAL EQUIPMENT CORPORATION. TURBOChannel Hardware Specification. 1991.
• 14: DIGITAL EQUIPMENT CORPORATION. PMADD-AA TurboChannel Ethernet Module Functional Specification, Rev. 1.2. Workstation Systems Engineering, 1990.
• 15: Richard L. Sites, Alpha architecture reference manual, Digital Press, Newton, MA, 1992
• 16: FORE SYSTEMS. TCA-IO0 TURBOchannel ATM Computer Interface, User's Manual. FORE Systems, Pittsburgh, Pa. 1992
• 17: C. A. R. Hoare, Communicating sequential processes, Communications of the ACM, v.21 n.8, p.666-677, Aug. 1978 [doi>10.1145/359576.359585]
• 18: JOHNSON, D. B., AND ZWAENEPOEL, W. The Peregrine high performance RPC system. Tech. Rep. COMP TR91-152, Dept. of Computer Science, Rme Univ., 1991.
• 19: H. Kanakia , D. Cheriton, The VMP network adapter board (NAB): high-performance network communication for multiprocessors, Symposium proceedings on Communications architectures and protocols, p.175-187, August 16-18, 1988, Stanford, California, United States
• 20: David Keppel, A portable interface for on-the-fly instruction space modification, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.86-95, April 08-11, 1991, Santa Clara, California, United States
• 21: Kai Li , Paul Hudak, Memory coherence in shared virtual memory systems, ACM Transactions on Computer Systems (TOCS), v.7 n.4, p.321-359, Nov. 1989 [doi>10.1145/75104.75105]
• 22: H. Massalin , C. Pu, Threads and input/output in the synthesis kernal, Proceedings of the twelfth ACM symposium on Operating systems principles, p.191-201, November 1989
• 23: Robert M. Metcalfe , David R. Boggs, Ethernet: distributed packet switching for local computer networks, Communications of the ACM, v.19 n.7, p.395-404, July 1976 [doi>10.1145/360248.360253]
• 24: Jeffrey C. Mogul , Anita Borg, The effect of context switches on cache performance, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.75-84, April 08-11, 1991, Santa Clara, California, United States
• 25: MULLENDER, S. J., AND TANENBAUM, A.S. The design of a capability-based operating system. Comput. J. 29, 4 (1986), 289-299.
• 26: OUSTERHOUT, J. K. Why aren't operating systems getting faster as fast as hardware? In Proceedings of the Summer 1990 USENIX Conference (June 1990), 247-256.
• 27: Michael D. Schroeder , Michael Burrows, Performance of the Firefly RPC, ACM Transactions on Computer Systems (TOCS), v.8 n.1, p.1-17, Feb. 1990 [doi>10.1145/77648.77653]
• 28: SPEC. Newsletter benchmark results. Systems Performance Evaluation Cooperative, 1990.
• 29: SUN MICROSYSTEMS, INC. SBus Specification B.O. Sun Microsystems, Inc., Mountain View, Calif., 1990.
• 30: Charles P. Thacker , Lawrence C. Stewart , Edwin H. Satterthwaite, Jr., Firefly: A Multiprocessor Workstation, IEEE Transactions on Computers, v.37 n.8, p.909-920, August 1988 [doi>10.1109/12.2243]
• 31: C. Brendan S. Traw , Jonathan M. Smith, A high-performance host interface for ATM networks, Proceedings of the conference on Communications architecture & protocols, p.317-325, September 03-06, 1991, Zurich, Switzerland
• 32: ULM, J. N. A timed-token ring local area network and its performance characteristics. In Proceedings of the 7th IEEE Conference on Local Computer Networks (Feb. 1982). IEEE, New York, 50-56.
• 33: R. van Renesse , H. van Staveren , A. D. Tanenbaum, The performance of the Amoeba distributed operating system, Software—Practice & Experience, v.19 n.3, p.223-234, March 1989 [doi>10.1002/spe.4380190303]
• 34: MINZER, S.E. Broadband ISDN and asynchronous transfer mode (ATM). IEEE Commun. Mag. 27, 9 (Sept. 1989), 17-24, 57.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management
          Subjects: Network communication

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices
          Subjects: Channels and controllers

  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.2 Network Protocols
          Subjects: Protocol architecture (OSI model)
      C.2.4 Distributed Systems
          Subjects: Distributed applications; Network operating systems

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management
          Subjects: Message sending
      D.4.7 Organization and Design
          Subjects: Distributed systems

General Terms:
Design, Measurement, Performance

Keywords:
ATM networks, host-network interfaces, interprocess communication, remote procedure calls, transport level protocols