Software support for outboard buffering and checksumming

ABSTRACT

Data copying and checksumming are the most expensive operations when doing high-bandwidth network IO over a high-speed network. Under some conditions, outboard buffering and checksumming can eliminate accesses to the data, thus making communication less expensive and faster. One of the scenarios in which outboard buffering pays off is the common case of applications accessing the network using the Berkeley sockets interface and the Internet protocol stack. In this paper we describe the changes that were made to a BSD protocol stack to make use of a network adaptor that supports outboard buffering and checksumming. Our goal is not only to achieve "single copy" communication for application that use sockets, but to also have efficient communication for in-kernel applications and for applications using other networks. Performance measurements show that for large reads and writes the single-copy path through the stack is significantly more efficient than the original implementation.

REFERENCES

• 1: D. Borman, R. Braden, and V. Jacobson. Tcp extensions for high performance. Request for Comments 1323, May 1992.
• 2: Jose Brustoloni. Exposed buffering and subdatagram flow control for ATM LANs. In Proceedings of the 19th Conference on Local Computer Networks, pages 324-334. IEEE, October 1994.
• 3: David D. Clark, Van Jacobson, John Romkey, and Howard Salwen. An analysis of tcp processing overhead. IEEE Communications Magazine, 27(6):23-29, June 1989.
• 4: Jim Crapuchettes. TURBOchannel Interface ASIC Functional Specification. TRI/ADD Program, DEC, revision 0.6, preliminary edition, 1992.
• 5: Chris Dalton, Greg Watson, David Banks, Costas Calamvokis, Aled Edwards, and John Lumley. Afterburner. IEEE Network Magazine, 7(4):36-43, July 1993.
• 6: Peter Druschel , Larry L. Peterson, Fbufs: a high-bandwidth cross-domain transfer facility, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.189-202, December 05-08, 1993, Asheville, North Carolina, United States
• 7: Peter Druschel , Larry L. Peterson , Bruce S. Davie, Experiences with a high-speed network adaptor: a software perspective, Proceedings of the conference on Communications architectures, protocols and applications, p.2-13, August 31-September 02, 1994, London, United Kingdom
• 8: Aled Edwards , Greg Watson , John Lumley , David Banks , Costas Calamvokis , C. Dalton, User-space protocols deliver high performance to applications on a low-cost Gb/s LAN, Proceedings of the conference on Communications architectures, protocols and applications, p.14-23, August 31-September 02, 1994, London, United Kingdom
• 9: Ken Hardwick, HIPPI world—the switch is the network, Proceedings of the thirty-seventh international conference on COMPCON, p.234-238, January 1992, San Francisco, California, United States
• 10: M. G. Hluchyj and M.J. Karol. Queueing in highperformance packet switching, iEEE Journal on Se~ lected Areas in Communication, 6(9):1587-1597, December 1988.
• 11: Van Jacobson. Efficient protocol implementation. ACM '90 SIGCOMM tutorial, September 1990.
• 12: Van Jacobson. pbufs. Personal communication, October 1992.
• 13: Samuel J. Leffler, Marshall Kirk McKusick, Michael J. Karels, and John S. Quarterman. The Design and Implementation of the 4.3BSD UNIX Operating System. Addison-Wesley Publishing Company, Reading, Massachusetts, 1989.
• 14: J. Postel. Transmission control protocol. Request for Comments 793, September 1981.
• 15: K.K. Ramakrishnan. Performance considerations in designing network interfaces. IEEE Journal on SelectedAreas in Communication, 11 (2):203-219, February 1993.
• 16: Richard F. Rashid, Robert V. Baron. A. Forin, David B. Golub, Michael Jones, Daniel Julin, D. Orr, and R. Sanzi. Mach: A foundation for open systems. In Proceedings of the Second IEEE Workshop on Workstation Operating Systems, pages 109-113, September 1989.
• 17: Peter Steenkiste, Analyzing communication latency using the Nectar communication processor, Conference proceedings on Communications architectures & protocols, p.199-209, August 17-20, 1992, Baltimore, Maryland, United States
• 18: P. Steenkiste , M. Hemy , T. Mummert , B. Zill, Architecture and evaluation of a high-speed networking subsystem for distributed-memory systems, Proceedings of the 21ST annual international symposium on Computer architecture, p.154-163, April 18-21, 1994, Chicago, Illinois, United States
• 19: Peter A. Steenkiste, A systematic approach to host interface design for high-speed networks, Computer, v.27 n.3, p.47-57, March 1994 [doi>10.1109/2.268886]
• 20: Peter Steenkiste , Brian Zill , H. T. Kung , Steven Schlick , Jim Hughes , Bob Kowalski , John Mullaney, A Host Interface Architecture for High-Speed Networks, Proceedings of the IFIP TC6/WG6.4 Fourth International Conference on High Performance Networking IV, p.31-46, December 14-18, 1992
• 21: Linda Walmer and Mary Thompson. A Programmer's Guide to the Mach System Calls. Carnegie Mellon University, 1989.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.5 Reliability, Testing, and Fault-Tolerance**
          Subjects: Error-checking**

  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.0 General
          Subjects: Data communications

General Terms:
Algorithms, Performance