On the elusive benefits of protocol offload

ABSTRACT

Periodic order-of-magnitude jumps in Ethernet bandwidth regularly reawaken interest in TCP/IP transport protocol offload. This time the jump to 10-Gigabit Ethernet coincides with the emergence of new network storage protocols (iSCSI and DAFS), and vendors are combining these with offload NICs to position IP as a competitor to FibreChannel and other SAN interconnects. But what benefits will offload show for application performance?Several recent studies have presented conflicting data to argue that offload either does or does not benefit applications. But the evidence from empirical studies is often little better than anecdotal. The principles that determine the results are not widely understood, except for the first principle: Your Mileage May Vary.This paper outlines fundamental performance properties of transport offload and other techniques for low-overhead I/O in terms of four key ratios that capture the CPU-intensity of the application and the relative speeds of the host, NIC device, and network path. The study also reflects the role of offload as an enabler for direct data placement, which eliminates some communication overheads rather than merely shifting them to the NIC. The analysis applies to Internet services, streaming data, and other scenarios in which end-to-end throughput is limited by network bandwidth or processing overhead rather than latency.

REFERENCES

• 1: Offload White Paper. http://www.national.com/appinfo/networks/files/whitepaper1.pdf, 2003.
• 2: Jeffrey S. Chase. TCP implementation. In High-Performance TCP/IP Networking, Mahbub Hassan and Raj Jain, editors. Prentice Hall, December 2003. ISBN 0-13-064634-2. http://issg.cs.duke.edu/publications/tcp.pdf
• 3: Jeffrey S. Chase, Andrew J. Gallatin, and Kenneth G. Yocum. End system optimizations for high-speed TCP. IEEE Communications, Special Issue on High-Speed TCP, 39(4):68--74, April 2001.
• 4: David D. Clark, Van Jacobson, John Romkey, and Howard Salwen. An analysis of TCP processing overhead. IEEE Communications Magazine, 27(6):23--29, June 1989.
• 5: David Culler , Richard Karp , David Patterson , Abhijit Sahay , Klaus Erik Schauser , Eunice Santos , Ramesh Subramonian , Thorsten von Eicken, LogP: towards a realistic model of parallel computation, Proceedings of the fourth ACM SIGPLAN symposium on Principles and practice of parallel programming, p.1-12, May 19-22, 1993, San Diego, California, United States
• 6: Matt DeBergalis , Peter Corbett , Steve Kleiman , Arthur Lent , Dave Noveck , Tom Talpey , Mark Wittle, The Direct Access File System, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA
• 7: John L. Hennessy , David A. Patterson, Computer architecture (2nd ed.): a quantitative approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1996
• 8: Van Jacobson. 4BSD header prediction. ACM Computer Communication Review, 20(2):13--15, April 1990.
• 9: Jonathan Kay , Joseph Pasquale, The importance of non-data touching processing overheads in TCP/IP, Conference proceedings on Communications architectures, protocols and applications, p.259-268, September 13-17, 1993, San Francisco, California, United States
• 10: Kostas Magoutis , Salimah Addetia , Alexandra Fedorova , Margo I. Seltzer, Making the Most Out of Direct-Access Network Attached Storage, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA
• 11: Jeff Mogul. TCP offload is a dumb idea whose time has come. In Workshop on Hot Topics in Operating Systems (HotOS). May 2003.
• 12: M.Rangarajan, A. Bohra, K. Banerjee, E. Carrera, R. Bianchini, L. Iftode, and W. Zwaenepoel. TCP Servers: Offloading TCP Processing in Internet Servers---Design, Implementation and Performance. Technical Report, DCS-TR-481.
• 13: Prasenjit Sarkar , Sandeep Uttamchandani , Kaladhar Voruganti, Storage Over IP: When Does Hardware Support Help?, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Design, Measurement, Performance

Keywords:
IP, Protocol offload, RDMA, TCP, high-speed networking, network performance, network storage