TCP offload through connection handoff

ABSTRACT

This paper presents a connection handoff interface between the operating system and the network interface. Using this interface, the operating system can offload a subset of TCP connections in the system to the network interface, while the remaining connections are processed on the host CPU. Offloading can reduce computation and memory bandwidth requirements for packet processing on the host CPU. However, full TCP offloading may degrade system performance because finite processing and memory resources on the network interface limit the amount of packet processing and the number of connections. Using handoff, the operating system controls the number of offloaded connections in order to fully utilize the network interface without overloading it. Handoff is transparent to the application, and the operating system may choose to offload connections to the network interface or reclaim them from the interface at any time. A prototype system based on the modified FreeBSD operating system shows that handoff reduces the number of instructions and cache misses on the host CPU. As a result, the number of CPU cycles spent processing each packet decreases by 16--84%. Simulation results show handoff can improve web server throughput (SEPCweb99) by 15%, despite short-lived connections.

REFERENCES

• 1: K. Banerjee, A. Bohra, S. Gopalakrishnan, M. Rangarajan, and L. Iftode. Split-OS: An Operating System Architecture for Clusters of Intelligent Devices. In Work-in-Progress Session at the 18th Symposium on Operating Systems Principles, Oct. 2001.
• 2: D. D. Clark, V. Jacobson, J. Romkey, and H. Salwen. An Analysis of TCP Processing Overhead. IEEE Communications Magazine, pages 23--29, June 1989.
• 3: Peter Druschel , Larry L. Peterson, Fbufs: a high-bandwidth cross-domain transfer facility, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.189-202, December 05-08, 1993, Asheville, North Carolina, United States
• 4: D. Freimuth, E. Hu, J. LaVoie, R. Mraz, E. Nahum, P. Pradhan, and J. Tracey. Server Network Scalability and TCP Offload. In Proceedings of the 2005 Annual USENIX Technical Conference, pages 209--222, Apr. 2005.
• 5: H.-y. Kim and S. Rixner. Performance Characterization of the FreeBSD Network Stack. Computer Science Department, Rice University, June 2005. Technical Report TR05--450.
• 6: Y. Hoskote, B. A. Bloechel, G. E. Dermer, V. Erraguntla, D. Finan, J. Howard, D. Klowden, S. Narendra, G. Ruhl, J. W. Tschanz, S. Vangal, V. Veeramachaneni, H. Wilson, J. Xu, and N. Borkar. A TCP Offload Accelerator for 10 Gb/s Ethernet in 90-nm CMOS. IEEE Journal of Solid-State Circuits, 38(11): 1866--1875, Nov. 2003.
• 7: Karl Kleinpaste , Peter Steenkiste , Brian Zill, Software support for outboard buffering and checksumming, Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication, p.87-98, August 28-September 01, 1995, Cambridge, Massachusetts, United States
• 8: Peter S. Magnusson , Magnus Christensson , Jesper Eskilson , Daniel Forsgren , Gustav Hållberg , Johan Högberg , Fredrik Larsson , Andreas Moestedt , Bengt Werner, Simics: A Full System Simulation Platform, Computer, v.35 n.2, p.50-58, February 2002 [doi>10.1109/2.982916]
• 9: Srihari Makineni , Ravi Iyer, Architectural Characterization of TCP/IP Packet Processing on the Pentium® M Microprocessor, Proceedings of the 10th International Symposium on High Performance Computer Architecture, p.152, February 14-18, 2004 [doi>10.1109/HPCA.2004.10024]
• 10: Microsoft Corporation. Scalable Networking: Network Protocol Offload -- Introducing TCP Chimney, Apr. 2004. WinHEC Version.
• 11: Jeffrey Mogul , Lawrence Brakmo , David E. Lowell , Dinesh Subhraveti , Justin Moore, Unveiling the transport, ACM SIGCOMM Computer Communication Review, v.34 n.1, January 2004 [doi>10.1145/972374.972392]
• 12: J. C. Mogul. TCP offload is a dumb idea whose time has come. In Proceedings of HotOS IX: The 9th Workshop on Hot Topics in Operating Systems, pages 25--30, 2003.
• 13: Erich Nahum , David Yates , Jim Kurose , Don Towsley, Cache behavior of network protocols, Proceedings of the 1997 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.169-180, June 15-18, 1997, Seattle, Washington, United States
• 14: Vivek S. Pai , Mohit Aron , Gaurov Banga , Michael Svendsen , Peter Druschel , Willy Zwaenepoel , Erich Nahum, Locality-aware request distribution in cluster-based network servers, Proceedings of the eighth international conference on Architectural support for programming languages and operating systems, p.205-216, October 02-07, 1998, San Jose, California, United States
• 15: V. S. Pai, P. Druschel, and W. Zwaenepoel. Flash: An Efficient and Portable Web Server. In Proceedings of the USENIX 1999 Annual Technical Conference, pages 199--212, June 1999.
• 16: Vivek S. Pai , Peter Druschel , Willy Zwaenepoel, IO-lite: a unified I/O buffering and caching system, Proceedings of the third symposium on Operating systems design and implementation, p.15-28, February 1999, New Orleans, Louisiana, United States
• 17: A. E. Papathanasiou and E. V. Hensbergen. KNITS: Switch-based Connection Hand-off. In Proceedings of IEEE INFOCOM '02, volume 1, pages 332--341, 2002.
• 18: M. Rangarajan, A. Bohra, K. Banerjee, E. V. Carrera, R. Bianchini, L. Iftode, and W. Zwaenepoel. TCP Servers: Offloading TCP/IP Processing in Internet Servers. Design, Implementation, and Performance. Computer Science Department, Rutgers University, Mar. 2002. Technical Report DCR-TR-481.
• 19: Greg Regnier , Srihari Makineni , Ramesh Illikkal , Ravi Iyer , Dave Minturn , Ram Huggahalli , Don Newell , Linda Cline , Annie Foong, TCP Onloading for Data Center Servers, Computer, v.37 n.11, p.48-58, November 2004 [doi>10.1109/MC.2004.223]
• 20: Scott Rixner, Memory Controller Optimizations for Web Servers, Proceedings of the 37th annual IEEE/ACM International Symposium on Microarchitecture, p.355-366, December 04-08, 2004, Portland, Oregon [doi>10.1109/MICRO.2004.22]
• 21: Luigi Rizzo, Dummynet: a simple approach to the evaluation of network protocols, ACM SIGCOMM Computer Communication Review, v.27 n.1, p.31-41, Jan. 1997 [doi>10.1145/251007.251012]
• 22: A. Snoeren, D. G. Andersen, and H. Balakrishnan. Fine-Grained Failover Using Connection Migration. In Proceedings of the 3rd USENIX Symposium on Internet Technologies and Systems, Mar. 2001.
• 23: R. Westrelin, N. Fugier, E. Nordmark, K. Kunze, and E. Lemoine. Studying Network Protocol Offload With Emulation: Approach And Preliminary Results. In Proceedings of the 12th Annual IEEE Symposium on High Performance Interconnects, pages 84--90, Aug. 2004.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management

General Terms:
Experimentation, Performance

Keywords:
TCP offload, connection handoff, operating system, programmable network interface