Optimizing 10-Gigabit Ethernet for Networks of Workstations, Clusters, and Grids: A Case Study

ABSTRACT

This paper presents a case study of the 10-Gigabit Ethernet (10GbE) adapter from Intel R . Specifically, with appropriate optimizations to the configurations of the 10GbE adapter and TCP, we demonstrate that the 10GbE adapter can perform well in local-area, storage-area, system-area, and wide-area networks. For local-area, storage-area, and system-area networks in support of networks of workstations, network-attached storage, and clusters, respectively, we can achieve over 7-Gb/s end-to-end throughput and 12-µs end-to-end latency between applications running on Linux-based PCs. For the wide-area network in support of grids, we broke the recently-set Internet2 Land Speed Record by 2.5 times by sustaining an end-to-end TCP/IP throughput of 2.38 Gb/s between Sunnyvale, California and Geneva, Switzerland (i.e., 10,037 kilometers) to move over a terabyte of data in less than an hour. Thus, the above results indicate that 10GbE may be a cost-effective solution across a multitude of computing environments.

REFERENCES

• 1: M. Allman , V. Paxson , W. Stevens, TCP Congestion Control, RFC Editor, 1999
• 2: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 3: D. D. Clark, Window and Acknowledgement Strategy in TCP, RFC Editor, 1982
• 4: [4] D. Clark, V. Jacobson, J. Romkey, and H. Salwen, "An Analysis of TCP Processing Overhead," IEEE Communications, Vol. 27, No. 6, June, 1989, pp. 23-29.
• 5: [5] "Communication Streaming Architecture: Reducing the PCI Network Bottleneck," Intel Whitepaper 252451-002. Available at: http://www.intel.com/design/network/papers/252451.htm.
• 6: Mark K. Gardnery , Wu-chun Fengyz , Michael Broxtonyx , Adam Engelharty , Gus Hurwitzy, MAGNET: A Tool for Debugging, Analyzing and Adapting Computing Systems, Proceedings of the 3st International Symposium on Cluster Computing and the Grid, p.310, May 12-15, 2003
• 7: [7] G. Hurwitz, and W. Feng, "Initial End-to-End Performance Evaluation of 10-Gigabit Ethernet," Proceedings of Hot Interconnects 11 (HotI'03), August 2003.
• 8: [8] "Iperf 1.6 - The TCP/UDP Bandwidth Measurement Tool," http://dast.nlanr.net/Projects/Iperf/.
• 9: [9] M. Mathis, "Pushing Up the Internet MTU," Presentation to Miami Joint Techs, Feburary 2003.
• 10: [10] J. McCalpin, "STREAM: Sustainable Memory Bandwidth in High-Performance Computers," http://www.cs.virginia.edu/ stream/.
• 11: Robert M. Metcalfe , David R. Boggs, Ethernet: distributed packet switching for local computer networks, Communications of the ACM, v.19 n.7, p.395-404, July 1976 [doi>10.1145/360248.360253]
• 12: [12] "Myrinet Ethernet Emulation (TCP/IP & UDP/IP) Performance," http://www.myri.com/myrinet/performance/ip.html.
• 13: [13] "Myrinet Performance Measurements," http://www.myri.com/ myrinet/performance/index.html.
• 14: [14] "Netperf: Public Netperf Homepage," http://www.netperf.org/.
• 15: [15] "NetPIPE," http://www.scl.ameslab.gov/netpipe/.
• 16: [16] "NTTCP: New TTCP program," http://www.leo.org/~elmar/nttcp/.
• 17: Fabrizio Petrini , Wu-chun Feng , Adolfy Hoisie , Salvador Coll , Eitan Frachtenberg, The Quadrics Network: High-Performance Clustering Technology, IEEE Micro, v.22 n.1, p.46-57, January 2002 [doi>10.1109/40.988689]
• 18: Ian R. Philp , Yin-Ling Liong, The Scheduled Transfer (ST) Protocol, Proceedings of the Third International Workshop on Network-Based Parallel Computing: Communication, Architecture, and Applications, p.108-121, January 09, 1999
• 19: [19] A. Romanow, and S. Bailey, "An Overview of RDMA over IP," Proceedings of the First International Workshop on Protocols for Fast Long-Distance Networks (PFLDnet 2003), Feburary 2003.
• 20: [20] J. Stone, and C. Partridge, "When the CRC and TCP Checksum Disagree," Proceedings of ACM SIGCOMM 2000, August 2000.
• 21: [21] "TCPDUMP Public Repository," http://www.tcpdump.org.
• 22: [22] B. Tierney, "TCP Tuning Guide for Distributed Application on Wide-Area Networks," USENIX ;login:, Vol. 26, No. 1, February 2001.
• 23: [23] S. Ubik, and P. Chimbal, "Achieving Reliable High Performance in LFNs," Proceedings of Trans-European-Research and Education Networking Association Networking Conference (TERENA 2003), May 2003.
• 24: [24] W. Washington and C. Parkinson, An Introduction to Three-Dimensional Climate Modeling, University Science Books, 1991.