Wide-area performance profiling of 10GigE and InfiniBand technologies

ABSTRACT

For wide-area high-performance applications, light-paths provide 10Gbps connectivity, and multi-core hosts with PCI-Express can drive such data rates. However, sustaining such end-to-end application throughputs across connections of thousands of miles remains challenging, and the current performance studies of such solutions are very limited. We present an experimental study of two solutions to achieve such throughputs based on: (a) 10Gbps Ethernet with TCP/IP transport protocols, and (b) InfiniBand and its wide-area extensions. For both, we generate performance profiles over 10Gbps connections of lengths up to 8600 miles, and discuss the components, complexity, and limitations of sustaining such throughputs, using different connections and host configurations. Our results indicate that IB solution is better suited for applications with a single large flow, and 10GigE solution is better for those with multiple competing flows.

REFERENCES

• 1: "Internet2," http://www.internet2.edu.
• 2: "Energy Sciences Network," http://www.es.net.
• 3: N. S. V. Rao, W. R. Wing, S. M. Carter, and Q. Wu, "Ultrascience net: Network testbed for large-scale science applications," IEEE Communications Magazine, 2005, in press, expanded version available at www.csm.ornl.gov/ultranet.
• 4: End-To-End Provisioned Optical Network Testbed for Large-Scale eScience Application, http://www.ece.virginia.edu/mv/html-files/ein-home.html.
• 5: L. Xu, K. Harfoush, and I. Rhee, "Binary increase congestion control (bic) for fast long-distance networks," in INFOCOM, 2004.
• 6: I. Rhee and L. Xu, "Cubic: A new tcp-friendly high-speed tcp variant," in Proceedings of the Third International Workshop on Protocols for Fast Long-Distance Networks, 2005.
• 7: S. Floyd, HighSpeed TCP for Large Congestion Windows, RFC Editor, 2003
• 8: R. Shorten and D. Leith, "H-TCP: TCP for high-speed and longdistance networks," in Proceedings of the Third International Workshop on Protocols for Fast Long-Distance Networks, 2004.
• 9: Tom Kelly, Scalable TCP: improving performance in highspeed wide area networks, ACM SIGCOMM Computer Communication Review, v.33 n.2, April 2003 [doi>10.1145/956981.956989]
• 10: Mahbub Hassan , Raj Jain, High Performance TCP/IP Networking, Prentice-Hall, Inc., Upper Saddle River, NJ, 2003
• 11: Dina Katabi , Mark Handley , Charlie Rohrs, Congestion control for high bandwidth-delay product networks, Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications, August 19-23, 2002, Pittsburgh, Pennsylvania, USA
• 12: "bbcp," http://http://www.slac.stanford.edu/abh/bbcp/.
• 13: "Gt 4.0 gridftp," http://www.globus.org.
• 14: Obsidian Resarch Corporation, http://www.obsidianresearch.com/.
• 15: Network Equipment Technologies, http://www.net.com.
• 16: N. S. V. Rao, W. R. W. S. E. Hicks, S. W. Poole, F. A. Denap, S. M. Carter, and Q. Wu, "Ultrascience net: High-performance network research test-bed," in International Symposium on on Computer and Sensor Network Systems, 2008.
• 17: Weikuan Yu , Nageswara S. V. Rao , Jeffrey S. Vetter, Experimental Analysis of InfiniBand Transport Services on WAN, Proceedings of the 2008 International Conference on Networking, Architecture, and Storage, p.233-240, June 12-14, 2008 [doi>10.1109/NAS.2008.65]
• 18: E. He, P. V. Primet, and M. Welzl, "A survey of tranport protocols other than "standard" TCP," global Grid Form Report GFD-I.055.
• 19: Infiniband Trade Association, http://www.infinibandta.org.
• 20: OpenFabrics Alliance, http://www.openfabrics.org.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.5 Local and Wide-Area Networks

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.2 Network Protocols
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Algorithms, Design, Experimentation, Measurement, Performance

Keywords:
10GigE LAN/WAN-PHY, InfiniBand, TCP performance, high-performance networking