Towards 100 gbit/s ethernet: multicore-based parallel communication protocol design

ABSTRACT

Ethernet line rates are projected to reach 100 Gbits/s by as soon as 2010. While in principle suitable for high performance clustered and parallel applications, Ethernet requires matching improvements in the system software stack. In this paper we address several sources of CPU and memory system overhead in the I/O path at line rates reaching 80 Gbits/s (bi-directional), using multiple 10 Gbit/s links per system node. Key contributions of our work are the design of a parallel high-performance communication protocol that uses context-independent page-remapping to (a) reduce packet processing overheads; (b) reduce thread management and synchronization overheads; and (c) address affinity issues in NUMA multicore CPUs. Our design result in the full 40 Gbits/s of available one-way Ethernet bandwidth and in 57.6 Gbits/s (72%) of the 80 Gbits/s maximum bidirectional throughput (limited only by the memory system), while leaving ample CPU cycles for application processing.

REFERENCES

• 1: Soichiro Araki , Angelos Bilas , Cezary Dubnicki , Jan Edler , Koichi Konishi , James Philbin, User-space communication: a quantitative study, Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), p.1-16, November 07-13, 1998, San Jose, CA
• 2: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 3: J. Brustoloni. Interoperation of Copy Avoidance in Network and File I/O. In 18th Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE INFOCOM), New York, Mar. 1999.
• 4: F. Chiussi, D. Khotimsky, and S. Krishnan. Generalized Inverse Multiplexing of Switched ATM Connections. In Global Telecommunications Conference (IEEE GLOBECOM), Sydney, Australia, Nov. 1998.
• 5: Salvador Coll , Eitan Frachtenberg , Fabrizio Petrini , Adolfy Hoisie , Leonid Gurvits, Using Multirail Networks in High-Performance Clusters, Proceedings of the 3rd IEEE International Conference on Cluster Computing, p.15, October 08-11, 2001
• 6: Peter Druschel , Larry L. Peterson, Fbufs: a high-bandwidth cross-domain transfer facility, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.189-202, December 05-08, 1993, Asheville, North Carolina, United States
• 7: C. Dubnicki, A. Bilas, Y. Chen, S. Damianakis, and K. Li. VMMC-2: Efficient Support for Reliable, Connection-Oriented Communication. In Hot Interconnects V, Stanford, California, Aug. 1997.
• 8: J. Duncanson. Inverse multiplexing. IEEE Communications Magazine, 32(4):34--41, Apr. 1994.
• 9: B. Goglin. Improving Message Passing over Ethernet with I/OAT Copy Offload in Open-MX. In IEEE Cluster Computing, Tsukuba, Japan, Oct. 2008.
• 10: IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force. http://www.ieee802.org/3/ba/.
• 11: An Infiniband Technology Overview. Infiniband Trade Association, http://www.infinibandta.org/ibta.
• 12: S. Karlsson, S. Passas, G. Kotsis, and A. Bilas. MultiEdge: An Edge-based Communication Subsystem for Scalable Commodity Servers. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, California, Mar. 2007.
• 13: Jiuxing Liu , Balasubramanian Chandrasekaran , Weikuan Yu , Jiesheng Wu , Darius Buntinas , Sushmitha Kini , Dhabaleswar K. Panda , Pete Wyckoff, Microbenchmark Performance Comparison of High-Speed Cluster Interconnects, IEEE Micro, v.24 n.1, p.42-51, January 2004 [doi>10.1109/MM.2004.1268994]
• 14: Kostas Magoutis , Salimah Addetia , Alexandra Fedorova , Margo I. Seltzer, Making the Most Out of Direct-Access Network Attached Storage, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA
• 15: Jeffrey C. Mogul, TCP offload is a dumb idea whose time has come, Proceedings of the 9th conference on Hot Topics in Operating Systems, p.5-5, May 18-21, 2003, Lihue, Hawaii
• 16: Scott Pakin , Vijay Karamcheti , Andrew A. Chien, Fast Messages: Efficient, Portable Communication for Workstation Clusters and MPPs, IEEE Parallel & Distributed Technology: Systems & Technology, v.5 n.2, p.60-73, April 1997
• 17: S. Passas, G. Kotsis, S. Karlsson, and A. Bilas. Expoiting Spatial Parallelism in Ethernet-based Cluster Interconnects. In Workshop on Communication Architectures for Clusters (CAC). Held in Conjuction with IPDPS, Miami, Florida, Apr. 2008.
• 18: J. Pinkerton. The Case for RDMA, 2002. RDMA Consortium, http://www.rdmaconsortium.org/home/ The Case for RDMA02053.pdf.
• 19: Stream Control Transmission Protocol (SCTP). http://www.ietf.org/rfc/rfc2960.txt.
• 20: Piyush Shivam , Pete Wyckoff , Dhabaleswar Panda, EMP: zero-copy OS-bypass NIC-driven gigabit ethernet message passing, Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM), p.57-57, November 10-16, 2001, Denver, Colorado [doi>10.1145/582034.582091]
• 21: W. Richard Stevens , Gary R. Wright, TCP/IP illustrated (vol. 2): the implementation, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 22: Shinji Sumimoto , Kazuichi Ooe , Kouichi Kumon , Taisuke Boku , Mitsuhisa Sato , Akira Ukawa, A scalable communication layer for multi-dimensional hyper crossbar network using multiple gigabit ethernet, Proceedings of the 20th annual international conference on Supercomputing, June 28-July 01, 2006, Cairns, Queensland, Australia [doi>10.1145/1183401.1183418]
• 23: T. von Eicken , A. Basu , V. Buch , W. Vogels, U-Net: a user-level network interface for parallel and distributed computing (includes URL), Proceedings of the fifteenth ACM symposium on Operating systems principles, p.40-53, December 03-06, 1995, Copper Mountain, Colorado, United States
• 24: Matt Welsh , Anindya Basu , Thorsten von Eicken, ATM and Fast Ethernet Network Interfaces for User-level Communication, Proceedings of the 3rd IEEE Symposium on High-Performance Computer Architecture, p.332, February 01-05, 1997
• 25: Paul Willmann , Hyong-youb Kim , Scott Rixner , Vijay S. Pai, An Efficient Programmable 10 Gigabit Ethernet Network Interface Card, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.96-107, February 12-16, 2005 [doi>10.1109/HPCA.2005.6]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Network communications

General Terms:
Design, Performance

Keywords:
100 gbit/s ethernet, communication protocol design, multicore cpus, performance evaluation