miNI: reducing network interface memory requirements with dynamic handle lookup

ABSTRACT

Recent work in low-latency, high-bandwidth communication systems has resulted in building user--level Network Interface Controllers (NICs) and communication abstractions that support direct access from the NIC to applications virtual memory to avoid both data copies and operating system intervention. Such mechanisms require the ability to directly manipulate user--level communication buffers for delivering data and achieving protection. To provide such abilities, NICs must maintain appropriate translation data structures. Most user--level NICs manage these data structures statically, which results both in high memory requirements for the NIC and limitations on the total size and number of communication buffers that a NIC can handle.In this paper, we categorize the types of data structures used by NICs and propose dynamic handle lookup as a mechanism to manage such data structures dynamically. We implement our approach in a modern, user--level communication system and evaluate our system, miNL, with both micro-benchmarks and real applications. We also study the impact of various cache parameters on system performance. We find that, with appropriate cache tuning, our approach reduces the amount of NIC memory required in our system by a factor of two for the total NIC memory and by more than 80% for the lookup data structures. Moreover, by pinning physical memory automatically and on demand, our approach eliminates the limitations and complexities imposed by static memory pinning that is used in most user--level communication systems. Our approach increases execution time by at most 3% for all but one applications we examine.

REFERENCES

• 1: R. Azimi and A. Bilas. Evaluating the performance impact of dynamic handle lookup in modern network interfaces. In Proc. of the 2nd Annual Workshop on Novel Uses of System Area Networks SAN-2, Anaheim, 2003.
• 2: Mohammad Banikazemi , Bülent Abali , Dhabaleswar K. Panda, Comparison and Evaluation of Design Choices for Implementing the Virtual Interface Architecture (VIA), Proceedings of the 4th International Workshop on Network-Based Parallel Computing: Communication, Architecture, and Applications, p.145-161, January 08, 2000
• 3: T. von Eicken , A. Basu , V. Buch , W. Vogels, U-Net: a user-level network interface for parallel and distributed computing (includes URL), Proceedings of the fifteenth ACM symposium on Operating systems principles, p.40-53, December 03-06, 1995, Copper Mountain, Colorado, United States
• 4: A. Basu, M. Welsh, and T. von Eicken. Incorporating memory management into user-level network interfaces. http:/ /www2.cs.cornell.edu/U-Net/papers/unetmm.pdf, 1996.
• 5: Angelos Bilas , Cheng Liao , Jaswinder Pal Singh, Using network interface support to avoid asynchronous protocol processing in shared virtual memory systems, Proceedings of the 26th annual international symposium on Computer architecture, p.282-293, May 01-04, 1999, Atlanta, Georgia, United States
• 6: M. A. Blumrich , K. Li , R. Alpert , C. Dubnicki , E. W. Felten , J. Sandberg, Virtual memory mapped network interface for the SHRIMP multicomputer, Proceedings of the 21ST annual international symposium on Computer architecture, p.142-153, April 18-21, 1994, Chicago, Illinois, United States
• 7: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 8: P. Buonadonna, J. Coates, S. Low, and D. Culler. Millennium sort: A cluster-based application for windows nt using dcom, river primitives and the virtual interface architecture. In Proc. of the 3rd USENIX Windows NT Symposium, Seattle, July 1999.
• 9: Philip Buonadonna , Andrew Geweke , David Culler, An implementation and analysis of the virtual interface architecture, Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), p.1-15, November 07-13, 1998, San Jose, CA
• 10: Yuqun Chen , Angelos Bilas , Stefanos N. Damianakis , Cezary Dubnicki , Kai Li, UTLB: a mechanism for address translation on network interfaces, Proceedings of the eighth international conference on Architectural support for programming languages and operating systems, p.193-204, October 02-07, 1998, San Jose, California, United States
• 11: C. Dubnicki, A. Bilas, Y. Chen, S. Damianakis, and K. Li. VMMC-2: efficient support for reliable, connection-oriented communication. In Proc. of The 1997 IEEE Symposium on High Performance Interconnects (HOT Interconnects Aug. 1997. A short version of this appears in IEEE Micro, Jan/Feb, 1998.
• 12: D. Dunning and G. Regnier. The Virtual Interface Architecture. In Proc. of The 1997 IEEE Symposium on High Performance Interconnects (HOT Interconnects V), Aug. 1997.
• 13: Emulex. Gn9000/vi: 1gb/s vi/ip pci host bus adapter. http://www.emulex.com/products/viip/index.html.
• 14: Giganet. Giganet cLAN family of products. http://www.emulex.com/products.html, 2001.
• 15: R. Gillett , M. Collins , D. Pimm, Overview of memory channel network for PCI, Proceedings of the 41st IEEE International Computer Conference, p.244, February 25-28, 1996
• 16: David A. Patterson , John L. Hennessy, Computer architecture: a quantitative approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1990
• 17: M. Inc. Myrinet product list and prices. http://www.myri.com/myrinet/product_list.html, 2002.
• 18: InfiniBand Trade Association. Infiniband architecture specification, ver. 1.0. http://www.infinibandta. org, Oct. 2000.
• 19: Internet Engineering Task Force (IETF). iSCSI, version 08. In IP Storage (IPS), Internet Draft, Document: draft-ietf-ips-iscsi-08.txt, Sept. 2001.
• 20: Peter Jamieson , Angelos Bilas, CableS: Thread Control and Memory Management Extensions for Shared Virtual Memory Clusters, Proceedings of the 8th International Symposium on High-Performance Computer Architecture, p.263, February 02-06, 2002
• 21: Alan M. Mainwaring , David E. Culler, Design challenges of virtual networks: fast, general-purpose communication, Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming, p.119-130, May 04-06, 1999, Atlanta, Georgia, United States
• 22: National Energy Research Scientific Computing Center. M--via: A high performance modular via for linux. http://www.nersc.gov/research/FTG/via, 1998.
• 23: I. Schoinas , M. Hill, Address Translation Mechanisms In Network Interfaces, Proceedings of the 4th International Symposium on High-Performance Computer Architecture, p.219, January 31-February 04, 1998
• 24: Jeffrey Tang , Angelos Bilas, Tolerating Network Failures in System Area Networks, Proceedings of the 2002 International Conference on Parallel Processing (ICPP'02), p.121, August 18-21, 2002
• 25: Steven Cameron Woo , Moriyoshi Ohara , Evan Torrie , Jaswinder Pal Singh , Anoop Gupta, The SPLASH-2 programs: characterization and methodological considerations, Proceedings of the 22nd annual international symposium on Computer architecture, p.24-36, June 22-24, 1995, S. Margherita Ligure, Italy
• 26: Yuanyuan Zhou , Angelos Bilas , Suresh Jagannathan , Cezary Dubnicki , James F. Philbin , Kai Li, Experiences with VI communication for database storage, Proceedings of the 29th annual international symposium on Computer architecture, p.257, May 25-29, 2002, Anchorage, Alaska

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies

General Terms:
Design, Measurement, Performance

Keywords:
parallel architectures, system area networks