High performance and scalable I/O virtualization via self-virtualized devices

ABSTRACT

While industry is making rapid advances in system virtualization, for server consolidation and for improving system maintenance and management, it has not yet become clear how virtualization can contribute to the performance of high end systems. In this context, this paper addresses a key issue in system virtualization - how to efficiently virtualize I/O subsystems and peripheral devices. We have developed a novel approach to I/O virtualization, termed self-virtualized devices, which improves I/O performance by off loading select virtualization functionality onto the device. This permits guest virtual machines to more efficiently (i.e., with less overhead and reduced latency) interact with the virtualized device. The concrete instance of such a device developed and evaluated in this paper is a self-virtualized network interface (SV-NIC), targeting the high end NICs used in thehigh performance domain. The SV-NIC (1) provides virtual interfaces (VIFs) to guest virtual machines for an underlying physical device, the network interface, (2) manages the wayin which the device's physical resources are used by guest operating systems, and (3) provides high performance, low overhead network access to guest domains. Experimental results are attained in a prototyping environment using an IXP 2400-based ethernet board as a programmable network device. The SV-NIC scales to large numbers of VIFs and guests, and offers VIFs with 77% higher throughput and 53% less latency compared to the current standard virtualized device implementations on hyper visor-based platforms.

REFERENCES

• 1: ENP-2611 Data Sheet. http://www.radisys.com/files/ENP-2611_07-1236-05_0504_datasheet.pdf.
• 2: The geneseo proposal to enhance pci-express. http://www.intel.com/pressroom/archive/releases/20060927comp_a.htm.
• 3: Hypertransport interconnect. http://www.hypertransport.org/.
• 4: Infiniband interface. http://www.infinibandta.org/home.
• 5: Intel 21555 Non-transparent PCI-to-PCI Bridge. http://www.intel.com/design/bridge/21555.htm.
• 6: Intel Virtualization Technology Specification for the IA-32 Intel Architecture. ftp://download.intel.com/technology/computing/vptech/C97063-002.pdf.
• 7: Iperf. http://dast.nlanr.net/projects/Iperf.
• 8: OSA-Express for IBM eserver zSeries and S/390. www.ibm.com/servers/eserver/zseries/library/specsheets/pdf/g2219110.pdf.
• 9: Tcpdump/libpcap. http://www.tcpdump.org/.
• 10: The Cell Architecture. http://www.research.ibm.com/cell/.
• 11: The VMWare ESX Server.http://www.vmware.com/products/esx/.
• 12: Intel IXP2400 Network Processor: Hardware Reference Manual, October 2003.
• 13: R. Alverson. Red storm: A 10,000 node system with reliable, high bandwidth, low latency interconnect. In Proc. of Hot Chips 13, 2003.
• 14: Herbert Bos , Willem de Bruijn , Mihai Cristea , Trung Nguyen , Georgios Portokalidis, FFPF: fairly fast packet filters, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.24-24, December 06-08, 2004, San Francisco, CA
• 15: Tim Brecht , G. (John) Janakiraman , Brian Lynn , Vikram Saletore , Yoshio Turner, Evaluating network processing efficiency with processor partitioning and asynchronous I/O, Proceedings of the ACM SIGOPS/EuroSys European Conference on Computer Systems 2006, April 18-21, 2006, Leuven, Belgium
• 16: Patrick G. Bridges , Arthur B. MacCabe , Orran Krieger, System software for high end computing, ACM SIGOPS Operating Systems Review, v.40 n.2, April 2006 [doi>10.1145/1131322.1131327]
• 17: Sayantan Chakravorty , Celso L. Mendes , Laxmikant V. Kalé , Terry Jones , Andrew Tauferner , Todd Inglett , José Moreira, HPC-Colony: services and interfaces for very large systems, ACM SIGOPS Operating Systems Review, v.40 n.2, April 2006 [doi>10.1145/1131322.1131334]
• 18: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 19: Ada Gavrilovska , Kenneth Mackenzie , Karsten Schwan , Austen McDonald, Stream Handlers: Application-Specific Message Services on Attached Network Processors, Proceedings of the 10th Symposium on High Performance Interconnects HOT Interconnects (HotI'02), p.115, August 21-23, 2002
• 20: F. T. Hady et al. Platform Level Support for High Throughput Edge Applications: The Twin Cities Prototype. IEEE Network, July/August 2003.
• 21: Himanshu Raj and Ivan Ganev and Karsten Schwan and Jimi Xenidis. Self-Virtualized I/O: High Performance, Scalable I/O Virtualization in Multi-core Systems. Technical Report GIT-CERCS-06-02, CERCS, Georgia Tech, 2006.
• 22: J. Kong, I. Ganev, K. Schwan, and P. Widener. Cameracast: Flexible access to remote video sensors. In Multimedia Computing and Networking (MMCN'07), San Jose, CA, USA, Jan. 2007.
• 23: Jiuxing Liu , Wei Huang , Bulent Abali , Dhabaleswar K. Panda, High performance VMM-bypass I/O in virtual machines, Proceedings of the Annual Technical Conference on USENIX'06 Annual Technical Conference, p.3-3, May 30-June 03, 2006, Boston, MA
• 24: Derek McAuley , Rolf Neugebauer, A case for virtual channel processors, Proceedings of the ACM SIGCOMM workshop on Network-I/O convergence: experience, lessons, implications, August 25-27, 2003, Karlsruhe, Germany [doi>10.1145/944747.944758]
• 25: Aravind Menon , Jose Renato Santos , Yoshio Turner , G. (John) Janakiraman , Willy Zwaenepoel, Diagnosing performance overheads in the xen virtual machine environment, Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments, June 11-12, 2005, Chicago, IL, USA [doi>10.1145/1064979.1064984]
• 26: Scott Pakin , Mario Lauria , Andrew Chien, High performance messaging on workstations: Illinois fast messages (FM) for Myrinet, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.55-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224360]
• 27: I. Pratt et al. Xen 3.0 and the Art of Virtualization. In Proc. of the Ottawa Linux Symposium, 2005.
• 28: I. Pratt and K. Fraser. Arsenic: A User Accessible Gigabit Network Interface. In Proc. of INFOCOM, 2001.
• 29: G. Regnier et al. ETA: Experience with an Intel Xeon Processor as a Packet Processing Engine. IEEE Micro, 24(1):24--31, 2004.
• 30: M. Rosu, K. Schwan, and R. Fujimoto. Supporting Parallel Applications on Clusters of Workstations: The Virtual Communication Machine-based Architecture. In Proc. of Cluster Computing, 1998.
• 31: Jamal Hadi Salim , Robert Olsson , Alexey Kuznetsov, Beyond softnet, Proceedings of the 5th annual conference on Linux Showcase & Conference, p.18-18, November 05-10, 2001, Oakland, California
• 32: Volkmar Uhlig , Joshua LeVasseur , Espen Skoglund , Uwe Dannowski, Towards scalable multiprocessor virtual machines, Proceedings of the 3rd conference on Virtual Machine Research And Technology Symposium, p.4-4, May 06-07, 2004, San Jose, California
• 33: T. von Eicken , A. Basu , V. Buch , W. Vogels, U-Net: a user-level network interface for parallel and distributed computing (includes URL), Proceedings of the fifteenth ACM symposium on Operating systems principles, p.40-53, December 03-06, 1995, Copper Mountain, Colorado, United States
• 34: Andrew Warfield , Steven Hand , Keir Fraser , Tim Deegan, Facilitating the development of soft devices, Proceedings of the USENIX Annual Technical Conference 2005 on USENIX Annual Technical Conference, p.22-22, April 10-15, 2005, Anaheim, CA
• 35: Richard West , Yuting Zhang , Karsten Schwan , Christian Poellabauer, Dynamic Window-Constrained Scheduling of Real-Time Streams in Media Servers, IEEE Transactions on Computers, v.53 n.6, p.744-759, June 2004 [doi>10.1109/TC.2004.10]
• 36: P. Willmann et al. Concurrent Direct Network Access for Virtual Machine Monitors. In Proc. of HPCA, 2007.
• 37: Paul Willmann , Hyong-youb Kim , Scott Rixner , Vijay S. Pai, An Efficient Programmable 10 Gigabit Ethernet Network Interface Card, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.96-107, February 12-16, 2005 [doi>10.1109/HPCA.2005.6]
• 38: Hyong-youb Kim , Scott Rixner, TCP offload through connection handoff, Proceedings of the ACM SIGOPS/EuroSys European Conference on Computer Systems 2006, April 18-21, 2006, Leuven, Belgium
• 39: L. Youseff, R. Wolski, B. Gorda, and C. Krintz. Evaluating the Performance Impact of Xen on MPI and Process Execution For HPC Systems. In Proc. of International Workshop on Virtualization Technologies in Distributed Computing (VTDC), 2006.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Performance attributes

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management
          Subjects: Network communication

General Terms:
Design, Experimentation, Management, Measurement, Performance

Keywords:
virtual devices, virtualization