Achieving 10 Gb/s using safe and transparent network interface virtualization

ABSTRACT

This paper presents mechanisms and optimizations to reduce the overhead of network interface virtualization when using the driver domain I/O virtualization model. The driver domain model provides benefits such as support for legacy device drivers and fault isolation. However, the processing overheads incurred in the driver domain to achieve these benefits limit overall I/O performance. This paper demonstrates the effectiveness of two approaches to reduce driver domain overheads. First, Xen is modified to support multi-queue network interfaces to eliminate the software overheads of packet demultiplexing and copying. Second, a grant reuse mechanism is developed to reduce memory protection overheads. These mechanisms shift the bottleneck from the driver domain to the guest domains, improving scalability and enabling significantly higher data rates. This paper also presents and evaluates a series of optimizations that substantially reduce the I/O virtualization overheads in the guest domain. In combination, these mechanisms and optimizations increase the maximum throughput achieved by guest domains from 2.9Gb/s to full 10 Gigabit Ethernet link rates.

REFERENCES

• 1: Darren Abramson, Jeff Jackson, Sridhar Muthrasanallur, Gil Neiger, Greg Regnier, Rajesh Sankaran, Ioannis Schoinas, Rich Uhlig, Balaji Vembu, and John Wiegert. Intel virtualization technology for directed I/O. Intel Technology Journal, 10 (3), August 2006.
• 2: Advanced Micro Devices, Inc. IOMMU architectural specification. www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf, Feb 2007. PID 34434 Rev 1.20.
• 3: Shefali Chinni and Radhakrishna Hiremane. Virtual machine device queues. Intel Corp. White Paper, 2007.
• 4: Andy Chou , Junfeng Yang , Benjamin Chelf , Seth Hallem , Dawson Engler, An empirical study of operating systems errors, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 5: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 6: Intel Corp. Intel 82598 10 GbE ethernet controller open source datasheet, 2008. Revision 2.5.
• 7: Scott Devine, Edouard Bugnion, and Mendel Rosenblum. Virtualization system including a virtual machine monitor for a computer with a segmented architecture. VMware US Patent 6397242, Oct 1998.
• 8: Keir Fraser, Steve Hand, Rolf Neugebauer, Ian Pratt, Andrew Warfield, and Mark Williams. Safe hardware access with the Xen virtual machine monitor. In 1st Workshop on Operating System and Architectural Support for the on demand IT InfraStructure (OASIS), October 2004.
• 9: Leonid Grossman. Large Receive Offload implementation in Neterion 10GbE Ethernet driver. In Ottawa Linux Symposium (OLS), 2005.
• 10: Ram Huggahalli , Ravi Iyer , Scott Tetrick, Direct Cache Access for High Bandwidth Network I/O, Proceedings of the 32nd annual international symposium on Computer Architecture, p.50-59, June 04-08, 2005
• 11: Joshua LeVasseur , Volkmar Uhlig , Jan Stoess , Stefan Götz, Unmodified device driver reuse and improved system dependability via virtual machines, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.2-2, December 06-08, 2004, San Francisco, CA
• 12: Kieran Mansley, Greg Law, David Riddoch, Guido Barzini, Neil Turton, and Steven Pope. Getting 10 Gb/s from Xen: Safe and fast device access from unprivileged domains. In Euro-Par 2007 Workshops: Parallel Processing, 2007.
• 13: Aravind Menon , Willy Zwaenepoel, Optimizing TCP receive performance, USENIX 2008 Annual Technical Conference on Annual Technical Conference, p.85-98, June 22-27, 2008, Boston, Massachusetts
• 14: Aravind Menon , Jose Renato Santos , Yoshio Turner , G. (John) Janakiraman , Willy Zwaenepoel, Diagnosing performance overheads in the xen virtual machine environment, Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments, June 11-12, 2005, Chicago, IL, USA [doi>10.1145/1064979.1064984]
• 15: Aravind Menon , Alan L. Cox , Willy Zwaenepoel, Optimizing network virtualization in Xen, Proceedings of the annual conference on USENIX '06 Annual Technical Conference, p.2-2, May 30-June 03, 2006, Boston, MA
• 16: Microsoft. Hyper-V architecture. http://msdn.microsoft.com/en-us/library/cc768520.aspx.
• 17: Neterion. Product brief: Neterion X3100 series. http://www.neterion.com/products/pdfs/X3100ProductBrief.pdf, 2008.
• 18: }pciiovPCI SIG. I/O virtualization. www.pcisig.com/specifications/iov/.
• 19: Himanshu Raj , Karsten Schwan, High performance and scalable I/O virtualization via self-virtualized devices, Proceedings of the 16th international symposium on High performance distributed computing, June 25-29, 2007, Monterey, California, USA [doi>10.1145/1272366.1272390]
• 20: Scot Rixner, Network Virtualization: Breaking the Performance Barrier, Queue, v.6 n.1, January/February 2008 [doi>10.1145/1348583.1348592]
• 21: Jose Renato Santos , Yoshio Turner , G. Janakiraman , Ian Pratt, Bridging the gap between software and hardware techniques for I/O virtualization, USENIX 2008 Annual Technical Conference on Annual Technical Conference, p.29-42, June 22-27, 2008, Boston, Massachusetts
• 22: Constantine P. Sapuntzakis , Ramesh Chandra , Ben Pfaff , Jim Chow , Monica S. Lam , Mendel Rosenblum, Optimizing the migration of virtual computers, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060324]
• 23: Michael M. Swift , Brian N. Bershad , Henry M. Levy, Improving the reliability of commodity operating systems, ACM Transactions on Computer Systems (TOCS), v.23 n.1, p.77-110, February 2005 [doi>10.1145/1047915.1047919]
• 24: Paul Willmann , Jeffrey Shafer , David Carr , Aravind Menon , Scott Rixner , Alan L. Cox , Willy Zwaenepoel, Concurrent Direct Network Access for Virtual Machine Monitors, Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture, p.306-317, February 10-14, 2007 [doi>10.1109/HPCA.2007.346208]
• 25: Paul Willmann , Scott Rixner , Alan L. Cox, Protection strategies for direct access to virtualized I/O devices, USENIX 2008 Annual Technical Conference on Annual Technical Conference, p.15-28, June 22-27, 2008, Boston, Massachusetts
• 26: Edwin Zhai, Gregory D. Cummings, and Yaozu Dong. Live migration with pass-through device for linux vm. In Ottawa Linux Symposium (OLS), 2008.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Measurements

General Terms:
Design, Measurement, Performance

Keywords:
device drivers, i/o, networking, performance analysis, virtual machine, virtualization