Dynamic heterogeneity and the need for multicore virtualization

ABSTRACT

As the computing industry enters the multicore era, exponential growth in the number of transistors on a chip continues to present challenges and opportunities for computer architects and system designers. We examine one emerging issue in particular: that of dynamic heterogeneity, which can arise, even among physically homogeneous cores, from changing reliability, power, or thermal conditions, different cache and TLB contents, or changing resource configurations. This heterogeneity results in a constantly varying pool of hardware resources, which greatly complicates software's traditional task of assigning computation to cores. In part to address dynamic heterogeneity, we argue that hardware should take a more active role in the management of its computation resources. We propose hardware techniques to virtualize the cores of a multicore processor, allowing hardware to flexibly reassign the virtual processors that are exposed, even to a single operating system, to any subset of the physical cores. We show that multicore virtualization operates with minimal overhead, and that it enables several novel resource management applications for improving both performance and reliability.

REFERENCES

• 1: Nidhi Aggarwal , Parthasarathy Ranganathan , Norman P. Jouppi , James E. Smith, Configurable isolation: building high availability systems with commodity multi-core processors, Proceedings of the 34th annual international symposium on Computer architecture, June 09-13, 2007, San Diego, California, USA
• 2: W. J. Armstrong , R. L. Arndt , D. C. Boutcher , R. G. Kovacs , D. Larson , K. A. Lucke , N. Nayar , R. C. Swanberg, Advanced virtualization capabilities of POWER5 systems, IBM Journal of Research and Development, v.49 n.4/5, p.523-532, July 2005
• 3: David Bernick , Bill Bruckert , Paul Del Vigna , David Garcia , Robert Jardine , Jim Klecka , Jim Smullen, NonStop® Advanced Architecture, Proceedings of the 2005 International Conference on Dependable Systems and Networks, p.12-21, June 28-July 01, 2005 [doi>10.1109/DSN.2005.70]
• 4: Shekhar Borkar , Tanay Karnik , Siva Narendra , Jim Tschanz , Ali Keshavarzi , Vivek De, Parameter variations and impact on circuits and microarchitecture, Proceedings of the 40th conference on Design automation, June 02-06, 2003, Anaheim, CA, USA [doi>10.1145/775832.775920]
• 5: K. Bowman, S. Duvall, and J. Meindl. Impact of die-to-die and within-die parameter fluctuations on the maximum clock frequency distribution for gigascale integration. J. of Solid-State Circuits, 37(2):183--190, Feb 2002.
• 6: David Brooks , Margaret Martonosi, Dynamic Thermal Management for High-Performance Microprocessors, Proceedings of the 7th International Symposium on High-Performance Computer Architecture, p.171, January 20-24, 2001
• 7: Koushik Chakraborty , Philip M. Wells , Gurindar S. Sohi, Computation spreading: employing hardware migration to specialize CMP cores on-the-fly, Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, October 21-25, 2006, San Jose, California, USA
• 8: K. Chakraborty, P.M. Wells, and G.S. Sohi. A case for an over-provisioned multicore system: Energy efficient processing of multithreaded programs. Technical Report CS-TR-2007-1607, University of Wisconsin-Madison, Aug 2007.
• 9: Cristian Constantinescu, Trends and Challenges in VLSI Circuit Reliability, IEEE Micro, v.23 n.4, p.14-19, July 2003 [doi>10.1109/MM.2003.1225959]
• 10: J. Dorsey, S. Searles, M. Ciraula, S. Johnson, N. Bujanos, D. Wu, M. Braganza, S. Meyers, E. Fang, and R. Kumar. An integrated quad-core Opteron processor. pages 102--103, Feb. 2007.
• 11: Dan Ernst , Nam Sung Kim , Shidhartha Das , Sanjay Pant , Rajeev Rao , Toan Pham , Conrad Ziesler , David Blaauw , Todd Austin , Krisztian Flautner , Trevor Mudge, Razor: A Low-Power Pipeline Based on Circuit-Level Timing Speculation, Proceedings of the 36th annual IEEE/ACM International Symposium on Microarchitecture, p.7, December 03-05, 2003
• 12: Kinshuk Govil , Dan Teodosiu , Yongqiang Huang , Mendel Rosenblum, Cellular Disco: resource management using virtual clusters on shared-memory multiprocessors, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.154-169, December 12-15, 1999, Charleston, South Carolina, United States
• 13: M. Gschwind, P. Hofstee, B. Flachs, M. Hopkins, Y. Watanabe, and T. Yamazaki. A novel SIMD architecture for the Cell heterogeneous chip-multiprocessor. In Proc. of 17th Hot Chips, 2005.
• 14: S.H. Gunther, F. Binns, D.M. Carmean, and J.C. Hall. Managing the impact of increasing microprocessor power consumption. Intel Tech. J., Q1, 2001.
• 15: Samuel Norman Hamilton , Alex Orailoglu, Transient and Intermittent Fault Recovery without Rollback, Proceedings of the 13th International Symposium on Defect and Fault-Tolerance in VLSI Systems, p.252-260, November 02-04, 1998
• 16: Stavros Harizopoulos , Anastassia Ailamaki, Steps towards cache-resident transaction processing, Proceedings of the Thirtieth international conference on Very large data bases, p.660-671, August 31-September 03, 2004, Toronto, Canada
• 17: Engin Ipek , Meyrem Kirman , Nevin Kirman , Jose F. Martinez, Core fusion: accommodating software diversity in chip multiprocessors, Proceedings of the 34th annual international symposium on Computer architecture, June 09-13, 2007, San Diego, California, USA
• 18: Rakesh Kumar , Dean M. Tullsen , Parthasarathy Ranganathan , Norman P. Jouppi , Keith I. Farkas, Single-ISA Heterogeneous Multi-Core Architectures for Multithreaded Workload Performance, Proceedings of the 31st annual international symposium on Computer architecture, p.64, June 19-23, 2004, München, Germany
• 19: Christopher LaFrieda , Engin Ipek , Jose F. Martinez , Rajit Manohar, Utilizing Dynamically Coupled Cores to Form a Resilient Chip Multiprocessor, Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, p.317-326, June 25-28, 2007 [doi>10.1109/DSN.2007.100]
• 20: James R. Larus , Michael Parkes, Using Cohort-Scheduling to Enhance Server Performance, Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference, p.103-114, June 10-15, 2002
• 21: James Laudon, Performance/Watt: the new server focus, ACM SIGARCH Computer Architecture News, v.33 n.4, November 2005 [doi>10.1145/1105734.1105737]
• 22: Tao Li , Lizy Kurian John , Anand Sivasubramaniam , N. Vijaykrishnan , Juan Rubio, Understanding and improving operating system effects in control flow prediction, Proceedings of the 10th international conference on Architectural support for programming languages and operating systems, October 05-09, 2002, San Jose, California
• 23: Peter S. Magnusson , Magnus Christensson , Jesper Eskilson , Daniel Forsgren , Gustav Hållberg , Johan Högberg , Fredrik Larsson , Andreas Moestedt , Bengt Werner, Simics: A Full System Simulation Platform, Computer, v.35 n.2, p.50-58, February 2002 [doi>10.1109/2.982916]
• 24: Dennis McEvoy, The architecture of Tandem's NonStop system, Proceedings of the ACM '81 conference, p.245, January 1981 [doi>10.1145/800175.809886]
• 25: D. Nellans, R. Balasubramonian, and E. Brunvand. A case for increased operating system support in chip multi-processors. In Proc. of 2nd IBM Watson P=ac2, 2005.
• 26: J.K. Ousterhout. Scheduling techniques for concurrent systems. In Distributed Computing Systems, 1982.
• 27: Mohamed Gomaa , Michael D. Powell , T. N. Vijaykumar, Heat-and-run: leveraging SMT and CMP to manage power density through the operating system, Proceedings of the 11th international conference on Architectural support for programming languages and operating systems, October 07-13, 2004, Boston, MA, USA
• 28: Karthikeyan Sankaralingam , Ramadass Nagarajan , Haiming Liu , Changkyu Kim , Jaehyuk Huh , Doug Burger , Stephen W. Keckler , Charles R. Moore, Exploiting ILP, TLP, and DLP with the polymorphous TRIPS architecture, Proceedings of the 30th annual international symposium on Computer architecture, June 09-11, 2003, San Diego, California
• 29: Semiconductor Industry Association. International technology roadmap for semiconductors: Executive summary, 2005.
• 30: Jeremy W. Sheaffer , David P. Luebke , Kevin Skadron, The visual vulnerability spectrum: characterizing architectural vulnerability for graphics hardware, Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware, September 03-04, 2006, Vienna, Austria [doi>10.1145/1283900.1283902]
• 31: Premkishore Shivakumar , Michael Kistler , Stephen W. Keckler , Doug Burger , Lorenzo Alvisi, Modeling the Effect of Technology Trends on the Soft Error Rate of Combinational Logic, Proceedings of the 2002 International Conference on Dependable Systems and Networks, p.389-398, June 23-26, 2002
• 32: Kevin Skadron , Mircea R. Stan , Wei Huang , Sivakumar Velusamy , Karthik Sankaranarayanan , David Tarjan, Temperature-aware microarchitecture, Proceedings of the 30th annual international symposium on Computer architecture, June 09-11, 2003, San Diego, California
• 33: Timothy J. Slegel , Robert M. Averill III , Mark A. Check , Bruce C. Giamei , Barry W. Krumm , Christopher A. Krygowski , Wen H. Li , John S. Liptay , John D. MacDougall , Thomas J. McPherson , Jennifer A. Navarro , Eric M. Schwarz , Kevin Shum , Charles F. Webb, IBM's S/390 G5 Microprocessor Design, IEEE Micro, v.19 n.2, p.12-23, March 1999 [doi>10.1109/40.755464]
• 34: Jared C. Smolens , Brian T. Gold , Babak Falsafi , James C. Hoe, Reunion: Complexity-Effective Multicore Redundancy, Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture, p.223-234, December 09-13, 2006 [doi>10.1109/MICRO.2006.42]
• 35: Gurindar S. Sohi , Scott E. Breach , T. N. Vijaykumar, Multiscalar processors, Proceedings of the 22nd annual international symposium on Computer architecture, p.414-425, June 22-24, 1995, S. Margherita Ligure, Italy
• 36: Daniel J. Sorin , Milo M. K. Martin , Mark D. Hill , David A. Wood, SafetyNet: improving the availability of shared memory multiprocessors with global checkpoint/recovery, Proceedings of the 29th annual international symposium on Computer architecture, May 25-29, 2002, Anchorage, Alaska
• 37: Sun Microsystems, Inc. Sun fire high-end and midrange systems dynamic reconfiguration user's guide. Viewed 12/19/2007.
• 38: Volkmar Uhlig , Joshua LeVasseur , Espen Skoglund , Uwe Dannowski, Towards scalable multiprocessor virtual machines, Proceedings of the 3rd conference on Virtual Machine Research And Technology Symposium, p.4-4, May 06-07, 2004, San Jose, California
• 39: VMware. ESX Server -- best practices using VMware virtual SMP. Viewed 5/03/2006.
• 40: Carl A. Waldspurger, Memory resource management in VMware ESX server, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060307]
• 41: Philip M. Wells , Koushik Chakraborty , Gurindar S. Sohi, Hardware support for spin management in overcommitted virtual machines, Proceedings of the 15th international conference on Parallel architectures and compilation techniques, September 16-20, 2006, Seattle, Washington, USA [doi>10.1145/1152154.1152176]
• 42: Philip M. Wells , Koushik Chakraborty , Gurindar S. Sohi, Adapting to intermittent faults in multicore systems, Proceedings of the 13th international conference on Architectural support for programming languages and operating systems, March 01-05, 2008, Seattle, WA, USA
• 43: Philip M. Wells , Koushik Chakraborty , Gurindar S. Sohi, Mixed-mode multicore reliability, Proceeding of the 14th international conference on Architectural support for programming languages and operating systems, March 07-11, 2009, Washington, DC, USA
• 44: Matt Welsh , David Culler , Eric Brewer, SEDA: an architecture for well-conditioned, scalable internet services, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 45: K. Wonyoung, G. Meeta, W. Gu-Yeon, and B. David. System level analysis of fast, per-core DVFS using on-chip switching regulators. In Proc. of 14th HPCA, February 2008.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.1 Process Management
          Subjects: Concurrency
      D.4.8 Performance

General Terms:
Design, Experimentation, Measurement, Performance