Cellular disco: resource management using virtual clusters on shared-memory multiprocessors

ABSTRACT

Despite the fact that large-scale shared-memory multiprocessors have been commercially available for several years, system software that fully utilizes all their features is still not available, mostly due to the complexity and cost of making the required changes to the operating system. A recently proposed approach, called Disco, substantially reduces this development cost by using a virtual machine monitor that laverages the existing operating system technology. In this paper we present a system called Cellular Disco that extends the Disco work to provide all the advantages of the hardware partitioning and scalable operating system approaches. We argue that Cellular Disco can achieve these benefits at only a small fraction of the development cost of modifying the operating system. Cellular Disco effectively turns a large-scale shared-memory multiprocessor into a virtual cluster that supports fault containment and heterogeneity, while avoiding operating system scalability bottlenecks. Yet at the same time, Cellular Disco preserves the benefits of a shared-memory multiprocessor by implementing dynamic, fine-grained resource sharing, and by allowing users to overcommit resources such as processors and memory. This hybrid approach requires a scalable resource manager that makes local decisions with limited information while still providing good global performance and fault containment. In this paper we describe our experience with a Cellular Disco prototype on a 32-processor SGI Origin 2000 system. We show that the execution time penalty for this approach is low, typically within 10% of the best available commercial operating system formost workloads, and that it can manage the CPU and memory resources of the machine significantly better than the hardware partitioning approach.

REFERENCES

• 1: James M. Barton , Nawaf Bitar, A Scalable Multi-Discipline, Multiple-Processor Scheduling Framework for IRIX, Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, p.45-69, April 25, 1995
• 2: Edouard Bugnion , Scott Devine , Kinshuk Govil , Mendel Rosenblum, Disco: running commodity operating systems on scalable multiprocessors, ACM Transactions on Computer Systems (TOCS), v.15 n.4, p.412-447, Nov. 1997 [doi>10.1145/265924.265930]
• 3: J. Chapin , M. Rosenblum , S. Devine , T. Lahiri , D. Teodosiu , A. Gupta, Hive: fault containment for shared-memory multiprocessors, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.12-25, December 03-06, 1995, Copper Mountain, Colorado, United States
• 4: COMPAQ. 2000. OpenVMS Galaxy. Available: http://www.openvms.compaq.com/availability/ galaxy.html.
• 5: CREASY, R. 1981. The origin of the VM/370 time-sharing system. IBM J. Res. Develop 25,5, 483-490.
• 6: M. J. Feeley , W. E. Morgan , E. P. Pighin , A. R. Karlin , H. M. Levy , C. A. Thekkath, Implementing global memory management in a workstation cluster, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.201-212, December 03-06, 1995, Copper Mountain, Colorado, United States
• 7: GALLES,M.AND WILLIAMS, E. 1994. Performance optimizations, implementation, and verifi-cation of the SGI Challenge multiprocessor. In T. N. Mudge and B. D. Shriver Eds., Proceedings of the 27th Hawaii International Conference on System Sciences. Volume 1: Architecture (Los Alamitos, CA, USA, January 1994), pp. 134-143. IEEE Computer Society Press.
• 8: Ben Gamsa , Orran Krieger , Jonathan Appavoo , Michael Stumm, Tornado: maximizing locality and concurrency in a shared memory multiprocessor operating system, Proceedings of the third symposium on Operating systems design and implementation, p.87-100, February 1999, New Orleans, Louisiana, United States
• 9: GOLDBERG, R. P. 1974. Survey of virtual machine research. IEEE Computer Magazine 7,4 (June), 34-45.
• 10: IBM. 2000. The K42 Project. Available: http://www.research.ibm.com/K42/index.html.
• 11: Gerry Kane , Joe Heinrich, MIPS RISC architectures, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 12: Paul A. Karger , Mary Ellen Zurko , Douglas W. Bonin , Andrew H. Mason , Clifford E. Kahn, A Retrospective on the VAX VMM Security Kernel, IEEE Transactions on Software Engineering, v.17 n.11, p.1147-1165, November 1991 [doi>10.1109/32.106971]
• 13: J. Kuskin , D. Ofelt , M. Heinrich , J. Heinlein , R. Simoni , K. Gharachorloo , J. Chapin , D. Nakahira , J. Baxter , M. Horowitz , A. Gupta , M. Rosenblum , J. Hennessy, The Stanford FLASH multiprocessor, Proceedings of the 21ST annual international symposium on Computer architecture, p.302-313, April 18-21, 1994, Chicago, Illinois, United States
• 14: James Laudon , Daniel Lenoski, The SGI Origin: a ccNUMA highly scalable server, Proceedings of the 24th annual international symposium on Computer architecture, p.241-251, June 01-04, 1997, Denver, Colorado, United States
• 15: LEVY,H.M.AND LIPMAN, P. H. 1982. Virtual memory management in the VAX/VMS operating system. Computer 15, 3 (March), 35-41.
• 16: Nancy A. Lynch, Distributed Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1996
• 17: MILOJICIC,D.S.,DOUGLIS, F., PAINDAVEINE, Y., WHEELER, R., AND ZHOU, S. 1996. Process migration. Technical report (December), TOG Research Institute.
• 18: Richard Rashid , Avadis Tevanin, Jr. , Michael Young , David Golub , Robert Baron, Machine-Independent Virtual Memory Management for Paged Uniprocessor and Multiprocessor Architectures, IEEE Transactions on Computers, v.37 n.8, p.896-908, August 1988 [doi>10.1109/12.2242]
• 19: Mendel Rosenblum , Edouard Bugnion , Scott Devine , Stephen A. Herrod, Using the SimOS machine simulator to study complex computer systems, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.7 n.1, p.78-103, Jan. 1997 [doi>10.1145/244804.244807]
• 20: SEAWRIGHT,L.H.AND MACKINNON, R. A. 1979. VM/370: a study of multiplicity and useful-ness. IBM Systems Journal 18, 1, 4-17.
• 21: SEQUENT COMPUTER SYSTEMS. 2000. Application Region Manager. Available: http://www. sequent.com/dcsolutions/agile_wp1.html.
• 22: SGI. 2000. IRIX6.5. Available: http://www.sgi.com/software/irix6.5.
• 23: SPEC. 2000. SPECweb96 Benchmark. Available: http://www.spec.org/osg/web96.
• 24: SUN MICROSYSTEMS. 2000. Sun Enterprise 10000: Dynamic System Domains. Available: http://www.sun.com/servers/highend/10000/Tour/domains.html.
• 25: Dan Teodosiu , Joel Baxter , Kinshuk Govil , John Chapin , Mendel Rosenblum , Mark Horowitz, Hardware fault containment in scalable shared-memory multiprocessors, Proceedings of the 24th annual international symposium on Computer architecture, p.73-84, June 01-04, 1997, Denver, Colorado, United States
• 26: TRANSACTION PROCESSING PERFORMANCE COUNCIL. 1997. TPC Benchmark D (Decision Sup-port) Standard Specification. TPC, San Jose, CA.
• 27: UNISYS. 2000. Cellular MultiProcessing Architecture. Available: http://www.unisys.com/ marketplace/ent/cmp.html.
• 28: Ben Verghese , Scott Devine , Anoop Gupta , Mendel Rosenblum, Operating system support for improving data locality on CC-NUMA compute servers, Proceedings of the seventh international conference on Architectural support for programming languages and operating systems, p.279-289, October 01-04, 1996, Cambridge, Massachusetts, United States
• 29: VMWARE. 2000. VMware Virtual Platform. Available: http://www.vmware.com/products/ virtualplatform.html.
• 30: Steven Cameron Woo , Moriyoshi Ohara , Evan Torrie , Jaswinder Pal Singh , Anoop Gupta, The SPLASH-2 programs: characterization and methodological considerations, Proceedings of the 22nd annual international symposium on Computer architecture, p.24-36, June 22-24, 1995, S. Margherita Ligure, Italy

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.1 Process Management

Additional Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.2 Storage Management
      D.4.5 Reliability

General Terms:
Design, Management, Performance, Reliability

Keywords:
fault containment, resource managment, scalable multiprocessors, virtual machines