Application-aware management of parallel simulation collections

ABSTRACT

This paper presents a system deployed on parallel clusters to manage a collection of parallel simulations that make up a computational study. It explores how such a system can extend traditional parallel job scheduling and resource allocation techniques to incorporate knowledge specific to the study.

Using a UINTAH-based helium gas simulation code (ARCHES) and the SimX system for multi-experiment computational studies, this paper demonstrates that, by using application-specific knowledge in resource allocation and scheduling decisions, one can reduce the run time of a computational study from over 20 hours to under 4.5 hours on a 32-processor cluster, and from almost 11 hours to just over 3.5 hours on a 64-processor cluster.

REFERENCES

• 1: David Abramson , Andrew Lewis , Tom Peachey , Clive Fletcher, An automatic design optimization tool and its application to computational fluid dynamics, Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM), p.25-25, November 10-16, 2001, Denver, Colorado [doi>10.1145/582034.582059]
• 2: D. Abramson , R. Sosic , J. Giddy , B. Hall, Nimrod: a tool for performing parametrised simulations using distributed workstations, Proceedings of the 4th IEEE International Symposium on High Performance Distributed Computing, p.112, August 02-04, 1995
• 3: H. Casanova and J. Dongarra. Netsolve: A network server for computational science problems. Intl. J. of Supercomp. Appl. and High Perf. Comp., 11(3):212--223, 1997.
• 4: J. Davison de St. Germain , John McCorquodale , Steven G. Parker , Christopher R. Johnson, Uintah: A Massively Parallel Problem Solving Environment, Proceedings of the 9th IEEE International Symposium on High Performance Distributed Computing, p.33, August 01-04, 2000
• 5: P. E. DesJardin, T. J. O'Hern, and S. R. Tieszen. Large eddy simulation and experimental measurements of the near-field of a large turbulent helium plume. Physics of Fluids, 16(6):1866--1883, 2004.
• 6: Marcio Faerman , Adam Birnbaum , Henri Casanova , Francine Berman, Resource Allocation for Steerable Parallel Parameter Searches, Proceedings of the Third International Workshop on Grid Computing, p.157-168, November 18, 2002
• 7: D. G. Feitelson. Job scheduling in multiprogrammed parallel systems. IBM Research Report RC 19790 (87657), Aug 1997.
• 8: Fujita and Yamashita. Approximation algorithms for multiprocessor scheduling problem. TIEICE, 2000.
• 9: Matthias Gries, Methods for evaluating and covering the design space during early design development, Integration, the VLSI Journal, v.38 n.2, p.131-183, December 2004 [doi>10.1016/j.vlsi.2004.06.001]
• 10: A. Messac. Physical programming: Effective optimization for computational design. AIAA Journal, 31(4):149--158, 1996.
• 11: Jarek Nabrzyski , Jennifer M. Schopf , Jan Weglarz, Grid resource management: state of the art and future trends, Kluwer Academic Publishers, Norwell, MA, 2004
• 12: Steven G. Parker , Christopher R. Johnson, SCIRun: a scientific programming environment for computational steering, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.52-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224354]
• 13: Slc , Ut 84112, An Integrated Problem Solving Environment: The SCIRun Computational Steering System, Proceedings of the Thirty-First Annual Hawaii International Conference on System Sciences-Volume 7, p.147, January 06-09, 1998 [doi>10.1109/HICSS.1998.649208]
• 14: J. Schmidt and C. Johnson. DefibSim: An interactive defibrillation device design tool. In Proc. EMBS Conf., 1995.
• 15: M. Scott and E. Antonsson. Preliminary vehicle structure design: An industrial application of imprecision in engineering design.
• 16: J. Spinti, J. Thornock, E. Eddings, P. Smith, and A. Sarofim. Transport Phenomena in Fires, chapter Heat Transfer to objects in pool fires. Witpress, 2008.
• 17: S. Srinivasan, S. Krishnamoorthy, and P. Sadayappan. A robust scheduling strategy for moldable scheduling of parallel jobs. Cluster, 00:92, 2003.
• 18: Srividya Srinivasan , Vijay Subramani , Rajkumar Kettimuthu , Praveen Holenarsipur , P. Sadayappan, Effective Selection of Partition Sizes for Moldable Scheduling of Parallel Jobs, Proceedings of the 9th International Conference on High Performance Computing, p.174-183, December 18-21, 2002
• 19: Vijay Subramani , Rajkumar Kettimuthu , Srividya Srinivasan , Jeanette Johnston , P. Sadayappan, Selective Buddy Allocation for Scheduling Parallel Jobs on Clusters, Proceedings of the IEEE International Conference on Cluster Computing, p.107, September 23-26, 2002
• 20: Douglas Thain , Todd Tannenbaum , Miron Livny, Distributed computing in practice: the Condor experience: Research Articles, Concurrency and Computation: Practice & Experience, v.17 n.2-4, p.323-356, February 2005 [doi>10.1002/cpe.v17:2/4]
• 21: B. Wilson, D. Cappelleri, T. W. Simpson, and M. Frecker. Efficient pareto frontier exploration using surrogate approximations. Optimization and Engineering, 2(1):31--50, 2001.
• 22: S. Yau, K. Damevski, V. Karamcheti, S. Parker, and D. Zorin. Result reuse in design space exploration: A study in system support for interactive parallel computing. In Proc. IPDPS, 2008.
• 23: S. Yau, E. Grinspun, V. Karamcheti, and D. Zorin. Sim-X: Parallel system software for interactive multi-experiment computational studies. In Proc. IPDPS, 2006.
• 24: S. M. Yau, E. Grinspun, V. Karamcheti, and D. Zorin. SimX meets SCIRun: A component-based implementation of a computational study system. In NSFNGS Workshop, IPDPS, pages 1--6, 2007.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.7 Organization and Design
          Subjects: Batch processing systems**

Additional Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.m Miscellaneous

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.7 Organization and Design
          Subjects: Hierarchical design**; Distributed systems

General Terms:
Design, Performance

Keywords:
high-throughput computing, parallel system