Large scale parallel structured AMR calculations using the SAMRAI framework

ABSTRACT

This paper discusses the design and performance of the parallel data communication infrastructure in SAMRAI, a software framework for structured adaptive mesh refinement (SAMR) multi-physics applications. We describe requirements of such applications and how SAMRAI abstractions manage complex data communication operations found in them. Parallel performance is characterized for two adaptive problems solving hyperbolic conservation laws on up to 512 processors of the IBM ASCI Blue Pacific system. Results reveal good scaling for numerical and data communication operations but poorer scaling in adaptive meshing and communication schedule construction phases of the calculations. We analyze the costs of these different operations, addressing key concerns for scaling SAMR computations to large numbers of processors, and discuss potential changes to improve our current implementation.

REFERENCES

• 1: Francis J. Alexander , Alejandro L. Garcia, The direct simulation Monte Carlo method, Computers in Physics, v.11 n.6, p.588-593, Nov./Dec. 1997 [doi>10.1063/1.168619]
• 2: S. B. Baden, P. Colella, D. Shalit, and B. V. Straalen. Abstract KeLP. In 10th SIAM Conference on Parallel Processing for Scientific Computing, Portsmouth, Virginia, March 2001.
• 3: J. Bell and P. Colella. AMR software packages. See http://seesar.lbl.gov/AMR/index.html.
• 4: M. J. Berger , P. Colella, Local adaptive mesh refinement for shock hydrodynamics, Journal of Computational Physics, v.82 n.1, p.64-84, May 1989 [doi>10.1016/0021-9991(89)90035-1]
• 5: M. J. Berger and J. Oliger. Adaptive mesh refinement for hyperbolic partial differential equations. Journal of Computational Physics, 53:484-512, 1984.
• 6: M. J. Berger and I. Rigoutsos. An algorithm for point clustering and grid generation. IEEE Transactions on Systems, Man., and Cybernetics, 21:1278-1286, September 1991.
• 7: G. Bryan and M. L. Norman. Simulation x-ray clusters with adaptive mesh refinement. In Proceedings of the 12th Kingston meeting on Theoretical Astrophysics: Computational Astrophysics (ASP Conference Series, 123), 1997. eds. D. A. Clarke and M. J. West, p. 363.
• 8: A. C. Calder , B. C. Curtis , L. J. Dursi , B. Fryxell , P. MacNeice , K. Olson , P. Ricker , R. Rosner , F. X. Timmes , H. M. Tufo , J. W. Turan , M. Zingale , G. Henry, High performance reactive fluid flow simulations using adaptive mesh refinement on thousands of processors, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.56-es, November 04-10, 2000, Dallas, Texas, United States
• 9: M. R. Dorr, F. X. Garaizar, and J. A. F. Hittinger. Simulation of laser plasma filamentation using adaptive mesh refinement. Technical Report UCRL-JC-138330, LLNL, 2001. Submitted to J. Comput. Phys.
• 10: Stephen J. Fink , Scott B. Baden , Scott R. Kohn, Flexible Communication Mechanisms for Dynamic Structured Applications, Proceedings of the Third International Workshop on Parallel Algorithms for Irregularly Structured Problems, p.203-215, August 19-21, 1996
• 11: Erich Gamma , Richard Helm , Ralph Johnson , John Vlissides, Design patterns: elements of reusable object-oriented software, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 12: Alejandro L. Garcia , John B. Bell , William Y. Crutchfield , Berni J. Alder, Adaptive mesh and algorithm refinement using direct simulation Monte Carlo, Journal of Computational Physics, v.154 n.1, p.134-155, Sept. 1, 1999 [doi>10.1006/jcph.1999.6305]
• 13: Gutman. Use of morton space-filling curve for load balance. Dr. Dobb's Journal, pages 115-121, July 1999.
• 14: R. Hornung. A hybrid model for gas dynamics: Continuum-DSMC with AMR. In First SIAM Conference on Computational Science and Engineering, Washington D.C., Sept 21-23 2000. Also available as Lawrence Livermore National Laboratory technical report UCRL-VG-139774.
• 15: R. D. Hornung and S. R. Kohn. The use of object-oriented design patterns in the SAMRAI structured AMR framework. In Proceedings of the First Workshop on Object Oriented Methods for Inter-operable Scientific and Engineering Computing, 1998. See http://www.llnl.gov/CASC/SAMRAI.
• 16: R. D. Hornung and S. R. Kohn. Managing application complexity in the SAMRAI object-oriented framework. Concurrency: Theory and Practice, 2001. (to appear).
• 17: Scott Richard Kohn , Scott B. Baden, A parallel software infrastructure for dynamic block-irregular scientific calculations, 1995
• 18: Manish Parashar , James C. Browne , Carter Edwards , Kenneth Klimkowski, A common data management infrastructure for adaptive algorithms for PDE solutions, Proceedings of the 1997 ACM/IEEE conference on Supercomputing (CDROM), p.1-22, November 15-21, 1997, San Jose, CA [doi>10.1145/509593.509649]
• 19: J. Saltz, H. Berryman, and J. Wu. Multiprocessors and run-time compilation. Concurrency: Practice and Experience, 3(6):573-592, 1991.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.0 General
          Subjects: Data communications

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.3 Interconnections (subsystems)
          Subjects: Parallel I/O

  J. Computer Applications
  J.2 PHYSICAL SCIENCES AND ENGINEERING
      Subjects: Physics

General Terms:
Algorithms, Design, Performance