Achieving extreme resolution in numerical cosmology using adaptive mesh refinement: resolving primordial star formation

ABSTRACT

As an entry for the 2001 Gordon Bell Award in the "special" category, we describe our 3-d, hybrid, adaptive mesh refinement (AMR) code Enzo designed for high-resolution, multiphysics, cosmological structure formation simulations. Our parallel implementation places no limit on the depth or complexity of the adaptive grid hierarchy, allowing us to achieve unprecedented spatial and temporal dynamic range. We report on a simulation of primordial star formation which develops over 8000 subgrids at 34 levels of refinement to achieve a local refinement of a factor of 10¹² in space and time. This allows us to resolve the properties of the first stars which form in the universe assuming standard physics and a standard cosmological model. Achieving extreme resolution requires the use of 128-bit extended precision arithmetic (EPA) to accurately specify the subgrid positions. We describe our EPA AMR implementation on the IBM SP2 Blue Horizon system at the San Diego Supercomputer Center.

REFERENCES

• 1: T. Abel, P. Anninos, Y. Zhang & M. Norman, "Modeling primordial gas in numerical cosmology", New Astron., 2, (1997), pp. 181-208.
• 2: T. Abel, G. Bryan and M. L. Norman, "The Formation and Fragmentation of Primordial Molecular Clouds", Astrophys. J., 540, 39 (2000).
• 3: T. Abel, G. Bryan and M. L. Norman, "Forming the First Star in the Universe", in Physics of Galaxy Formation, eds. M. Umemura & H. Susa, ASP Conference Series Vol. 151, (in press).
• 4: P. Anninos, Y. Zhang, T. Abel & M. Norman, "Cosmological Hydrodynamics with multi-species chemistry and nonequilibrium ionization and cooling", New Astron. 2 (1997), pp. 209-224.
• 5: David H. Bailey, Algorithm 719; Multiprecision translation and execution of FORTRAN programs, ACM Transactions on Mathematical Software (TOMS), v.19 n.3, p.288-319, Sept. 1993 [doi>10.1145/155743.155767]
• 6: M. J. Berger , P. Colella, Local adaptive mesh refinement for shock hydrodynamics, Journal of Computational Physics, v.82 n.1, p.64-84, May 1989 [doi>10.1016/0021-9991(89)90035-1]
• 7: M.J. Berger and I. Rigoutsos, "An Algorithm for Point Clustering and Grid Generation," IEEE Transactions on Systems, Man and Cybernetics, Vol. 21, No. 5, Sep. 1991.
• 8: G.L. Bryan, M.L. Norman, J.M. Stone, R. Cen, and J.P. Ostriker, "A Piecewise Parabolic Method for Cosmological Hydrodynamics," Computer Physics Communication, Vol. 89, 1995, pp. 149-168.
• 9: G.L. Bryan and M.L. Norman, "A Hybrid AMR Application for Cosmology and Astrophysics," in IMA Volume 117 on Structured Adaptive Mesh Refinement (SAMR) Grid Methods, eds. S.B. Baden, N.P. Chrisochoides, D. Gannon, and M.L. Norman, (Springer: New York), pp. 165-170 (2000).
• 10: Greg L. Bryan, Fluids in the universe: adaptive mesh refinement in cosmology, Computing in Science and Engineering, v.1 n.2, p.46-53, March-April 1999 [doi>10.1109/5992.753046]
• 11: Colberg, J. M. et al. "Clustering of galaxy clusters in cold dark matter universes", Mon. Not. R. Astron. Soc., 319 (2000), pp. 209-214.
• 12: Toshiyuki Fukushige , Junichiro Makino, N-body simulation of galaxy formation on GRAPE-4 special-purpose computer, Proceedings of the 1996 ACM/IEEE conference on Supercomputing (CDROM), p.48-es, January 01-01, 1996, Pittsburgh, Pennsylvania, United States [doi>10.1145/369028.369130]
• 13: Junichiro Makino , Toshiyuki Fukushige , Masaki Koga, A 1.349 Tflops simulation of black holes in a galactic center on GRAPE-6, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.43-es, November 04-10, 2000, Dallas, Texas, United States
• 14: M.L. Norman and G.L. Bryan, "Cosmological Adaptive Mesh Refinement," in Numerical Astrophysics, eds. S. Miyama & K. Tomisaka, (Kluwer: Dordrecht) (1999), pp. 19-28.
• 15: M. Norman, T. Abel, and G. Bryan, "First Structure Formation and the First Stars", in The First Stars, eds. A. Weiss, T. Abel & V. Hill, (Springer: Heidelberg), (2000), pp. 250-260.
• 16: J. P. Ostriker, "Astronomical Tests of the Cold Dark Matter Scenario", Annu. Rev. Astron. Astrophys. 1993 31: 689-716.
• 17: J.M. Stone and M.L. Norman, "ZEUS-2D: A radiation magnetohydro-dynamics code for astrophysical flows in two space dimensions.", 1992, Astrophysics. J. Supp. 80, 753
• 18: V. Springel, SDM White, G. Tormen, G. & G. Kaufmann "Populating a cluster of galaxies --- I. Results at z=0", Mon. Not. R. Astron. Soc., submitted (astro-ph/0012055)
• 19: M. S. Warren , J. K. Salmon, Astrophysical N-body simulations using hierarchical tree data structures, Proceedings of the 1992 ACM/IEEE conference on Supercomputing, p.570-576, November 16-20, 1992, Minneapolis, Minnesota, United States
• 20: M. S. Warren, J. K. Salmon et al., "A Treecode at 430 Gigaflops on ASCI Red", in Proceedings of Supercomputing 1997, 1997.
• 21: P.R. Woodward and P. Colella, "A Piecewise Parabolic Method for Gas-Dynamical Simulations" 1984, J. Comput. Physics, 54, 174.
• 22: Zhiling Lan , Valerie E. Taylor , Greg Bryan, Dynamic Load Balancing for Structured Adaptive Mesh Refinement Applications, Proceedings of the 2001 International Conference on Parallel Processing, p.571-579, September 03-07, 2001

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.4 MATHEMATICAL SOFTWARE
      Subjects: Parallel and vector implementations

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture
          Subjects: Imaging geometry

  J. Computer Applications
  J.2 PHYSICAL SCIENCES AND ENGINEERING
      Subjects: Physics

General Terms:
Algorithms, Design, Performance

Keywords:
adaptive mesh refinement, numerical cosmology, parallel algorithms