Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

The UNIC code is being developed as part of the DOE's Nuclear Energy Advanced Modeling and Simulation (NEAMS) program. UNIC is an unstructured, deterministic neutron transport code that allows a highly detailed description of a nuclear reactor. The primary goal of our simulation efforts is to reduce the uncertainties and biases in reactor design calculations by progressively replacing existing multilevel averaging (homogenization) techniques with more direct solution methods based on first principles. Since the neutron transport equation is seven dimensional (three in space, two in angle, one in energy, and one in time), these simulations are among the most memory and computationally intensive in all of computational science. In order to model the complex physics of a reactor core, billions of spatial elements, hundreds of angles, and thousands of energy groups are necessary, leading to problem sizes with petascale degrees of freedom. Therefore, these calculations exhaust memory resources on current and even next-generation architectures. In this paper, we present UNIC simulation results for two important representative problems in reactor design and analysis---PHENIX and ZPR-6. In each case, UNIC shows good weak scalability on up to 163,840 cores of Blue Gene/P (Argonne) and 122,800 cores of XT5 (Oak Ridge). While our current per processor performance is less than ideal, we demonstrate a clear ability to effectively utilize the leadership computing platforms. Over the coming months, we aim to improve the per processor performance while maintaining the high parallel efficiency by employing better algorithms such as spatial p- and h-multigrid preconditioners, optimized matrix-tensor operations, and weighted partitioning for better load balancing. Combining these additional algorithmic improvements with the availability of larger parallel machines should allow us to realize our long-term goal of explicit geometry coupled multiphysics reactor simulations. In the long run, these high-fidelity simulations will be able to replace expensive mockup experiments and reduce the uncertainty in crucial reactor design and operational parameters.

REFERENCES

• 1: E. E. Lewis and Jr. W. F. Miller, "Computational Methods of Neutron Transport," John Wiley&Sons, New York, 1984.
• 2: G. Palmiotti, M. A. Smith, C. Rabiti, M. Leclere, D. Kaushik, A. Siegel, B. Smith, E. E. Lewis, "UNÌC: Ultimate Neutronic Investigation Code," Joint International Topical Meeting on Mathematics&Computation and Supercomputing in Nuclear Applications, Monterey, California, April 15--19, 2007.
• 3: Argonne National Laboratory, Argonne Leadership Computing Facility, http://www.alcf.anl.gov.
• 4: Oak Ridge National Laboratory, National Center for Computational Sciences, http://www.nccs.gov.
• 5: C. Rabiti, E. Wolters, M. A. Smith, G. Palmiotti, "Spherical Quadratures for the Discrete Ordinates Method," Trans. Am. Nucl. Soc 96, Boston, MA, June 24--28, 2007.
• 6: J. Lamarsh, "Introduction to Nuclear Engineering," Addison-Wesley Publishing Co, Massachusetts, 1983.
• 7: R. E. Alcouffe, R. S. Baker, J. A. Dahl, S. A. Turner, and Robert Ward, "PARTISN: A Time-Dependent, Parallel Neutral Particle Transport Code System," LA-UR-05-3925, May 2005.
• 8: W. A. Rhoades, R. L. Childs, M. B. Emmett, and S. N. Cramer, "Application of the Three-Dimensional Oak Ridge Transport Code," Proc. Am. Nucl. Soc. Topical Meeting on Reactor Physics and Shielding," pp. 225--238, Chicago, September 17--19, 1984.
• 9: Sjoden, G. and Haghighat, A., "PENTRAN#8482;: Parallel Environment Neutral particle TRANsport in 3-D Cartesian Geometry", Proc. Int. Conf. on Mathematical Methods and Supercomputing for Nuclear Applications, pp. 232--234. Saratoga Springs, NY (1997). Performance of PENTRAN#8482; 3-D Parallel Particle Transport Code.
• 10: G. Palmiotti, E. E. Lewis and C. B. Carrico, "VARIANT: VARIational Anisotropic Nodal Transport for Multidimensional Cartesian and Hexagonal Geometry Calculation," ANL-95/40 Argonne National Laboratory, 1995.
• 11: C. R. E. de Oliveira, A. J. H. Goddard, "EVENT - A Multidimensional Finite Element-Spherical Harmonics Radiation Transport Code," Proceedings of the OECD International Seminar on 3D Deterministic Radiation Transport Codes, Paris, December 01--02, 1996.
• 12: John M. McGhee, Randy M. Roberts, Jim E. Morel, "The DANTE Boltzmann Transport Sove: An Unstructured Mesh, 3-D, Spherical Harmonics Algorithm Compatible with Parallel Computer Architectures," Jnt. Int. Conf. on Math. Meth. and super. For Nuc. Apps., Saratoga Springs, New York, Oct. 5--10, 1997.
• 13: T. A. Wareing, J. M. McGhee, J. E. Morel, and S. D. Pautz, "Discontinuous Finite Element Sn Methods on 3-D Unstructured Meshes," Nuclear Science and Engineering, 138:1--13, 2001.
• 14: A. Siegel, D. Kaushik, P. Fischer, G. Palmiotti, M. A. Smith, C. Rabiti, J. Ragusa, "Software Design of SHARP," Joint International Topical Meeting on Mathematics&Computation and Supercomputing in Nuclear Applications, Monterey, California, April 15--19, 2007.
• 15: Rabiti, M. A. Smith, G. Palmiotti, "A Three-dimensional Method of Characteristics on Unstructured Tetrahedral Meshes," Trans. Am. Nucl. Soc. 96, 470, 2007.
• 16: M. A. Smith, G. Palmiotti, C. Rabiti, D. Kaushik, A. Siegel, B. Smith, E. E. Lewis, "PNFE Component of the UNÌC Code," Joint International Topical Meeting on Mathematics&Computation and Supercomputing in Nuclear Applications, Monterey, California, April 15--19, 2007.
• 17: S. Balay, K. R. Buschelman, W. D. Gropp, D. K. Kaushik, M. G. Knepley, L. C. McInnes, and B. F. Smith, "PETSc Home Page," http://www.mcs.anl.gov/petsc.
• 18: M. A. Smith, G. Palmiotti, et al., "Benchmark on Deterministic Transport Calculations Without Spatial Homogenization (MOX Fuel Assembly 3-D Extension Case)," OECD/NEA document, NEA/NSC/DOC(2005)16, October 2005.
• 19: Personal Communication, Frederic Varaine, CEA, "IAEA CRP on PHENIX End of Life Tests; Control Rod Withdrawal," 2009.
• 20: Lawrence Livermore National Laboratory, "Visit: A Distributed, Parallel, Visualization Tool," https://wci.llnl.gov/codes/visit/home.html.
• 21: Sandia National Laboratory, "CUBIT: Geometry and Mesh Generation Toolkit," http://cubit.sandia.gov/.
• 22: Amine Abou-Rjeili and George Karypis, "Multilevel Algorithms for Partitioning Power-Law Graphs," IEEE International Parallel&Distributed Processing Symposium (IPDPS), 2006.
• 23: NEA Nuclear Science Committee, "International Handbook of Evaluated Criticality Safety Benchmark Experiments," NEA/NSC/DOC(95)03, September 2007.
• 24: W. D. Gropp, D. K. Kaushik, D. E. Keyes, and B. F. Smith, "Toward Realistic Performance Bounds for Implicit CFD Codes," In D. Keyes, A. Ecer, J. Periaux, N. Satofuka, and P. Fox, editors, Proceedings of Parallel CFD'99, pages 233--240. Elsevier, 1999.
• 25: D. Kaushik, W. Gropp, M. Minkoff, B. Smith, "Improving the Performance of Tensor Matrix Vector Multiplication in Cumulative Reaction Probability Based Quantum Chemistry Codes," Proceedings of the 15th International Conference on High Performance Computing (HiPC 2008), Bangalore, India, Springer LNCS5374:120--130, December, 2008.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.2 PHYSICAL SCIENCES AND ENGINEERING
      Subjects: Physics

Additional Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.1 Large and Medium ("Mainframe") Computers
          Subjects: Super (very large) computers

  F. Theory of Computation
  F.1 COMPUTATION BY ABSTRACT DEVICES
      F.1.2 Modes of Computation
          Subjects: Parallelism and concurrency

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Performance

Keywords:
discrete ordinates, memory bandwidth, neutron transport, nuclear reactors, parallel scalability