High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors

ABSTRACT

SPECFEM3D_GLOBE is a spectral-element application enabling the simulation of global seismic wave propagation in 3D anelastic, anisotropic, rotating and self-gravitating Earth models at unprecedented resolution. A fundamental challenge in global seismology is to model the propagation of waves with periods between 1 and 2 seconds, the highest frequency signals that can propagate clear across the Earth. These waves help reveal the 3D structure of the Earth's deep interior and can be compared to seismographic recordings. We broke the 2 second barrier using the 62K processor Ranger system at TACC. Indeed we broke the barrier using just half of Ranger, by reaching a period of 1.84 seconds with sustained 28.7 Tflops on 32K processors. We obtained similar results on the XT4 Franklin system at NERSC and the XT4 Kraken system at University of Tennessee Knoxville, while a similar run on the 28K processor Jaguar system at ORNL, which has better memory bandwidth per processor, sustained 35.7 Tflops (a higher flops rate) with a 1.94 shortest period.

Thus we have enabled a powerful new tool for seismic wave simulation, one that operates in the same frequency regimes as nature; in seismology there is no need to pursue periods much smaller because higher frequency signals do not propagate across the entire globe.

We employed performance modeling methods to identify performance bottlenecks and worked through issues of parallel I/O and scalability. Improved mesh design and numbering results in excellent load balancing and few cache misses. The primary achievements are not just the scalability and high teraflops number, but a historic step towards understanding the physics and chemistry of the Earth's interior at unprecedented resolution.

REFERENCES

• 1: Computational Infastructure for Geodynamics (CIG).
• 2: IPM: Integrated Performance Monitoring.
• 3: Canuto, C., Hussaini, M. Y., Quarteroni, A. and Zang, T. A. Spectral methods in fluid dynamics. Springer-Verlag, New York, 1998.
• 4: Chaljub, E. and Valette, B. Spectral element modelling of three-dimensional wave propagation in a self-gravitating Earth with an arbitrarily stratified outer core. Geophys. J. Int., 158 (131--141).
• 5: Cohen, G., Joly, P. and Tordjman, N. Construction and analysis of higherorder finite elements with mass lumping for the wave equation. Proceedings of the second international conference on mathematical and numerical aspects of wave propagation, SIAM. 152--160.
• 6: Fischer, P. F. and Rønquist, E. M. Spectral-element methods for large scale parallel Navier-Stokes calculations. Comput. Methods Appl. Mech. Engrg., 116. 69--76.
• 7: Komatitsch, D., Labarta, J. and Michéa, D. A 21 billion degrees of freedom, 2.5 terabytes simulation of seismic wave propagation in the inner core of the Earth on MareNostrum. Proceedings of the 8th World Congress on Computational Mechanics (WCCM8) and the 5th European Congress on Computational Methods in Applied Sciences and Engineering (ECCOMAS 2008).
• 8: Komatitsch, D. and Tromp, J. Introduction to the spectral-element method for 3-D seismic wave propagation. Geophys. J. Int., 139 (3). 806--822.
• 9: Komatitsch, D. and Tromp, J. Spectral-element simulations of global seismic wave propagation-I. Validation. Geophys. J. Int., 149 (2). 390--412.
• 10: Komatitsch, D. and Tromp, J. Spectral-element Simulations of Global Seismic Wave Propagation-II. 3-D Models, Oceans, Rotation, and Self-Gravitation. Geophys. J. Int., 150. 303--318.
• 11: Dimitri Komatitsch , Seiji Tsuboi , Chen Ji , Jeroen Tromp, A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.4, November 15-21, 2003
• 12: Komatitsch, D. and Vilotte, J. P. The Spectral-element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bull. Seismol. Soc. Am., 88 (2). 368--392.
• 13: Liu, Q. and Tromp, J. Finite-Frequency Kernel Based on Adjoint Methods. Bulletin of the Seismological Society of America, 96 (6). 2383--2397.
• 14: Patera, A. T. A Spectral element method for fluid dynamics: laminar flow in a channel expansion. J. Comput. Phys., 54. 468--488.
• 15: Priolo E., Carcione, J. M. and Seriani, G. Numerical simulation of interface waves by high-order spectral modeling techniques. J. Acoust. Soc. Am., 95 (2). 681--693.
• 16: C. Ronchi , R. Iacono , Pier S. Paolucci, The “cubed sphere”: a new method for the solution of partial differential equations in spherical geometry, Journal of Computational Physics, v.124 n.1, p.93-114, March 1996 [doi>10.1006/jcph.1996.0047]
• 17: Sadourny, R. Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Mon. Wea. Rev, 100. 136--144.
• 18: PSiNS Tracer and Simulator, Performance Modeling and Characterization Lab, SDSC, San Diego, CA, http://www.sdsc.edu/pmac/projects/psins.html

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Parallel

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

  J. Computer Applications
  J.2 PHYSICAL SCIENCES AND ENGINEERING
      Subjects: Earth and atmospheric sciences

General Terms:
Algorithms, Design, Experimentation, Measurement, Performance