A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator

ABSTRACT

We use 1944 processors of the Earth Simulator to model seismic wave propagation resulting from large earthquakes. Simulations are conducted based upon the spectral-element method, a high-degree finite-element technique with an exactly diagonal mass matrix. We use a very large mesh with 5.5 billion grid points (14.6 billion degrees of freedom). We include the full complexity of the Earth, i.e., a three-dimensional wave-speed and density structure, a 3-D crustal model, ellipticity as well as topography and bathymetry. A total of 2.5 terabytes of memory is needed. Our implementation is purely based upon MPI, with loop vectorization on each processor. We obtain an excellent vectorization ratio of 99.3%, and we reach a performance of 5 teraflops (30% of the peak performance) on 38% of the machine. The very high resolution of the mesh allows us to perform fully three-dimensional calculations at seismic periods as low as 5 seconds.

REFERENCES

• 1: [1] C. Bassin, G. Laske, and G. Masters. The current limits of resolution for surface wave tomography in North America. EOS, 81:F897, 2000.
• 2: [2] E. Chaljub. Modélisation numérique de la propagation d'ondes sismiques en géométrie sphérique: application à la sismologie globale (Numerical modeling of the propagation of seismic waves in spherical geometry: applications to global seismology). PhD thesis, Université Paris VII Denis Diderot, Paris, France, 2000.
• 3: [3] F. A. Dahlen and J. Tromp. Theoretical Global Seismology. Princeton University Press, Princeton, 1998.
• 4: [4] A. M. Dziewonski and D. L. Anderson. Preliminary reference Earth model. Phys. Earth Planet. Inter., 25:297-356, 1981.
• 5: [5] D. Eberhart-Phillips, P. J. Haeussler, J. T. Freymueller, A. D. Frankel, C. M. Rubin, P. Craw, N. A. Ratchkovski, G. Anderson, G. A. Carver, A. J. Crone, T. E. Dawson, H. Fletcher, R. Hansen, E. L. Harp, R. A. Harris, D. P. Hill, S. Hreinsdottir, R. W. Jibson, L. M. Jones, R. Kayen, D. K. Keefer, C. F. Larsen, S. C. Moran, S. F. Personius, G. Plafker, B. Sherrod, K. Sieh, N. Sitar, and W. K. Wallace. The 2002 Denali fault earthquake, Alaska: a large magnitude, slip-partitioned event. Science, 300:1113-1118, 2003.
• 6: [6] E. Faccioli, F. Maggio, R. Paolucci, and A. Quarteroni. 2D and 3D elastic wave propagation by a pseudospectral domain decomposition method. J. Seismol., 1:237-251, 1997.
• 7: [7] C. Ji, D. V. Helmberger, and D. J. Wald. Preliminary slip history of the 2002 Denali earthquake. EOS Trans. AGU., 83, 2002.
• 8: [8] D. Komatitsch. Méthodes spectrales et éléments spectraux pour l'équation de l'élastodynamique 2D et 3D en milieu hétérogène (Spectral and spectral-element methods for the 2D and 3D elastodynamics equations in heterogeneous media). PhD thesis, Institut de Physique du Globe, Paris, France, 1997.
• 9: [9] D. Komatitsch and J. Tromp. Introduction to the spectral-element method for 3-D seismic wave propagation. Geophys. J. Int., 139:806-822, 1999.
• 10: [10] D. Komatitsch and J. Tromp. Spectral-element simulations of global seismic wave propagation-I. Validation. Geophys. J. Int., 149:390-412, 2002.
• 11: [11] D. Komatitsch and J. Tromp. Spectral-element simulations of global seismic wave propagation-II. 3-D models, oceans, rotation, and self-gravitation. Geophys. J. Int., 150:303-318, 2002.
• 12: [12] D. Komatitsch and J. P. Vilotte. The spectral-element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bull. Seismol. Soc. Am., 88(2):368-392, 1998.
• 13: [13] R. A. Page, G. Plafker, and H. Pulpan. Block rotation in east-central Alaska: a framework for evaluating earthquake potential? Geology, 23:629-632, 1995.
• 14: [14] J. Ritsema, H. J. Van Heijst, and J. H. Woodhouse. Complex shear velocity structure imaged beneath Africa and Iceland. Science, 286:1925-1928, 1999.
• 15: C. Ronchi , R. Iacono , Pier S. Paolucci, The “cubed sphere”: a new method for the solution of partial differential equations in spherical geometry, Journal of Computational Physics, v.124 n.1, p.93-114, March 1996 [doi>10.1006/jcph.1996.0047]
• 16: [16] R. Sadourny. Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Monthly Weather Review, 100:136-144, 1972.
• 17: [17] G. Seriani. 3-D large-scale wave propagation modeling by a spectral element method on a Cray T3E multiprocessor. Comput. Methods Appl. Mech. Engrg., 164:235-247, 1998.
• 18: Satoru Shingu , Hiroshi Takahara , Hiromitsu Fuchigami , Masayuki Yamada , Yoshinori Tsuda , Wataru Ohfuchi , Yuji Sasaki , Kazuo Kobayashi , Takashi Hagiwara , Shin-ichi Habata , Mitsuo Yokokawa , Hiroyuki Itoh , Kiyoshi Otsuka, A 26.58 Tflops global atmospheric simulation with the spectral transform method on the Earth Simulator, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-19, November 16, 2002, Baltimore, Maryland