Parallel Iterative Solvers of GeoFEM with Selective Blocking Preconditioning for Nonlinear Contact Problems on the Earth Simulator

ABSTRACT

An efficient parallel iterative method with selective blocking preconditioning has been developed for symmetric multiprocessor (SMP) cluster architectures with vector processors such as the Earth Simulator. This method is based on a three-level hybrid parallel programming model, which includes message passing for inter-SMP node communication, loop directives by OpenMP for intra-SMP node parallelization and vectorization for each processing element (PE). This method provides robust and smooth convergence and excellent vector and parallel performance in 3D geophysical simulations with contact conditions performed on the Earth Simulator. The selective blocking preconditioning is much more efficient than ILU(1) and ILU(2). Performance for the complicated Southwest Japan model with more than 23 M DOF on 10 SMP nodes (80 PEs) of the Earth Simulator was 161.7 GFLOPS, corresponding to 25.3% of the peak performance for hybrid programming model, and 190.4 GFLOPS (29.8% of the peak performance) for flat MPI, respectively.

REFERENCES

• 1: [1] Earth Simulator Center Web Site: http://www.es.jamstec.go.jp/
• 2: [2] GeoFEM Web Site: http://geofem.tokyo.rist.or.jp/
• 3: [3] Iizuka, M., Okuda, H. and Yagawa, G.: "Nonlinear Structural Subsystem of GeoFEM forFault Zone Analysis", Pure and Applied Geophysics, Vol. 157 (2000), pp. 2105-2124.
• 4: [4] Nakajima, K. and Okuda, H. "Parallel Iterative Solvers with Selective Blocking Preconditioning for Simulations of Fault Zone Contact", 2001 International Conference on Preconditioning Techniques for Large Sparse Matrix Problems in Industrial Applications (Preconditioning 2001), Tahoe City, CA, USA, 2001, Journal of Numerical Algebra with Applications (in press).
• 5: [5] Nakajima, K.: "OpenMP/MPI Hybrid vs. Flat MPI on the Earth Simulator: Parallel Iterative Solvers for Finite Element Method", International Workshop on OpenMP: Experiences and Implementations (WOMPEI 2003), Tokyo, Japan, Lecture Notes in Computer Science 2858 (in press), Springer, 2003.
• 6: [6] Nakajima, K., Okuda, H.: "Parallel Iterative Solvers with Localized ILU Preconditioning for Unstructured Grids on Workstation Clusters", International Journal for Computational Fluid Dynamics, Vol. 12 (1999) pp. 315-322.
• 7: Barry F. Smith , Petter E. Bjørstad , William D. Gropp, Domain decomposition: parallel multilevel methods for elliptic partial differential equations, Cambridge University Press, New York, NY, 1996
• 8: Jack J. Dongarra , Lain S. Duff , Danny C. Sorensen , Henk A. Vander Vorst, Numerical Linear Algebra for High Performance Computers, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1998
• 9: J. Liou , T. E. Tezduyar, Clustered element-by-element computations for fluid flow, Parallel computational fluid dynamics: implementations and results, MIT Press, Cambridge, MA, 1992
• 10: [10] Accelerated Strategic Computing Initiative (ASCI) Web Site : http://www.llnl.gov/asci/
• 11: Franck Cappello , Daniel Etiemble, MPI versus MPI+OpenMP on IBM SP for the NAS benchmarks, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.12-es, November 04-10, 2000, Dallas, Texas, United States
• 12: [12] Djomehri, M.J. and Jin, H.H.: "Hybrid MPI+OpenMP Programming of an Overset CFD Solver and Performance Investigations", NASA/NAS Technical Report (NASA Ames Research Center), NAS-02-002, (2002). 16.
• 13: [13] Falgout, R. and Jones, J. : "Multigrid on Massively Parallel Architectures", Sixth European Multigrid Conference , Ghent, Belgium, September 27-30, 1999.
• 14: Leonid Oliker , Xiaoye Li , Parry Husbands , Rupak Biswas, Effects of Ordering Strategies and Programming Paradigms on Sparse Matrix Computations, SIAM Review, v.44 n.3, p.373-393, 2002 [doi>10.1137/S00361445003820]
• 15: [15] MPI Web Site : http://www.mpi.org/
• 16: [16] OpenMP Web Site : http//www.openmp.org/
• 17: Rolf Rabenseifner, Communication Bandwidth of Parallel Programming Models on Hybrid Architectures, Proceedings of the 4th International Symposium on High Performance Computing, p.401-412, May 15-17, 2002
• 18: [18] Washio, T., Maruyama, K., Osoda, T., Shimizu, F. and Doi, S. : "Blocking and reordering to achieve highly parallel robust ILU preconditioners", RIKEN Symposium on Linear Algebra and its Applications, The Institute of Physical and Chemical Research, 1999, pp. 42-49.
• 19: [19] Washio, T., Maruyama, K., Osoda, T., Shimizu, F. and Doi, S. : "Efficient implementations of block sparse matrix operations on shared memory vector machines", SNA2000 : The Fourth International Conference on Supercomputing in Nuclear Applications, 2000.
• 20: Y. Saad, Iterative Methods for Sparse Linear Systems, Society for Industrial and Applied Mathematics, Philadelphia, PA, 2003
• 21: [21] Doi, S. and Washio : "Using Multicolor Ordering with Many Colors to Strike a Better Balance between Parallelism and Convergence", RIKEN Symposium on Linear Algebra and its Applications, The Institute of Physical and Chemical Research, 1999, pp. 19-26.
• 22: [22] Kerbyson, D.J., Hoisie, A. and Wasserman, H. "A Comparison Between the Earth Simulator and AlphaServer Systems using Predictive Application Performance Models", LA-UR-02-5222, Los Alamos National Laboratory, USA, 2002.
• 23: [23] Ezure, S., Okuda, H. and Nakajima, K. : "Parallel Mesh Relocation, Parallel Finite Element Analysis, Large-Scale Simulation", RIST/Tokyo GeoFEM Report 2002-012, 2002.
• 24: Yousef Saad , Jun Zhang, BILUTM: A Domain-Based Multilevel Block ILUT Preconditioner for General Sparse Matrices, SIAM Journal on Matrix Analysis and Applications, v.21 n.1, p.279-299, Aug. 1999 [doi>10.1137/S0895479898341268]