Optimizing threaded MPI execution on SMP clusters

ABSTRACT

Our previous work has shown that using threads to execute MPI programs can yield great performance gain on multiprogrammed shared-memory machines. This paper investigates the design and implementation of a thread-based MPI system on SMP clusters. Our study indicates that with a proper design for threaded MPI execution, both point-to-point and collective communication performance can be improved substantially, compared to a process-based MPI implementation in a cluster environment. Our contribution includes a hierarchy-aware and adaptive communication scheme for threaded MPI execution and a thread-safe network device abstraction that uses event-driven synchronization and provides separated collective and point-to-point communication channels. This paper describes the implementation of our design and illustrates its performance advantage on a Linux SMP cluster.

REFERENCES

• 1: Thomas E. Anderson , David E. Culler , David A. Patterson , and the NOW team, A Case for NOW (Networks of Workstations), IEEE Micro, v.15 n.1, p.54-64, February 1995 [doi>10.1109/40.342018]
• 2: Donald J.Becker,Thomas Sterling,D niel Sav rese, John E.Dorband,Ud y A.Ranawak,and Charles V. P cker.BEOWULF:parallel workstation for scienti .c computation.In Proceeding of International Conference on Parallel Proce ing ,1995.More Beowulf papers re in http://www.beowulf.org.
• 3: G.Burns,,V.R diy ,R.Daoud,and R.Machiraju.All About Trollius,1990.Ohio Supercomputer Center.
• 4: G.Burns,R.Daoud,and J.Vaigl.LAM:An Open Cluster Environment for MPI.Ohio Supercomputer Center.
• 5: David E. Culler , Anoop Gupta , Jaswinder Pal Singh, Parallel Computer Architecture: A Hardware/Software Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 6: A. J. Ferrari , V. S. Sunderam, TPVM: distributed concurrent computing with lightweight processes, Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing - HPDC '95, p.211, August 02-04, 1995
• 7: W.Gropp and E.Lusk.An Abstract Device Definition to Support the Implementation of High-Level Point-to-Point Message P ssing Interface.Technical Report MCS-P392-1193,Argonne N tional L boratory,1994.
• 8: William Gropp , Ewing Lusk , Nathan Doss , Anthony Skjellum, A high-performance, portable implementation of the MPI message passing interface standard, Parallel Computing, v.22 n.6, p.789-828, Sept. 1996 [doi>10.1016/0167-8191(96)00024-5]
• 9: Parry Husbands , James C. Hoe, MPI-StarT: delivering network performance to numerical applications, Proceedings of the 1998 ACM/IEEE conference on Supercomputing (CDROM), p.1-15, November 07-13, 1998, San Jose, CA
• 10: Thilo Kielmann , Rutger F. H. Hofman , Henri E. Bal , Aske Plaat , Raoul A. F. Bhoedjang, MagPIe: MPI's collective communication operations for clustered wide area systems, Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming, p.131-140, May 04-06, 1999, Atlanta, Georgia, United States
• 11: Mario Lauria , Andrew Chien, MPI-FM: high performance MPI on workstation clusters, Journal of Parallel and Distributed Computing, v.40 n.1, p.4-18, Jan. 10, 1997 [doi>10.1006/jpdc.1996.1264]
• 12: Steven S. Lumetta , Alan M. Mainwaring , David E. Culler, Multi-protocol active messages on a cluster of SMP's, Proceedings of the 1997 ACM/IEEE conference on Supercomputing (CDROM), p.1-22, November 15-21, 1997, San Jose, CA [doi>10.1145/509593.509596]
• 13: L.MPI.Home page of lam "local rea multicomputer. "http://www.lsc.nd.edu/lam/".
• 14: MPI-Forum.MPI Forum,1999.http://www.mpi-forum.org.
• 15: Sundeep Prakash , Rajive L. Bagrodia, MPI-SIM: using parallel simulation to evaluate MPI programs, Proceedings of the 30th conference on Winter simulation, p.467-474, December 13-16, 1998, Washington, D.C., United States
• 16: Kai Shen , Hong Tang , Tao Yang, Adaptive two-level thread management for fast MPI execution on shared memory machines, Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM), p.49-es, November 14-19, 1999, Portland, Oregon, United States [doi>10.1145/331532.331581]
• 17: Steve Sistare , Rolf vandeVaart , Eugene Loh, Optimization of MPI collectives on clusters of large-scale SMP's, Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM), p.23-es, November 14-19, 1999, Portland, Oregon, United States [doi>10.1145/331532.331555]
• 18: Hong Tang , Kai Shen , Tao Yang, Program transformation and runtime support for threaded MPI execution on shared-memory machines, ACM Transactions on Programming Languages and Systems (TOPLAS), v.22 n.4, p.673-700, July 2000 [doi>10.1145/363911.363920]
• 19: F.Wong and D.Culler.Message passing interface on ctive messages."http://now.cs.berkeley.edu/F stcomm/MPI/".
• 20: H.Zhou and A.Geist.LPVM:step towards multithread PVM.Concurrency -Practice and Experience ,1997.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS

General Terms:
Design, Performance

Keywords:
MPI, SMP clusters, communication optimization, multi-threading