An analysis of the impact of MPI overlap and independent progress

ABSTRACT

The overlap of computation and communication has long been considered to be a significant performance benefit for applications. Similarly, the ability of MPI to make independent progress (that is, to make progress on outstanding communication operations while not in the MPI library) is also believed to yield performance benefits. Using an intelligent network interface to offload the work required to support overlap and independent progress is thought to be an ideal solution, but the benefits of this approach have been poorly studied at the application level. This lack of analysis is complicated by the fact that most MPI implementations do not sufficiently support overlap or independent progress. Recent work has demonstrated a quantifiable advantage for an MPI implementation that uses offload to provide overlap and independent progress. This paper extends this previous work by further qualifying the source of the performance advantage (offload, overlap, or independent progress).

REFERENCES

• 1: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• 2: R. Brightwell. A new MPI implementation for Cray SHMEM. Technical report, Sandia National Laboratories.
• 3: R. Brightwell and K. Underwood. Evaluation of an eager protocol optimization for MPI. In Proceedings of EuroPVM/MPI, September 2003.
• 4: R. Brightwell and K. D. Underwood. An analysis of NIC resource usage for offloading MPI. In Proceedings of the 2002 Workshop on Communication Architecture for Clusters, Santa Fe, NM, April 2004.
• 5: R. Brightwell and K. D. Underwood. An initial analysis of the impact of overlap and independent progress for mpi. In submitted, 2004.
• 6: Ron Brightwell , Lance Shuler, Design and Implementation of MPI on Puma Portals, Proceedings of the Second MPI Developers Conference, p.18, July 01-02, 1996
• 7: Cray Research, Inc. SHMEM Technical Note for C, SG-2516 2.3, October 1994.
• 8: Infiniband Trade Association. http://www.innibandta.org, 1999.
• 9: J. Liu, B. Chandrasekaran, J. Wu, W. Jiang, S. Kini, W. Yu, D. Buntinas, P. Wyckoff, and D. K. Panda. Performance comparison of MPI implementations over InfiniBand, Myrinet and Quadrics. In The International Conference for High Performance Computing and Communications (SC2003), November 2003.
• 10: A. B. Maccabe, R. Riesen, and D. W. van Dresser. Dynamic processor modes in Puma. Bulletin of the Technical Committee on Operating Systems and Application Environments (TCOS), 8(2):4--12, 1996.
• 11: Fabrizio Petrini , Wu-chun Feng , Adolfy Hoisie , Salvador Coll , Eitan Frachtenberg, The Quadrics Network: High-Performance Clustering Technology, IEEE Micro, v.22 n.1, p.46-57, January 2002 [doi>10.1109/40.988689]
• 12: Fabrizio Petrini , Darren J. Kerbyson , Scott Pakin, The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.55, November 15-21, 2003
• 13: L. Shuler, C. Jong, R. Riesen, D. van Dresser, A. B. Maccabe, L. A. Fisk, and T. M. Stallcup. The Puma operating system for massively parallel computers. In Proceeding of the 1995 Intel Supercomputer User's Group Conference. Intel Supercomputer User's Group, 1995.
• 14: Timothy G. Mattson , David Scott , Stephen R. Wheat, A TeraFLOP Supercomputer in 1996: The ASCI TFLOP System, Proceedings of the 10th International Parallel Processing Symposium, p.84-93, April 15-19, 1996
• 15: K. D. Underwood and R. Brightwell. The impact of mpi queue usage on mpi latency. In submitted, 2004.
• 16: Jeffrey S. Vetter , Frank Mueller, Communication Characteristics of Large-Scale Scientific Applications for Contemporary Cluster Architectures, Proceedings of the 16th International Parallel and Distributed Processing Symposium, p.96, April 15-19, 2002
• 17: Frederick C. Wong , Richard P. Martin , Remzi H. Arpaci-Dusseau , David E. Culler, Architectural requirements and scalability of the NAS parallel benchmarks, Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM), p.41-es, November 14-19, 1999, Portland, Oregon, United States [doi>10.1145/331532.331573]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management
          Subjects: Message sending

General Terms:
Measurement, Performance

Keywords:
MPI, message passing, overlap