How Well Can Simple Metrics Represent the Performance of HPC Applications?

ABSTRACT

In this paper, a systematic study of the effects of complexity of prediction methodology on its accuracy for a set of real applications on a variety of HPC systems is performed. Results indicate that the use of any single, simple synthetic metric to predict performance does an inadequate job, and the use of a linear combination of these simple metrics with optimized weights also performs poorly. Better, however, are methodologies that rely on the convolution of an application "transfer function" based on tracing information with system performance data measured by simple benchmarks. This latter methodology can predict performance with an average accuracy of 80%, based on the current work.

REFERENCES

• 1: 1. Top500, www.top500.org.
• 2: 2. J. Dongarra, P. Luszczek, & A. Petitet, "The LINPACK benchmark: past, present and future", Concurrency and Computation: Practice and Experience, vol. 15, pp. 1- 18, 2003.
• 3: 3. E. Joseph, C. G. Willard, M. Swenson, & D. Goldfarb, "A new HPC technical computing benchmark: the IDC balanced rating", IDC Bulletin W.
• 4: 4. High Performance Computing Modernization Program, www.hpcmo.hpc.mil.
• 5: 5. J. McCalpin, "Memory bandwidth and machine balance in current high performance computers", IEEE Technical Committee on Computer Architecture Newsletter.
• 6: 6. HPC Challenge Benchmarks, http://icl.cs.utk.edu/hpcc/.
• 7: 7. PMaC HPC Benchmark Suite, http://www.sdsc.edu/ pmac/.
• 8: John L. Gustafson , Rajat Todi, Conventional Benchmarks as a Sample of the Performance Spectrum, Proceedings of the Thirty-First Annual Hawaii International Conference on System Sciences-Volume 7, p.514, January 06-09, 1998 [doi>10.1109/HICSS.1998.649247]
• 9: 9. L. Carrington, A. Snavely, N. Wolter, & X. Gao, "A performance prediction framework for scientific applications", Workshop on Performance Modeling and Analysis-ICCS, Melbourne, 2003.
• 10: 10. A. Snavely, X. Gao, C. Lee, N. Wolter, & J. Labarta, "Performance modeling of HPC applications", Parallel Computing, Dresden, 2003.
• 11: S. Browne , J. Dongarra , N. Garner , K. London , P. Mucci, A scalable cross-platform infrastructure for application performance tuning using hardware counters, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.42-es, November 04-10, 2000, Dallas, Texas, United States
• 12: Jeffrey K. Hollingsworth , Allan Snavely , Simone Sbaraglia , K. Ekanadham, EMPS: An Environment for Memory Performance Studies, Proceedings of the 19th IEEE International Parallel and Distributed Processing Symposium (IPDPS'05) - Workshop 10, p.223.2, April 04-08, 2005 [doi>10.1109/IPDPS.2005.196]
• 13: Allan Snavely , Laura Carrington , Nicole Wolter , Jesus Labarta , Rosa Badia , Avi Purkayastha, A framework for performance modeling and prediction, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-17, November 16, 2002, Baltimore, Maryland
• 14: 14. R. Badia, G. Rodriguez, & J. Labarta, "Deriving analytical models from a limited number of runs", Parallel Computing, Dresden, 2003.
• 15: 15. Ad Emmen, "IDC reports latest supercomputer rankings based on the IDC balanced rating test", Primeur Monthly, May 16, 2002, http://www.hoise. com/primeur/02/articles/monthly/AE-PR-06-02-45. html.
• 16: 16. D. Bailey, J. Barton, T. Lasinski, H. Simon, "The NAS parallel benchmarks", International Journal of Supercomputer Applications, 1991.
• 17: 17. SPEC, http://www.spec.org/.
• 18: Gabriel Marin , John Mellor-Crummey, Cross-architecture performance predictions for scientific applications using parameterized models, Proceedings of the joint international conference on Measurement and modeling of computer systems, June 10-14, 2004, New York, NY, USA
• 19: 19. L. Svobodova, Computer System Performance Measurement and Evaluation Methods: Analysis and Applications (Elsevier, N. Y. 1976).
• 20: 20. R. S., Ballansc, J. A. Cocke, and H. G. Kolsky, The Lookahead Unit, Planning a Computer System, (McGraw-Hill, New York, 1962).
• 21: 21. L. T. Boland, G. D. Granito, A. V. Marcotte, B. V. Messina, and J. W. Smith, "The IBM system 360/Model9: Storage System", IBMJ. Res. And Develop., vol. 11, pp. 54-79, 1967.
• 22: 22. D. Burger, T. M. Austin, and S. Bennett, "Evaluating future microprocessors: The simplescalar tool set", Tech. Rep. CS-TR-1996-1308, University of Wisconsin-Madison, 1996.
• 23: 23. J. O. Murphey and R. M. Wade, "The IBM 360/195", Datamation, vol. 16:4, pp. 72-79, 1970.
• 24: 24. G. S. Tjaden and M. J. Flynn, "Detection and Parallel Execution of Independent Instructions", IEEE Trans. Comptrs., vol. C-19 pp. 889-895, 1970.
• 25: Jack L. Lo , Joel S. Emer , Henry M. Levy , Rebecca L. Stamm , Dean M. Tullsen , S. J. Eggers, Converting thread-level parallelism to instruction-level parallelism via simultaneous multithreading, ACM Transactions on Computer Systems (TOCS), v.15 n.3, p.322-354, Aug. 1997 [doi>10.1145/263326.263382]
• 26: Babak Falsafi , David A. Wood, Modeling cost/performance of a parallel computer simulator, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.7 n.1, p.104-130, Jan. 1997 [doi>10.1145/244804.244808]
• 27: Jeff Gibson , Robert Kunz , David Ofelt , Mark Horowitz , John Hennessy , Mark Heinrich, FLASH vs. (Simulated) FLASH: closing the simulation loop, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.49-58, November 2000, Cambridge, Massachusetts, United States
• 28: Alan Jay Smith , Rafael H. Saavedra, Measuring Cache and TLB Performance and Their Effect on Benchmark Runtimes, IEEE Transactions on Computers, v.44 n.10, p.1223-1235, October 1995 [doi>10.1109/12.467697]
• 29: Rafael H. Saavedra , Alan J. Smith, Analysis of benchmark characteristics and benchmark performance prediction, ACM Transactions on Computer Systems (TOCS), v.14 n.4, p.344-384, Nov. 1996 [doi>10.1145/235543.235545]
• 30: Rafael H. Saavedra , Alan Jay Smith, Performance Characterization of Optimizing Compilers, IEEE Transactions on Software Engineering, v.21 n.7, p.615-628, July 1995 [doi>10.1109/32.392982]
• 31: Celso L. Mendes , Daniel A. Reed, Integrated Compilation and Scalability Analysis for Parallel Systems, Proceedings of the 1998 International Conference on Parallel Architectures and Compilation Techniques, p.385, October 12-18, 1998
• 32: 32. C. L. Mendes and D. A. Reed, "Performance Stability and Prediction", IEEE /USP International Workshop on High Performance Computing, 1994.
• 33: Jens Simon , Jens-Michael Wierum, Accurate Performance Prediction for Assively Parallel Systems and Its Applications, Proceedings of the Second International Euro-Par Conference on Parallel Processing-Volume II, p.675-688, August 26-29, 1996
• 34: Mark E. Crovella , Thomas J. LeBlanc, Parallel performance prediction using lost cycles analysis, Proceedings of the 1994 ACM/IEEE conference on Supercomputing, November 14-18, 1994, Washington, D.C. [doi>10.1145/602770.602870]
• 35: Zhichen Xu , Xiaodong Zhang , Lin Sun, Semi-empirical multiprocessor performance predictions, Journal of Parallel and Distributed Computing, v.39 n.1, p.14-28, Nov. 25, 1996 [doi>10.1006/jpdc.1996.0151]
• 36: Gheith A. Abandah , Edward S. Davidson, Modeling the Communication Performance of the IBM SP2, Proceedings of the 10th International Parallel Processing Symposium, p.249-257, April 15-19, 1996
• 37: Eric L. Boyd , Waqar Azeem , Hsien-Hsin Lee , Tien-Pao Shih , Shih-Hao Hung , Edward S. Davidson, A Hierarchical Approach to Modeling and Improving the Performance of Scientific Applications on the KSR1, Proceedings of the 1994 International Conference on Parallel Processing, p.188-192, August 15-19, 1994 [doi>10.1109/ICPP.1994.30]
• 38: Adolfy Hoisie , Olaf M. Lubeck , Harvey J. Wasserman, Performance Analysis of Wavefront Algorithms on Very-Large Scale Distributed Systems, Workshop on Wide Area Networks and High Performance Computing, p.171-187, September 01-05, 1998
• 39: Adolfy Hoisie , Olaf Lubeck , Harvey Wasserman, Scalability Analysis of Multidimensional Wavefront Algorithms on Large-Scale SMP Clusters, Proceedings of the The 7th Symposium on the Frontiers of Massively Parallel Computation, p.4, February 21-25, 1999
• 40: 40. D. J. Kerbyson, A. Hoisie, and H. J. Wasserman, "Modeling the Performance of Large-Scale Systems", Keynote paper, UK Performance Engineering Workshop (UKPEW03), July, 2003.
• 41: Yong Luo , Olaf M. Lubeck , Harvey Wasserman , Federico Bassetti , Kirk W. Cameron, Development and validation of a hierarchical memory model incorporating CPU- and memory-operation overlap model, Proceedings of the 1st international workshop on Software and performance, p.152-163, October 12-16, 1998, Santa Fe, New Mexico, United States [doi>10.1145/287318.287351]
• 42: 42. A. Spooner and D. Kerbyson, "Identification of Performance Characteristics from Multi-view Trace Analysis", Proc. Of Int. Conf. On Computational Science (ICCS), part 3 2659, pp. 936-945, 2003.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Modeling techniques

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies; Performance attributes
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.1 Large and Medium ("Mainframe") Computers
          Subjects: Super (very large) computers

General Terms:
Design, Experimentation, Measurement, Performance