Supercomputer performance evaluation and the Perfect Benchmarks

ABSTRACT

In the past three years, the Perfect BenchmarkTM Suite has evolved from a supercomputer performance evaluation plan, presented by Kuck and Sameh at the 1987 International Conference on Supercomputing, to a vigorous international activity. This paper surveys the current state of this supercomputer performance evaluation effort with particular focus on the adopted methodology. While there has been considerable success in achieving the goals of the plan, some issues remain unresolved, and new questions have surfaced.

REFERENCES

• 1: D. BAILEY AND J. BARTON, The NAS Kernel Benchmark Program, Tech. Rep. 86711, NASA Ames Technical Memorandum, 1985.
• 2: M. BERRY ET AL., The Perfect Club Benchmarks: Effective Performance Evaluation of Supercomputers, The International Journal of Supercomputer Applications, 3 (1989), pp. 5- 40.
• 3: H. CURNOW AND B. WICHMAN, A Synthetic Benchmark~ Comput. :I., 19 (1976), pp. 43-49.
• 4: G.CYBENKO,Networking Pee- 965, Center for Supercomputing Research and Development, University of illinois at Urbana, Technical Report, 1990.
• 5: J. DONGARRA, Performance of Various Computers Using Standard Linear Equations Softwave in a Fortran Environment, Tech. Rep. 23, Argonne National Lab Technical Report, 1988.
• 6: Jack Dongarra , Joanne Martin , Jack Vorlton, Computer benchmarking: paths and pitfalls, IEEE Spectrum, v.24 n.7, p.38-43, July 1987
• 7: R. H. HILL, The Art of Benchmarking, The Spang Robinson Report on Supercomputing and Parallel Processing, 3 (1989).
• 8: P. F. Koss, Application Performance on Supercomputers, Tech. Rep. 847, Center for Supercomputing Research and Development, University of Illinois, January 1989.
• 9: D. J. KucK AND A. H. SAMEH, 21 gupercomputing Performance Evaluation Plan, Tech. Rep. 692, Center for Supercomputing Research and Development, University of Illinois, June 1987.
• 10: A. MAZONY, Performance Measurement Intrusion and Perturbation Analysis, Tech. R.ep. 923, Center for Supercomputing Research and Development, University of Illinois at Urbana, Technical Report, 1990.
• 11: F. McMAI~ON, The Livermore Fortran Kernels: A Test of the Numerical Performance Range, tech. rep., Lawrence Livermore Lab, 1986.
• 12: P. MESSlNA ST AL., Benchmarking Advanced Architecture Computers, Tech. Rep. C3P712, Caltech Concurrent Supercomputing Facility, June 1989.
• 13: L. POINTER, Perfect: Performance Evaluation foe Cost Effective Transformations: Report 2, Tech. Rep. 964, Center for Supercomputing Research and Development, University of Illinois at Urbana, Technical Report, 1990.
• 14: R. H. Saavedra-Barrera , A. J. Smith , E. Miya, Machine Characterization Based on an Abstract High-Level Language Machine, IEEE Transactions on Computers, v.38 n.12, p.1659-1679, December 1989 [doi>10.1109/12.40845]
• 15: J. E. SMX'rH, Charactevizin9 Computer Per- (1988).
• 16: J. UNIEJEWSKI, SPEC Benchmark Suite: Designed for Today's Advanced Systems. SPEC Newsletter, Fall 1989. Volttme 1, Issue 1.
• 17: J. WORLTON, Understanding Supercomputer Benchmarks, Datamation, (1984), pp. 121- 129.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.1 Large and Medium ("Mainframe") Computers
          Subjects: Super (very large) computers

Additional Classification:
  C. Computer Systems Organization

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.2 Installation Management
          Subjects: Benchmarks

General Terms:
Design, Performance, Standardization, Theory