The exigency of benchmark and compiler drift: designing tomorrow's processors with yesterday's tools

ABSTRACT

Due to the amount of time required to design a new processor, one set of benchmark programs may be used during the design phase while another may be the standard when the design is finally delivered. Using one benchmark suite to design a processor while using a different, presumably more current, suite to evaluate its ultimate performance may lead to sub-optimal design decisions if there are large differences between the characteristics of the two suites and their respective compilers. We call this changes across time "drift". To evaluate the impact of using yesterday's benchmark and compiler technology to design tomorrow's processors, we compare common benchmarks from the SPEC 95 and SPEC 2000 benchmark suites. Our results yield three key conclusions. First, we show that the amount of drift, for common programs in successive SPEC benchmark suites, is significant. In SPEC 2000, the main memory access time is a far more significant performance bottleneck than in SPEC 95, while less significant SPEC 2000 performance bottlenecks include the L2 cache latency, the L1 I-cache size, and the number of reorder buffer entries. Second, using two different statistical techniques, we show that compiler drift is not as significant as benchmark drift. Third, we show that benchmark and compiler drift can have a significant impact on the final design decisions. Specifically, we use a one-parameter-at-a-time optimization algorithm to design two different year-2000 processors, one optimized for SPEC 95 and the other optimized for SPEC 2000, using the energy-delay product (EDP) as the optimization criterion. The results show that using SPEC 95 to design a year-2000 processor results in an 18.5% larger EDP and a 20.8% higher CPI than using the SPEC 2000 benchmarks to design the corresponding processor. Finally, we make a few recommendations to help computer architects minimize the effects of benchmark and compiler drift.

REFERENCES

• 1: Bell Jr., R. and John, L., "The Case for Automatic Synthesis of Miniature Benchmarks," In Proceedings of the Workshop on Modeling, Benchmarking, and Simulation (MoBS '05) (Madison, WI, USA, June 4--8, 2005), 88--97.
• 2: Robert H. Bell, Jr. , Lizy K. John, Improved automatic testcase synthesis for performance model validation, Proceedings of the 19th annual international conference on Supercomputing, June 20-22, 2005, Cambridge, Massachusetts [doi>10.1145/1088149.1088164]
• 3: David Brooks , Vivek Tiwari , Margaret Martonosi, Wattch: a framework for architectural-level power analysis and optimizations, Proceedings of the 27th annual international symposium on Computer architecture, p.83-94, June 2000, Vancouver, British Columbia, Canada
• 4: David M. Brooks , Pradip Bose , Stanley E. Schuster , Hans Jacobson , Prabhakar N. Kudva , Alper Buyuktosunoglu , John-David Wellman , Victor Zyuban , Manish Gupta , Peter W. Cook, Power-Aware Microarchitecture: Design and Modeling Challenges for Next-Generation Microprocessors, IEEE Micro, v.20 n.6, p.26-44, November 2000 [doi>10.1109/40.888701]
• 5: Doug Burger , Todd M. Austin, The SimpleScalar tool set, version 2.0, ACM SIGARCH Computer Architecture News, v.25 n.3, p.13-25, June 1997 [doi>10.1145/268806.268810]
• 6: Calder, B., Grunwald, D., and Zorn, B., "Quantifying Behavioral Differences Between C and C++ Programs," Journal of Programming Languages, 2, 4, (1994), 313--351.
• 7: Carlton, A., "Lessons Learned from 072.sc", SPEC Newsletter, (Mar. 1995).
• 8: Youngsoo Choi , Allan Knies , Luke Gerke , Tin-Fook Ngai, The impact of if-conversion and branch prediction on program execution on the Intel® Itanium™ processor, Proceedings of the 34th annual ACM/IEEE international symposium on Microarchitecture, December 01-05, 2001, Austin, Texas
• 9: Jozo J. Dujmovic , Ivo Dujmovic, Evolution and evaluation of SPEC benchmarks, ACM SIGMETRICS Performance Evaluation Review, v.26 n.3, p.2-9, Dec. 1998 [doi>10.1145/306225.306228]
• 10: Lieven Eeckhout , Hans Vandierendonck , Koenraad De Bosschere, Workload Design: Selecting Representative Program-Input Pairs, Proceedings of the 2002 International Conference on Parallel Architectures and Compilation Techniques, p.83-94, September 22-25, 2002
• 11: Greg Hamerly , Erez Perelman , Brad Calder, How to use SimPoint to pick simulation points, ACM SIGMETRICS Performance Evaluation Review, v.31 n.4, p.25-30, March 2004 [doi>10.1145/1054907.1054913]
• 12: John L. Hennessy , David A. Patterson, Computer Architecture: A Quantitative Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2003
• 13: John L. Henning, SPEC CPU2000: Measuring CPU Performance in the New Millennium, Computer, v.33 n.7, p.28-35, July 2000 [doi>10.1109/2.869367]
• 14: Henning, J., "SPEC CPU2000 Memory Footprint," http://www.spec.org/cpu2000/analysis/memory
• 15: Lilja, D., "Measuring Computer Performance," Cambridge University Press, New York, NY, 2000.
• 16: Phansalkar, A., Joshi, A., Eeckhout, L., and John, L., "Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites," Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS '05) (Austin, TX, March 20--22, 2005), 10--20.
• 17: Plackett, R. and Burman, J. "The Design of Optimum Multifactorial Experiments," Biometrika, 33, 4, (June 1946), 305--325.
• 18: Timothy Sherwood , Erez Perelman , Greg Hamerly , Brad Calder, Automatically characterizing large scale program behavior, Proceedings of the 10th international conference on Architectural support for programming languages and operating systems, October 05-09, 2002, San Jose, California
• 19: http://www.cs.ucsd.edu/~calder/simpoint
• 20: Kevin Skadron , Margaret Martonosi , David I. August , Mark D. Hill , David J. Lilja , Vijay S. Pai, Challenges in Computer Architecture Evaluation, Computer, v.36 n.8, p.30-36, August 2003 [doi>10.1109/MC.2003.1220579]
• 21: http://www.spec.org
• 22: H. Vandierendonck , K. De Bosschere, Eccentric and fragile benchmarks, Proceedings of the 2004 IEEE International Symposium on Performance Analysis of Systems and Software, p.2-11, March 10-12, 2004
• 23: http://www.eecs.umich.edu/~chriswea/benchmarks/SPEC 2000.html
• 24: Weicker, R., "An Example of Benchmark Obsolescence: 023.eqntott," SPEC Newsletter, (Dec. 1995).
• 25: Wm. A. Wulf , Sally A. McKee, Hitting the memory wall: implications of the obvious, ACM SIGARCH Computer Architecture News, v.23 n.1, p.20-24, March 1995 [doi>10.1145/216585.216588]
• 26: Joshua J. Yi , David J. Lilja , Douglas M. Hawkins, A Statistically Rigorous Approach for Improving Simulation Methodology, Proceedings of the 9th International Symposium on High-Performance Computer Architecture, p.281, February 08-12, 2003
• 27: Joshua J. Yi , Sreekumar V. Kodakara , Resit Sendag , David J. Lilja , Douglas M. Hawkins, Characterizing and Comparing Prevailing Simulation Techniques, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.266-277, February 12-16, 2005 [doi>10.1109/HPCA.2005.8]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Measurement techniques

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.6 Simulation Output Analysis

General Terms:
Measurement, Performance

Keywords:
benchmark drift, compiler drift, microprocessor design