Analysis of redundancy and application balance in the SPEC CPU2006 benchmark suite

ABSTRACT

The recently released SPEC CPU2006 benchmark suite is expected to be used by computer designers and computer architecture researchers for pre-silicon early design analysis. Partial use of benchmark suites by researchers, due to simulation time constraints, compiler difficulties, or library or system call issues is likely to happen; but a random subset can lead to misleading results. This paper analyzes the SPEC CPU2006 benchmarks using performance counter based experimentation from several state of the art systems, and uses statistical techniques such as principal component analysis and clustering to draw inferences on the similarity of the benchmarks and the redundancy in the suite and arrive at meaningful subsets.

The SPEC CPU2006 benchmark suite contains several programs from areas such as artificial intelligence and includes none from the electronic design automation (EDA) application area. Hence there is a concern on the application balance in the suite. An analysis from the perspective of fundamental program characteristics shows that the included programs offer characteristics broader than the EDA programs' space. A subset of 6 integer programs and 8 floating point programs can yield most of the information from the entire suite.

REFERENCES

• 1: M. Alt "Performance Modeling Using Compilers" White paper Intel Corp. http://cache--www.intel.com/cd/00/00/22/64/226491_226491.pdf
• 2: Daniel Citron, MisSPECulation: partial and misleading use of SPEC CPU2000 in computer architecture conferences, Proceedings of the 30th annual international symposium on Computer architecture, June 09-11, 2003, San Diego, California
• 3: IEEE Micro staff, The Use and Abuse of SPEC: An ISCA Panel, IEEE Micro, v.23 n.4, p.73-77, July 2003 [doi>10.1109/MM.2003.1225977]
• 4: K. Dixit: Overview of the SPEC Benchmarks. "The Benchmark Handbook", Chapter 9, 1993.
• 5: J. Dongarra, K. London, S. Moore, P. Mucci, D. Terpstra, "Using PAPI for hardware performance monitoring on Linux Systems" Conference on Linux Clusters: The HPC Revolution, Linux Clusters Institute, June 2001.
• 6: Jozo J. Dujmovic , Ivo Dujmovic, Evolution and evaluation of SPEC benchmarks, ACM SIGMETRICS Performance Evaluation Review, v.26 n.3, p.2-9, Dec. 1998 [doi>10.1145/306225.306228]
• 7: G. Dunteman, Principal Components Analysis, Sage Publications, 1989.
• 8: Lieven Eeckhout , Hans Vandierendonck , Koen De Bosschere, Designing Computer Architecture Research Workloads, Computer, v.36 n.2, p.65-71, February 2003 [doi>10.1109/MC.2003.1178050]
• 9: L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Quantifying the impact of input data sets on program behavior and its applications", Journal of Instruction Level Parallelism, vol 5, pp. 1--33, 2003.
• 10: Ran Giladi , Niv Ahituv, SPEC as a Performance Evaluation Measure, Computer, v.28 n.8, p.33-42, August 1995 [doi>10.1109/2.402073]
• 11: John L. Henning, SPEC CPU2000: Measuring CPU Performance in the New Millennium, Computer, v.33 n.7, p.28-35, July 2000 [doi>10.1109/2.869367]
• 12: John L. Henning, Performance counters and development of SPEC CPU2006, ACM SIGARCH Computer Architecture News, v.35 n.1, March 2007 [doi>10.1145/1241601.1241623]
• 13: Lizy Kurian John , Purnima Vasudevan , Jyotsna Sabarinathan, Workload Characterization: Motivation, Goals and Methodology, Proceedings of the Workload Characterization: Methodology and Case Studies, p.3, November 29-29, 1998
• 14: L. K. John , V. Reddy , P. T. Hulina , L. D. Coraor, Program balance and its impact on high performance RISC architectures, Proceedings of the 1st IEEE Symposium on High-Performance Computer Architecture, p.370, January 22-25, 1995
• 15: Ajay Joshi , Aashish Phansalkar , Lieven Eeckhout , Lizy Kurian John, Measuring Benchmark Similarity Using Inherent Program Characteristics, IEEE Transactions on Computers, v.55 n.6, p.769-782, June 2006 [doi>10.1109/TC.2006.85]
• 16: T. Lafage and A. Seznec, "Choosing Representative Slices of Program Execution for Microarchitecture Simulations: A Preliminary Application to the Data Stream", Workshop on Workload Characterization (WWC-2000), Sept 2000.
• 17: Chi-Keung Luk , Robert Cohn , Robert Muth , Harish Patil , Artur Klauser , Geoff Lowney , Steven Wallace , Vijay Janapa Reddi , Kim Hazelwood, Pin: building customized program analysis tools with dynamic instrumentation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 18: H. McGhan, SPEC CPU2006 Benchmark Suite, Microprocessor Report, October 10, 2006.
• 19: A. Phansalkar, A. Joshi, L. Eeckhout, and L. K. John, "Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites". IEEE International Symposium on Performance Analysis of Systems and Software. March 2005, pp 10--20.
• 20: A. Phansalkar, Joshi, A. L. Eeckhout, and L.K. John "Four Generations of SPEC CPU Benchmarks: What has changed and what has not?", Technical Report TR-041026-1, Laboratory of Computer Architecture, The University of Texas at Austin. 2004.
• 21: J. Reilly. Presentation at IEEE International Symposium on Workload Characterization, Oct 2006 http://www.iiswc.org/iiswc2006/IISWC2006S2.1.pdf
• 22: Timothy Sherwood , Erez Perelman , Greg Hamerly , Brad Calder, Automatically characterizing large scale program behavior, Proceedings of the 10th international conference on Architectural support for programming languages and operating systems, October 05-09, 2002, San Jose, California
• 23: H. Vandierendonck, K. Bosschere, "Many Benchmarks Stress the Same Bottlenecks", Proc. of the Workshop on Computer Architecture Evaluation using Commerical Workloads (CAECW-7), pp. 57--71, 2004.
• 24: Reinhold P. Weicker, An Overview of Common Benchmarks, Computer, v.23 n.12, p.65-75, December 1990 [doi>10.1109/2.62094]
• 25: T. Wenisch, R. Wunderlich, B. Falsafi, and J. Hoe, "Applying SMARTS to SPEC CPU2000", CALCM Technical Report 2003-1, Carnegie Mellon University, June 2003.
• 26: Joshua J. Yi , David J. Lilja, Simulation of Computer Architectures: Simulators, Benchmarks, Methodologies, and Recommendations, IEEE Transactions on Computers, v.55 n.3, p.268-280, March 2006 [doi>10.1109/TC.2006.44]
• 27: J. Yi, R. Sendag, L. Eeckhout, A. Joshi, D. Lilja, and L. K. John, "Evaluating Benchmark Subsetting Approaches" International Symposium on Workload Characterization, October 2006, pp 93--104.
• 28: Joshua J. Yi , David J. Lilja , Douglas M. Hawkins, A Statistically Rigorous Approach for Improving Simulation Methodology, Proceedings of the 9th International Symposium on High-Performance Computer Architecture, p.281, February 08-12, 2003

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Modeling techniques; Performance attributes

General Terms:
Experimentation, Measurement, Performance

Keywords:
SPEC, benchmark, clustering, microprocessor performance counters