Distilling the essence of proprietary workloads into miniature benchmarks

ABSTRACT

Benchmarks set standards for innovation in computer architecture research and industry product development. Consequently, it is of paramount importance that these workloads are representative of real-world applications. However, composing such representative workloads poses practical challenges to application analysis teams and benchmark developers (1) real-world workloads are intellectual property and vendors hesitate to share these proprietary applications; and (2) porting and reducing these applications to benchmarks that can be simulated in a tractable amount of time is a nontrivial task. In this paper, we address this problem by proposing a technique that automatically distills key inherent behavioral attributes of a proprietary workload and captures them into a miniature synthetic benchmark clone. The advantage of the benchmark clone is that it hides the functional meaning of the code but exhibits similar performance characteristics as the target application. Moreover, the dynamic instruction count of the synthetic benchmark clone is substantially shorter than the proprietary application, greatly reducing overall simulation time for SPEC CPU, the simulation time reduction is over five orders of magnitude compared to entire benchmark execution. Using a set of benchmarks representative of general-purpose, scientific, and embedded applications, we demonstrate that the power and performance characteristics of the synthetic benchmark clone correlate well with those of the original application across a wide range of microarchitecture configurations.

REFERENCES

• 1: Robert H. Bell, Jr. , Lizy K. John, Improved automatic testcase synthesis for performance model validation, Proceedings of the 19th annual international conference on Supercomputing, June 20-22, 2005, Cambridge, Massachusetts [doi>10.1145/1088149.1088164]
• 2: Bell, R. and John, L. 2006. Efficient power analysis using synthetic testcases. In Proceedings of the IEEE International Symposium on Workload Characterization. 110--118.
• 3: David Brooks , Vivek Tiwari , Margaret Martonosi, Wattch: a framework for architectural-level power analysis and optimizations, Proceedings of the 27th annual international symposium on Computer architecture, p.83-94, June 2000, Vancouver, British Columbia, Canada
• 4: Burger, D. and Austin, T. 1997. The SimpleScalar toolset, version 2.0, University of Wisconsin-Madison, Computer Sciences Department Tech. Rep. #1342.
• 5: Dhruba Chandra , Fei Guo , Seongbeom Kim , Yan Solihin, Predicting Inter-Thread Cache Contention on a Chip Multi-Processor Architecture, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.340-351, February 12-16, 2005 [doi>10.1109/HPCA.2005.27]
• 6: Peter M. Chen , David A. Patterson, A new approach to I/O performance evaluation: self-scaling I/O benchmarks, predicted I/O performance, Proceedings of the 1993 ACM SIGMETRICS conference on Measurement and modeling of computer systems, p.1-12, May 10-14, 1993, Santa Clara, California, United States
• 7: Coleman, C. 1998. Using Inline Assembly with Gcc. See www.cs.virginia.edu/~clc5q/gcc-inline-asm.pdf.
• 8: Jamison D. Collins , Hong Wang , Dean M. Tullsen , Christopher Hughes , Yong-Fong Lee , Dan Lavery , John P. Shen, Speculative precomputation: long-range prefetching of delinquent loads, Proceedings of the 28th annual international symposium on Computer architecture, p.14-25, June 30-July 04, 2001, Göteborg, Sweden
• 9: Conte, T. and Hwu, W.-M. 1990. Benchmark characterization for experimental system evaluation. In Proceedings of the 1990 Hawaii International Conference on System Sciences (HICSS). Architecture Track, vol. I. 6--16.
• 10: Thomas M. Conte , Mary Ann Hirsch , Kishore N. Menezes, Reducing State Loss For Effective Trace Sampling of Superscalar Processors, Proceedings of the 1996 International Conference on Computer Design, VLSI in Computers and Processors, p.468-477, October 07-09, 1996
• 11: Curnow, H. and Wichman, B. 1976. A synthetic benchmark. Comput. J. 19, 1, 43--49.
• 12: P. K. Dubey , G. B. Adams, III , M. J. Flynn, Instruction Window Size Trade-Offs and Characterization of Program Parallelism, IEEE Transactions on Computers, v.43 n.4, p.431-442, April 1994 [doi>10.1109/12.278481]
• 13: Lieven Eeckhout , Koenraad De Bosschere, Hybrid Analytical-Statistical Modeling for Efficiently Exploring Architecture and Workload Design Spaces, Proceedings of the 2001 International Conference on Parallel Architectures and Compilation Techniques, p.25, September 08-12, 2001
• 14: Lieven Eeckhout , Sebastien Nussbaum , James E. Smith , Koen De Bosschere, Statistical Simulation: Adding Efficiency to the Computer Designer's Toolbox, IEEE Micro, v.23 n.5, p.26-38, September 2003 [doi>10.1109/MM.2003.1240210]
• 15: Lieven Eeckhout , Robert H. Bell Jr. , Bastiaan Stougie , Koen De Bosschere , Lizy K. John, Control Flow Modeling in Statistical Simulation for Accurate and Efficient Processor Design Studies, Proceedings of the 31st annual international symposium on Computer architecture, p.350, June 19-23, 2004, München, Germany
• 16: Stijn Eyerman , Lieven Eeckhout , Koen De Bosschere, Efficient design space exploration of high performance embedded out-of-order processors, Proceedings of the conference on Design, automation and test in Europe: Proceedings, March 06-10, 2006, Munich, Germany
• 17: Domenico Ferrari, On the foundations of artificial workload design, Proceedings of the 1984 ACM SIGMETRICS conference on Measurement and modeling of computer systems, p.8-14, January 1984
• 18: Davy Genbrugge , Lieven Eeckhout, Memory Data Flow Modeling in Statistical Simulation for the Efficient Exploration of Microprocessor Design Spaces, IEEE Transactions on Computers, v.57 n.1, p.41-54, January 2008 [doi>10.1109/TC.2007.70783]
• 19: Haungs, M., Sallee, P., and Farrens, M. 2000. Branch transition rate: A new metric for improved branch classification analysis. In Proceedings of the International Symposium on High Performance Computer Architecture. 241--250.
• 20: John L. Henning, SPEC CPU2000: Measuring CPU Performance in the New Millennium, Computer, v.33 n.7, p.28-35, July 2000 [doi>10.1109/2.869367]
• 21: Hsieh, C. and Pedram, M. 1998. Microprocessor power estimation using profile-driven program synthesis. IEEE Trans. Comput. Aided Design Integrated Circ. Sys. 17, 11, 1080--1089.
• 22: Iyengar, V. and Trevillyan, L. 1996. Evaluation and generation of reduced traces for benchmarks. Tech. Rep. RC20610. IBM Research Division. T. J. Watson Research Center.
• 23: Vijay S. Iyengar , Louise H. Trevillyan , Pradip Bose, Representative Traces for Processor Models with Infinite Cache, Proceedings of the 2nd IEEE Symposium on High-Performance Computer Architecture, p.62, February 03-07, 1996
• 24: Joshi, A., Eeckhout, L., Bell, R., and John, L. 2006a. Performance cloning: A technique for disseminating proprietary applications as benchmarks. In Proceedings of the IEEE International Symposium on Workload Characterization. 105--115.
• 25: Joshi, A., Yi, J., Bell, R., Eeckhout, L., John, L., and Lilja, D. 2006b. Evaluating the efficacy of statistical simulation for design space exploration. In Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software. 70--79.
• 26: Thierry Lafage , André Seznec, Choosing representative slices of program execution for microarchitecture simulations: a preliminary application to the data stream, Workload characterization of emerging computer applications, Kluwer Academic Publishers, Norwell, MA, 2001
• 27: Chi-Keung Luk , Robert Cohn , Robert Muth , Harish Patil , Artur Klauser , Geoff Lowney , Steven Wallace , Vijay Janapa Reddi , Kim Hazelwood, Pin: building customized program analysis tools with dynamic instrumentation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 28: Derek B. Noonburg , John P. Shen, Theoretical modeling of superscalar processor performance, Proceedings of the 27th annual international symposium on Microarchitecture, p.52-62, November 30-December 02, 1994, San Jose, California, United States [doi>10.1145/192724.192730]
• 29: Sébastien Nussbaum , James E. Smith, Modeling Superscalar Processors via Statistical Simulation, Proceedings of the 2001 International Conference on Parallel Architectures and Compilation Techniques, p.15-24, September 08-12, 2001
• 30: Mark Oskin , Frederic T. Chong , Matthew Farrens, HLS: combining statistical and symbolic simulation to guide microprocessor designs, Proceedings of the 27th annual international symposium on Computer architecture, p.71-82, June 2000, Vancouver, British Columbia, Canada
• 31: J. Ringenberg , C. Pelosi , D. Oehmke , T. Mudge, Intrinsic Checkpointing: A Methodology for Decreasing Simulation Time Through Binary Modification, Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software, 2005, p.78-88, March 20-22, 2005 [doi>10.1109/ISPASS.2005.1430561]
• 32: Rafael H. Saavedra , Alan J. Smith, Analysis of benchmark characteristics and benchmark performance prediction, ACM Transactions on Computer Systems (TOCS), v.14 n.4, p.344-384, Nov. 1996 [doi>10.1145/235543.235545]
• 33: Timothy Sherwood , Erez Perelman , Greg Hamerly , Brad Calder, Automatically characterizing large scale program behavior, Proceedings of the 10th international conference on Architectural support for programming languages and operating systems, October 05-09, 2002, San Jose, California
• 34: SimPoint Website. http://www-cse.ucsd.edu/~calder/simpoint/
• 35: Kevin Skadron , Margaret Martonosi , David I. August , Mark D. Hill , David J. Lilja , Vijay S. Pai, Challenges in Computer Architecture Evaluation, Computer, v.36 n.8, p.30-36, August 2003 [doi>10.1109/MC.2003.1220579]
• 36: Sorenson, E. and Flanagan, J. 2002. Evaluating synthetic trace models using locality surfaces. In Proceedings of the IEEE International Workshop on Workload Characterization. 23--33.
• 37: K. Sreenivasan , A. J. Kleinman, On the construction of a representative synthetic workload, Communications of the ACM, v.17 n.3, p.127-133, March 1974 [doi>10.1145/360860.360863]
• 38: Srivastava, A. and Eustace, A. 1994. ATOM: A system for building customized program analysis tools. Tech. Rep. 94/2, Western Research Lab, Compaq (Mar.).
• 39: Artour Stoutchinin , José N. Amaral , Guang R. Gao , James C. Dehnert , Suneel Jain , Alban Douillet, Speculative Prefetching of Induction Pointers, Proceedings of the 10th International Conference on Compiler Construction, p.289-303, April 02-06, 2001
• 40: Reinhold P. Weicker, An Overview of Common Benchmarks, Computer, v.23 n.12, p.65-75, December 1990 [doi>10.1109/2.62094]
• 41: Reinhold P. Weicker, Dhrystone: a synthetic systems programming benchmark, Communications of the ACM, v.27 n.10, p.1013-1030, Oct 1984 [doi>10.1145/358274.358283]
• 42: Roland E. Wunderlich , Thomas F. Wenisch , Babak Falsafi , James C. Hoe, SMARTS: accelerating microarchitecture simulation via rigorous statistical sampling, Proceedings of the 30th annual international symposium on Computer architecture, June 09-11, 2003, San Diego, California
• 43: Wing Shing Wong , Robert J. T. Morris, Benchmark Synthesis Using the LRU Cache Hit Function, IEEE Transactions on Computers, v.37 n.6, p.637-645, June 1988 [doi>10.1109/12.2202]
• 44: Youfeng Wu, Efficient discovery of regular stride patterns in irregular programs and its use in compiler prefetching, Proceedings of the ACM SIGPLAN 2002 Conference on Programming language design and implementation, June 17-19, 2002, Berlin, Germany

INDEX TERMS

Primary Classification:
  B. Hardware
  B.8 Performance and Reliability

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Design, Experimentation, Performance

Keywords:
Benchmarks, benchmark cloning, workload characterization