Measuring Experimental Error in Microprocessor Simulation

ABSTRACT

Abstract: We measure the experimental error that arises from the use of non-validated simulators in computer architecture research, with the goal of increasing the rigor of simulation- based studies. We describe the methodology that we used to validate a microprocessor simulator against a Compaq DS-10L workstation, which contains an Alpha 21264 processor. Our evaluation suite consists of a set of 21 microbenchmarks that stress different aspects of the 21264 microarchitecture. Using the microbenchmark suite as the set of workloads, we describe how we reduced our simulator error to an arithmetic mean of 2%, and include details about the specific aspects of the pipeline that required extra care to reduce the error. We show how these low-level optimizations reduce average error from 40% to less than 20% on macrobenchmarks drawn from the SPEC2000 suite. Finally, we examine the degree to which performance optimizations are stable across different simulators, showing that researchers would draw different conclusions, in some cases, if using validated simulators.

REFERENCES

• 1: Vikas Agarwal , M. S. Hrishikesh , Stephen W. Keckler , Doug Burger, Clock rate versus IPC: the end of the road for conventional microarchitectures, Proceedings of the 27th annual international symposium on Computer architecture, p.248-259, June 2000, Vancouver, British Columbia, Canada
• 2: Bryan Black , John Paul Shen, Calibration of Microprocessor Performance Models, Computer, v.31 n.5, p.59-65, May 1998 [doi>10.1109/2.675637]
• 3: Pradip Bose , Thomas M. Conte, Performance Analysis and Its Impact on Design, Computer, v.31 n.5, p.41-49, May 1998 [doi>10.1109/2.675632]
• 4: [4] Doug Burger and Todd M. Austin. The simplescalar tool set version 2.0. Technical Report 1342, Department of Computer Sciences, University of Wisconsin-Madison, June 1997.
• 5: [5] Compaq Computer Corporation. Alpha 21264 Microprocessor Hardware Reference Manual, July 1999.
• 6: [6] Compaq Computer Corporation. Compiler Writer's Guide for the Alpha 21264, 1999.
• 7: José-Lorenzo Cruz , Antonio González , Mateo Valero , Nigel P. Topham, Multiple-banked register file architectures, Proceedings of the 27th annual international symposium on Computer architecture, p.316-325, June 2000, Vancouver, British Columbia, Canada
• 8: Vinodh Cuppu , Bruce Jacob , Brian Davis , Trevor Mudge, A performance comparison of contemporary DRAM architectures, Proceedings of the 26th annual international symposium on Computer architecture, p.222-233, May 01-04, 1999, Atlanta, Georgia, United States
• 9: Jeff Gibson , Robert Kunz , David Ofelt , Mark Horowitz , John Hennessy , Mark Heinrich, FLASH vs. (Simulated) FLASH: closing the simulation loop, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.49-58, November 2000, Cambridge, Massachusetts, United States
• 10: N. P. Jouppi , S. J. E. Wilton, Tradeoffs in two-level on-chip caching, Proceedings of the 21st annual international symposium on Computer architecture, p.34-45, April 18-21, 1994, Chicago, Illinois, United States
• 11: R. E. Kessler, The Alpha 21264 Microprocessor, IEEE Micro, v.19 n.2, p.24-36, March 1999 [doi>10.1109/40.755465]
• 12: E. J. McLellan , D. A. Webb, The Alpha 21264 Microprocessor Architecture, Proceedings of the International Conference on Computer Design, p.90, October 05-05, 1998
• 13: David Kroft, Lockup-free instruction fetch/prefetch cache organization, Proceedings of the 8th annual symposium on Computer Architecture, p.81-87, May 12-14, 1981, Minneapolis, Minnesota, United States
• 14: Peter S. Magnusson , Fredrik Dahlgren , Håkan Grahn , Magnus Karlsson , Fredrik Larsson , Fredrik Lundholm , Andreas Moestedt , Jim Nilsson , Per Stenström , Bengt Werner, SimICS/sun4m: a virtual workstation, Proceedings of the Annual Technical Conference on USENIX Annual Technical Conference, 1998, p.10-10, June 15-19, 1998, New Orleans, Louisiana
• 15: [15] J. McCalpin. The stream benchmark site. http://www.cs.virginia.edu/stream/.
• 16: Larry McVoy , Carl Staelin, lmbench: portable tools for performance analysis, Proceedings of the Annual Technical Conference on USENIX 1996 Annual Technical Conference, p.23-23, January 22-26, 1996, San Diego, CA
• 17: Vijay Pai , Parthasarathy Ranganathan , Sarita Adve, RSIM: a simulator for shared-memory multiprocessor and uniprocessor systems that exploit ILP, Proceedings of the 1997 workshop on Computer architecture education, p.6-es, January 01, 1997 [doi>10.1145/1275165.1275171]
• 18: Matt Reilly , John Edmondson, Performance Simulation of an Alpha Microprocessor, Computer, v.31 n.5, p.50-58, May 1998 [doi>10.1109/2.675634]
• 19: Mendel Rosenblum , Stephen A. Herrod , Emmett Witchel , Anoop Gupta, Complete Computer System Simulation: The SimOS Approach, IEEE Parallel & Distributed Technology: Systems & Technology, v.3 n.4, p.34-43, December 1995 [doi>10.1109/88.473612]
• 20: Gurindar S. Sohi, Instruction Issue Logic for High-Performance, Interruptible, Multiple Functional Unit, Pipelined Computers, IEEE Transactions on Computers, v.39 n.3, p.349-359, March 1990 [doi>10.1109/12.48865]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.3 Microcomputers
          Subjects: Microprocessors

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Modeling techniques; Measurement techniques

General Terms:
Design, Measurement, Performance