FLASH vs. (simulated) FLASH: closing the simulation loop

ABSTRACT

Simulation is the primary method for evaluating computer systems during all phases of the design process. One significant problem with simulation is that it rarely models the system exactly, and quantifying the resulting simulator error can be difficult. More importantly, architects often assume without proof that although their simulator may make inaccurate absolute performance predictions, it will still accurately predict architectural trends.This paper studies the source and magnitude of error in a range of architectural simulators by comparing the simulated execution time of several applications and microbenchmarks to their execution time on the actual hardware being modeled. The existence of a hardware gold standard allows us to find, quantify, and fix simulator inaccuracies. We then use the simulators to predict architectural trends and analyze the sensitivity of the results to the simulator configuration. We find that most of our simulators predict trends accurately, as long as they model all of the important performance effects for the application in question. Unfortunately, it is difficult to know what these effects are without having a hardware reference, as they can be quite subtle. This calls into question the value, for architectural studies, of highly detailed simulators whose characteristics are not carefully validated against a real hardware design.

REFERENCES

• 1: Robert C. Bedichek, Talisman: fast and accurate multicomputer simulation, ACM SIGMETRICS Performance Evaluation Review, v.23 n.1, p.14-24, May 1995
• 2: Doug Burger , Todd M. Austin, The SimpleScalar tool set, version 2.0, ACM SIGARCH Computer Architecture News, v.25 n.3, p.13-25, June 1997 [doi>10.1145/268806.268810]
• 3: M. Durbhakula, V. Pai, and S. Adve. Improving the Speed vs. Accuracy Tradeoff for Simulating Shared-Memory Multiprocessors with ILP Processors. Rice University ECE Technical Report 9802, June 1998.
• 4: Stephen R. Goldschmidt, Simulation of multiprocessors: accuracy and performance, Stanford University, Stanford, CA, 1993
• 5: Mark Andrew Heinrich , John L. Hennessy, The performance and scalability of distributed shared-memory cache coherence protocols, 1999
• 6: Mark Heinrich , Jeffrey Kuskin , David Ofelt , John Heinlein , Joel Baxter , Jaswinder Pal Singh , Richard Simoni , Kourosh Gharachorloo , David Nakahira , Mark Horowitz , Anoop Gupta , Mendel Rosenblum , John Hennessy, The performance impact of flexibility in the Stanford FLASH multiprocessor, Proceedings of the sixth international conference on Architectural support for programming languages and operating systems, p.274-285, October 05-07, 1994, San Jose, California, United States
• 7: M. Heinrich, D. Ofelt, M. Horowitz, and J. Hennessy. Hardware/Software Codesign of the Stanford FLASH Multiprocessor. In Proceedings of the IEEE Special Issue on Hardware/Software Co-design, 85(3), March 1997.
• 8: Cristina Hristea , Daniel Lenoski , John Keen, Measuring memory hierarchy performance of cache-coherent multiprocessors using micro benchmarks, Proceedings of the 1997 ACM/IEEE conference on Supercomputing (CDROM), p.1-12, November 15-21, 1997, San Jose, CA [doi>10.1145/509593.509638]
• 9: J. Kuskin , D. Ofelt , M. Heinrich , J. Heinlein , R. Simoni , K. Gharachorloo , J. Chapin , D. Nakahira , J. Baxter , M. Horowitz , A. Gupta , M. Rosenblum , J. Hennessy, The Stanford FLASH multiprocessor, Proceedings of the 21ST annual international symposium on Computer architecture, p.302-313, April 18-21, 1994, Chicago, Illinois, United States
• 10: P.S. Magnusson, F. Dahlgren, H. Grahn, et al. SimICS/sun4m: A VirtualWorstation. In Proceedings of the Usenix Annual Technical Conference, June 1998.
• 11: David James Ofelt , John L. Hennessy, Efficient performance prediction for modern microprocessors, 1999
• 12: Margaret Martonosi , David Ofelt , Mark Heinrich, Integrating performance monitoring and communication in parallel computers, Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.138-147, May 23-26, 1996, Philadelphia, Pennsylvania, United States
• 13: L. McVoy and C. Staelin. lmbench: Portable tools for performance analysis. USENIX technical conference, pages 279-284, January 1996.
• 14: V. S. Pai, P. Ranganathan, and S. V. Adve. RSIM Reference Manual version 1.0. Technical Report #9705, Department of Electrical and Computer Engineering, Rice University, August 1997.
• 15: Vijay S. Pai , Parthasarathy Ranganathan , Sarita V. Adve, The Impact of Instruction-Level Parallelism on Multiprocessor Performance and Simulation Methodology, Proceedings of the 3rd IEEE Symposium on High-Performance Computer Architecture, p.72, February 01-05, 1997
• 16: U. Prestor. Snbench homepage, on-line at http://www.cs.utah.edu/~uros/snbench.
• 17: Steven K. Reinhardt , Mark D. Hill , James R. Larus , Alvin R. Lebeck , James C. Lewis , David A. Wood, The Wisconsin Wind Tunnel: virtual prototyping of parallel computers, Proceedings of the 1993 ACM SIGMETRICS conference on Measurement and modeling of computer systems, p.48-60, May 10-14, 1993, Santa Clara, California, United States
• 18: M. Rosenblum. Personal Communication.
• 19: Mendel Rosenblum , Stephen A. Herrod , Emmett Witchel , Anoop Gupta, Complete Computer System Simulation: The SimOS Approach, IEEE Parallel & Distributed Technology: Systems & Technology, v.3 n.4, p.34-43, December 1995 [doi>10.1109/88.473612]
• 20: Standard Performance Evaluation Corporation. The SPEC95 Benchmark Suite. Details on-line at http://www.specbench.org/.
• 21: Stanford Parallel Applications for Shared Memory. SPLASH-2 web page, on-line at http://www-flash.stanford.edu/apps/SPLASH.
• 22: Chris Stolte , Robert Bosch , Pat Hanrahan , Mendel Rosenblum, Visualizing Application Behavior on Superscalar Processors, Proceedings of the 1999 IEEE Symposium on Information Visualization, p.10, October 24-29, 1999
• 23: Madhusudhan Talluri , Mark D. Hill, Surpassing the TLB performance of superpages with less operating system support, Proceedings of the sixth international conference on Architectural support for programming languages and operating systems, p.171-182, October 05-07, 1994, San Jose, California, United States
• 24: Jack E. Veenstra , Robert J. Fowler, MINT: A Front End for Efficient Simulation of Shared-Memory Multiprocessors, Proceedings of the Second International Workshop on Modeling, Analysis, and Simulation On Computer and Telecommunication Systems, p.201-207, January 31-February 02, 1994
• 25: Steven Cameron Woo , Moriyoshi Ohara , Evan Torrie , Jaswinder Pal Singh , Anoop Gupta, The SPLASH-2 programs: characterization and methodological considerations, Proceedings of the 22nd annual international symposium on Computer architecture, p.24-36, June 22-24, 1995, S. Margherita Ligure, Italy
• 26: Kenneth Yeager. Personal Communication.
• 27: Kenneth C. Yeager, The MIPS R10000 Superscalar Microprocessor, IEEE Micro, v.16 n.2, p.28-40, April 1996 [doi>10.1109/40.491460]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies