How not to lie with statistics: the correct way to summarize benchmark results

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

How not to lie with statistics: the correct way to summarize benchmark results

Full text

Pdf (358 KB)

Source

Communications of the ACM archive
Volume 29 , Issue 3 (March 1986) table of contents
The MIT Press scientific computation series

Pages: 218 - 221

Year of Publication: 1986

ISSN:0001-0782

Authors

Philip J. Fleming	AT&T Information Systems, Naperville, IL
John J. Wallace	The Foxboro Company, Foxboro, MA

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 45, Downloads (12 Months): 196, Citation Count: 22

Additional Information:

abstract references cited by index terms review collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/5666.5673 What is a DOI?

ABSTRACT

Using the arithmetic mean to summarize normalized benchmark results leads to mistaken conclusions that can be avoided by using the preferred method: the geometric mean.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Aczel. J, Funcfional Equations. Academic Press. New York. 1966. p. 239. A comprehensive textbook on functional equations.
	2	J. L. Heath, Re-evaluation of the RISC I, ACM SIGARCH Computer Architecture News, v.12 n.1, p.3-10, March 1984 [doi>10.1145/641602.641603]
	3	Patterson, D.A., and Sequin, C.H. A VLSI RISC. Computer 15. 9 (Sept. 1982). 6-21. The landmark paper formally introducing the RISC approach to computer architecture.

CITED BY 22

	David F. Hinnant, Accurate Unix Benchmarking: Art, Science, or Black Magic?, IEEE Micro, v.8 n.5, p.64-75, September 1988
	J. Helin , K. Kaski, Performance analysis of high-speed computers, Proceedings of the 1989 ACM/IEEE conference on Supercomputing, p.797-808, November 12-17, 1989, Reno, Nevada, United States
	Hal Berghel , Gregg Stubbendieck , Eric Traudt, Performance characteristics of microcomputer Prolog implementations, Proceedings of the 1986 ACM SIGSMALL/PC symposium on Small systems, p.64-71, December 1986, San Francisco, California, United States
	Vaughn Betz , Jonathan Rose, Using architectural “families” to increase FPGA speed and density, Proceedings of the 1995 ACM third international symposium on Field-programmable gate arrays, p.10-16, February 12-14, 1995, Monterey, California, United States
	Mark S. Johnstone , Paul R. Wilson, The memory fragmentation problem: solved?, ACM SIGPLAN Notices, v.34 n.3, p.26-36, March 1999
	R. J. Eickenmeyer , J. H. Patel, Performance evaluation of on-chip register and cache organizations, ACM SIGARCH Computer Architecture News, v.16 n.2, p.64-72, May 1988
	Gideon Intrater , Ilan Spillinger, Performance evaluation of a decoded instruction cache for variable instruction-length computers, ACM SIGARCH Computer Architecture News, v.20 n.2, p.106-113, May 1992
	Corinna G. Lee , Mark G. Stoodley, Simple vector microprocessors for multimedia applications, Proceedings of the 31st annual ACM/IEEE international symposium on Microarchitecture, p.25-36, November 1998, Dallas, Texas, United States
	G. D. Intrater , I. Y. Spillinger, Performance Evaluation of a Decoded Instruction Cache for Variable Instruction Length Computers, IEEE Transactions on Computers, v.43 n.10, p.1140-1150, October 1994
	Ran Giladi , Niv Ahituv, SPEC as a Performance Evaluation Measure, Computer, v.28 n.8, p.33-42, August 1995
	J. E. Smith, Characterizing computer performance with a single number, Communications of the ACM, v.31 n.10, p.1202-1206, Oct. 1988
	Carl Ponder, Performance variation across benchmark suites, ACM SIGMETRICS Performance Evaluation Review, v.18 n.3, p.42-48, Nov. 1990
	Jia-Lang Seng , S. Bing Yao , Alan R. Hevner, Requirements-driven database systems benchmark method, Decision Support Systems, v.38 n.4, p.629-648, January 2005
	John R. Mashey, War of the benchmark means: time for a truce, ACM SIGARCH Computer Architecture News, v.32 n.4, p.1-14, September 2004
	Claudia Fiorini , Enrico Martinelli , Fabio Massacci, How to fake an RSA signature by encoding modular root finding as a SAT problem, Discrete Applied Mathematics, v.130 n.2, p.101-127, 15 August 2003
	N. Fenton, Software Measurement: A Necessary Scientific Basis, IEEE Transactions on Software Engineering, v.20 n.3, p.199-206, March 1994
	Daniel Citron , Adham Hurani , Alaa Gnadrey, The harmonic or geometric mean: does it really matter?, ACM SIGARCH Computer Architecture News, v.34 n.4, p.18-25, September 2006
	Carl Ponder, Performance variation across benchmark suites, ACM SIGARCH Computer Architecture News, v.19 n.4, p.30-36, June 1991
	Dionisios N. Pnevmatikatos , Mark D. Hill, Cache performance of the integer SPEC benchmarks on a RISC, ACM SIGARCH Computer Architecture News, v.18 n.2, p.53-68, Jun. 1990
	Simon Peyton Jones , Simon Marlow, Secrets of the Glasgow Haskell Compiler inliner, Journal of Functional Programming, v.12 n.5, p.393-434, July 2002
	N. J. Gunther , M. T. Noga, ParcBench: a benchmark for shared-memory architectures, ACM SIGARCH Computer Architecture News, v.17 n.4, p.54-61, June 1989
	Shouxi Yang , Alberto Maria Segre , Bruno Codenotti, An optimal multiprocessor combinatorial auction solver, Computers and Operations Research, v.36 n.1, p.149-166, January, 2009

INDEX TERMS

Classification:
C. Computer Systems Organization
C.4 PERFORMANCE OF SYSTEMS
Subjects: Performance attributes; Measurement techniques

G. Mathematics of Computing

General Terms:
Measurement, Performance

REVIEW

"Giuseppe Antonio Caruso : Reviewer"

In this paper, the authors demonstrate why the geometric mean is appropriate for summarizing normalized benchmark results, and why the arithmetic mean, when used in this context, leads to grossly incorrect conclusions. In case one ass more...

Collaborative Colleagues:

Philip J. Fleming: colleagues

John J. Wallace: colleagues

Adobe Acrobat

QuickTime

Windows Media Player

Real Player