A statistical study of the accuracy of floating point number systems

ABSTRACT

This paper presents the statistical results of tests of the accuracy of certain arithmetic systems in evaluating sums, products and inner products, and analytic error estimates for some of the computations. The arithmetic systems studied are 6-digit hexadecimal and 22-digit binary floating point number representations combined with the usual chop and round modes of arithmetic with various numbers of guard digits, and with a modified round mode with guard digits. In a certain sense, arithmetic systems differing only in their use of binary or hexadecimal number representations are shown to be approximately statistically equivalent in accuracy. Further, the usual round mode with guard digits is shown to be statistically superior in accuracy to the usual chop mode in all cases save one. The modified round mode is found to be superior to the chop mode in all cases.

REFERENCES

• 1: Richard W Hamming, Numerical methods for scientists and engineers (2nd ed.), Dover Publications, Inc., New York, NY, 1986
• 2: Donald E. Knuth, The art of computer programming, volume 2 (3rd ed.): seminumerical algorithms, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997
• 3: Urabe, M. Roundoff error distribution in fixed-point multiplication and a remark about the rounding rule. SlAM J. Num. Anal. 5, 2 (June 1968), 202-210.
• 4: James H. Wilkinson, Rounding Errors in Algebraic Processes, Dover Publications, Incorporated, 1994
• 5: T. E. Hull , J. R. Swenson, Tests of probabilistic models for propagation of roundoff errors, Communications of the ACM, v.9 n.2, p.108-113, Feb. 1966 [doi>10.1145/365170.365212]
• 6: Peter Henrici, Test of probabilistic models for the propagation of roundoff errors, Communications of the ACM, v.9 n.6, p.409-410, June 1966 [doi>10.1145/365696.365698]
• 7: Brent, R.P. On the precision attainable with various floating-point number systems (to be published).
• 8: Toyohisa Kaneko , Bede Liu, On Local Roundoff Errors in Floating-Point Arithmetic, Journal of the ACM (JACM), v.20 n.3, p.391-398, July 1973 [doi>10.1145/321765.321771]

INDEX TERMS

Keywords:
error analysis, floating point arithmetic, guard digits, number representation, rounding