Evaluation of computer architectures via test programs

ABSTRACT

This article presents the evaluation of the Computer Family Architecture (CFA) candidate architectures via a set of test programs. The measures used to rank the computer architectures were S, the size of the test program, and M and R, two measures designed to estimate the principal components contributing to the nominal execution speed of the architecture. Descriptions of the twelve test programs and definitions of the S, M, and R measures are included here. The statistical design of the assignment of test programs to programmers is also discussed. Each program was coded from two to four times on each machine to minimize the uncertainty due to programmer variability. The final results show that for all three measures (S, M, and R) the Interdata 8/32 is the superior architecture, followed closely by the PDP-11, and the IBM S/370 trailed by a significant margin.

REFERENCES

• 1: Samuel H. Fuller , Harold S. Stone , William E. Burr, Initial selection and screening of the CFA candidate computer architectures, Proceedings of the June 13-16, 1977, national computer conference, June 13-16, 1977, Dallas, Texas [doi>10.1145/1499402.1499433]
• 2: Henry Lucas, Jr., Performance Evaluation and Monitoring, ACM Computing Surveys (CSUR), v.3 n.3, p.79-91, Sept. 1971 [doi>10.1145/356589.356590]
• 3: Bernwell, N. (editor), Benchmarking: Computer Evaluation and Measurement, John Wiley & Sons, New York, 1975.
• 4: Brian A. Wichmann, ALGOL Sixty Compilation and Assessment, Academic Press, Inc., Orlando, FL, 1974
• 5: Fuller, S. F., W. E. Burr, P. Shaman, and D. Lamb: Evaluation of Computer Architectures via Test Programs. Volume III of Computer Family Architecture Selection Committee Final Report, Naval Research Laboratory, Washington, D.C., 1 December 1976.
• 6: C. Gordon Bell , A. C. Newell, Computer structures: Readings and examples (McGraw-Hill computer science series), McGraw-Hill Pub. Co., 1971
• 7: Computer Review, GML Corporation, Lexington, Mass., 1976.
• 8: Stone, H. S. (editor), Introduction to Computer Architecture, Science Research Associates, Chicago, 1975.
• 9: Davies, O. L. (editor), Design and Analysis of Industrial Experiments, 2nd ed., Oliver and Boyd, Edinburgh, 1971.
• 10: Anderson, V. L. and R. A. McLean, Design of Experiments, a Realistic Approach, Marcel Dekker; Inc., New York, 1974.
• 11: Connor, W. S. and M. Zelen, Fractional Factorial Experiment Designs for Factors at Three Levels, National Bureau of Standards, Applied Mathematics Series 54, 1959.
• 12: Rao, C. R., Linear Statistical Inference and its Applications, 2nd ed., John Wiley & Sons, New York, 1973.
• 13: John J. Cornyn , William R. Smith , Aaron H. Coleman , William R. Svirsky, Life cycle cost models for comparing computer family architectures, Proceedings of the June 13-16, 1977, national computer conference, June 13-16, 1977, Dallas, Texas [doi>10.1145/1499402.1499437]
• 14: Box, G. E. P. and B. R. Cox, "An Analysis of Transformations", The Journal of the Royal Statistical Society, Series B, Vol. 26, 1964, pp. 211--252.