Issues in computer performance evaluation: some consensus, some divergence

ABSTRACT

This paper summarizes the results of an ACM/NBS Workshop on Computer Performance Evaluation. Computer Performance Evaluation (CPE) was selected as the subject of an ACM/NBS Workshop because of the significant leverage CPE activities can have on computer usage. This paper describes a number of conclusions abstracted from the discussions as well as presenting recommendations formally adopted by the participants. While several of these conclusions indicate that improvements are needed in performance analysis tools, another suggests that improved application of CPE could be achieved by better documentation of analysis approaches. More integration of data collection and modeling are considered necessary for the performance analysis field to develop its full potential. Participants noted that the common emphasis on data collection or modeling, to the exclusion of considering objectives, often seriously degrades the value of performance analyses; the only savings that really count from a performance analysis are the ones that appear on the bottom line of the balance sheet.

REFERENCES

• 1: Boehm, B. W., T. E. Bell, S. Jeffery, "Computer Performance Evaluation: An Examination of the Critical Issues," National Bureau of Standards Report.
• 2: "Opportunity for Greater Efficiency and Savings Through the Use of Evaluation Techniques in the Federal Government's Computer Operations," U.S. General Accounting Office, Report B-115369, August 22, 1972.
• 3: Hall, G., and J. Wixson, ed., <u>Computer Measurement and Evaluation: Selected Papers from the SHARE Project</u>, SHARE: New York.
• 4: <u>Proceedings, ACM SIGME Symposium</u>, February 1973.
• 5: Warner, C. D., <u>Proceedings, 1972 Fall Joint Computer Conference</u>, pp. 959--964.
• 6: "Savings from Performance Monitoring," <u>EDP Analyzer</u>, September 1972.
• 7: Miller, R. B., "Response Time in Man-Computer Conversational Transactions," IBM Technical Report TR 00.1660-1, January 1968.
• 8: Doherty, W., "Scheduling TSS/360 for Responsiveness," <u>Proceedings, 1970 Fall Joint Computer Conference</u>, pp. 97--111.
• 9: James Martin, Design of Real-Time Computer Systems, Prentice-Hall, Inc., Upper Saddle River, NJ, 1967
• 10: Edward Yourdon, Design of on-Line Computer Systems, Prentice Hall PTR, Upper Saddle River, NJ, 1972
• 11: Willman, A. O., and C. O'Donnell, "Through the Control 'Multiprocessor' Avionics Enters the Computer Era," <u>Astronautics and Aeronautics</u>, July 1970.
• 12: Boehm, B. W., "Software and Its Impact: A Quantitative Assessment," <u>Datamation</u>, May 1973, pp. 48--59.
• 13: Kenneth W. Kolence, The software empiricist, ACM SIGMETRICS Performance Evaluation Review, v.2 n.2, p.31-36, June 1973 [doi>10.1145/1113644.1113647]
• 14: Burns, J., "Ford CUE Study: Six Stages of Measurement," Computer Analysts and Programmers, Ltd., London, Nov. 1973.
• 15: Bell, T. E., B. W. Boehm, and R. A. Watson, "Computer Systems Performance Improvement: Framework and Initial Phases," The Rand Corporation, R-549-1-PR November 1972, Pg. 5.
• 16: Quade, E. S., and W. Boucher, <u>Systems Analysis and Policy Planning: Applications in Defense</u>, American Elsevier, 1968.
• 17: Morrison, R. A., "An Approach to Performance Measurement," IBM TR 00.2272, February 1972.
• 18: Noe, J. D., and N. W. Runstein, "Continuous Computer Performance Monitoring," University of Washington, July 1973.