Performance Evaluation and Monitoring

ABSTRACT

Three major purposes for evaluating the hardware and software performance of computer systems--selection evaluation, performance projection, and performance monitoring--are described. Eight techniques that have been used or suggested for evaluating performance are discussed. Each of these techniques is rated on its suitability for the three purposes of evaluation. Recommendations are made on the most appropriate technique for each evaluation purpose. These suggestions include the development of a comprehensive set of synthetic programs on an industry-wide basis for selection evaluation purposes. Simulation is recommended as the most suitable technique for performance projection. Finally, a number of hardware and software monitors are available for performance monitoring.

REFERENCES

• 1: ANACKER, W.; AND C. P. WANG. "Performance evaluation of computing systems with memory hierarchies." IEEE Trans. Computers EC-16, 6 (Dec. 1967), 764-773.
• 2: C. T. Apple, Evaluation and performance of computers: the program monitor—a device for program performance measurement, Proceedings of the 1965 20th national conference, p.66-75, August 24-26, 1965, Cleveland, Ohio, United States [doi>10.1145/800197.806034]
• 3: ARBUCKLE, R. A. "Computer analysis and thruput evaluation." Computers and Automation 15, 1 (Jan. 1966), 12-15, 19.
• 4: Philip R. Bagley, Two think pieces, Communications of the ACM, v.3 n.1, p.1, Jan. 1960 [doi>10.1145/366947.366953]
• 5: BALDWIN, F. R.; W. B. GIBSON; AND C. B. PO- LAND. "A multiprocessing approach to a large computer system." IBM Systems J. 1, 1 (Sept. 1962), 64-76.
• 6: BEMER, R. W.; AND A. L. ELLISON. "Software instrumentation systems for optimum performance." Proc. IFIP Congress 1968, Vol. 1, North- Holland Publ. Co., Amsterdam, pp. 520-524.
• 7: BONNER, A .J . "Using system monitor output to improve performance." IBM Systems J. 8, 4 (1969), 290-298.
• 8: BUCHOLZ, W. "A synthetic job for measuring system performance." IBM Systems J. 8, 4 (1969), 309-318.
• 9: ___, "A selected bibliography on computer system performance evaluation." Computer Group News (March 1969), 21-22.
• 10: BUDD, A.E. "A method for the evaluation of software: general description including benchmark considerations." Mitre Corp. Tech. Rep. 197, vol. 1, 1966.
• 11: ___, "A method for the evaluation of softwure: procedural language compilers--particularly COBOL and FORTRAN." Mitre Corp. Tech. Rep. 197, vol. 2, 1966.
• 12: ___, "A method for the evaluation of software: executive, operating and monitor systems." Mitre Corp. Tech. Rep. 197, vol. 3, 1966.
• 13: Peter Calingaert, System performance evaluation: survey and appraisal, Communications of the ACM, v.10 n.1, p.12-18, Jan. 1967 [doi>10.1145/363018.363040]
• 14: CAMPBELL, D. J.; AND W. J. HEFFNER. "Measurement and analysis of large operating systems during system development." Proc. AFIPS 1968 FJCC, Vol. 33, Pt. 1, AFIPS Press, Montvale, N.J., pp. 903-914.
• 15: CANTRELL, H. N. ; AND A. C. ELLISON. "Multiprogramming system performance and analysis." Proc. AF1PS 195'8 SJCC, Vol. 32, AFIPS Press, Montvale, N.J., pp. 213-221.
• 16: CHENG, P.S. "Trace-driven system modeling." IBM Systems J. 8, 4 (1959), 250-289.
• 17: DOPPING, O. "Test problems used for evaluation of computers." BIT 2, 4 (1962). 197-202.
• 18: DRUMMOND, M.E. "A perspective on systems performance evaluation." IBM Systems J. 8, 4 (1969), 252-263.
• 19: GLINKA, L. R.; R. M. BRUSH; AND A. J. UNGAa. "Design, thru simulation, of a multiple-access information system." Proc. AFIPS 1967 FJCC, Vol. 31, AFIPS Press, Montvale, N.J., pp. 437- 447.
• 20: HERMAN, DONALD J. "ScERT: a computer evaluation tool." Datamation 13, 2 (Feb. 1967), 26-28.
• 21: HERMAN, D. J.; AND F. C. IHREa. "The use of a computer to evaluate computers." Proc. AFIPS 1964 SJCC, Vol. 25, Spartan Books, New York, pp. 383-395.
• 22: HUESMANN, L. R.; AND ROBERT P. GOLDBERG. "Evaluating computer systems through simulation." Computer J. 10, 2 (Aug. 1967), 150-155.
• 23: George K. Hutchinson, A computer center simulation project, Communications of the ACM, v.8 n.9, p.559-568, Sept. 1965 [doi>10.1145/365559.365615]
• 24: ___, AND J. N. MAGUIRE. "Computer systems design and analysis through simulation." Proc. AF1PS 1965 FJCC, Vol. 27, Pt. 1, Spartan Books, New York, pp. 161-167.
• 25: IHRER, F. C. "Computer performance projected through simulation." Computers and Automation 16, 4 (April 1967), 22-27.
• 26: R. F. Hitti , E. O. Joslin, Evaluation and performance of computers: application benchmarks: the key to meaningful computer evaluations, Proceedings of the 1965 20th national conference, p.27-37, August 24-26, 1965, Cleveland, Ohio, United States [doi>10.1145/800197.806031]
• 27: ___, "Cost-value technique for evaluation of computer system proposals." Proc. AFIPS 1964 SJCC, Vol. 25, Spartan Books, New York, pp. 367-381.
• 28: ___, AND JOHN J. AIKEN. "The validity of basing computer selection on benchmark results." Computers and Automation 1S, 1 (Jan. 1966), 22-23.
• 29: KATZ, J. H. "Simulation of a multiprocessor computer system." Proc. AFIPS 1966 SJCC, Vol. 28, Spartan Books, New York, pp. 127-139.
• 30: Jesse H. Katz, An experimental model of system/360, Communications of the ACM, v.10 n.11, p.694-702, Nov. 1967 [doi>10.1145/363790.363808]
• 31: KEEFE, D. D. "Hierarchical control programs for systems evaluation." IBM Systems J. 7, 2 (1968), 123-133.
• 32: KNIGHT, K. E. "Changes in computer performance." Datamatiou 12, 9 (Sept. 1966), 40-- 54.
• 33: ___, "Evolving computer performance 1963- 1967." Datamation 14, 6 (Jan. 1968), 31-35.
• 34: LEHMAN, M. M.; AND J. L. ROSENFELD. "Performance of a simulated multi-programming system." Proc. AFIPS 1968 FJCC, Vol. 33, Pt. 2, AFIPS Press, Montvale, N.J., pp. 1431- 1442.
• 35: Lucas, HENRY C., JR. "Computer evaluation through benchmark testing." Unpublished paper, MIT, Cambridge, Mass., May, 1967.
• 36: MERIKALLIO, R. A.; AND F. C. HOLLAND. "Simulation design of a multiprocessing system." Proc. AFIPS 1968 FJCC, Vol. 33, Pt. 2, AFIPS Press, Montvale, N.J., pp. 1399-1410.
• 37: MURPHY, JESSE O.; AND ROBERT M. WADE. "The IBM 360/195 in a world of mixed job streams." Datamation 16, 4 (April 1970), 72- 79.
• 38: Norman R. Nielsen, The simulation of time sharing systems, Communications of the ACM, v.10 n.7, p.397-412, July 1967 [doi>10.1145/363427.363436]
• 39: Norman R. Nielsen, Computer simulation of computer system performance, Proceedings of the 1967 22nd national conference, p.581-590, January 1967, Washington, D.C., United States [doi>10.1145/800196.806027]
• 40: OPLER, ASCHER. "Measurement of software characteristics." Datamation 10, 7 (July 1964), 27-30.
• 41: PATRICK, ROBERT L. "Measuring performance." Datamation 10, 7 (July 1964), 24-27.
• 42: Eugene Raichelson , George Collins, A method for comparing the internal operating speeds of computers, Communications of the ACM, v.7 n.5, p.309-310, May 1964 [doi>10.1145/364099.364334]
• 43: Robert F. Rosin, Determining a computing center environment, Communications of the ACM, v.8 n.7, p.463-468, July 1965 [doi>10.1145/364995.365690]
• 44: SCHERR, ALLAN L, "Time-sharing measurement." Datamation 12, 4 (April 1956), 22-26.
• 45: Franklin D. Schulman, Hardware measurement device for IBM system/360 time sharing evaluation, Proceedings of the 1967 22nd national conference, p.103-109, January 1967, Washington, D.C., United States [doi>10.1145/800196.805980]
• 46: SCHWARTZ, EUGENE S. "Computer evaluation and selection." J. Data Management 6, 6 (June 1968), 58--62.
• 47: SEAMAN, P. H.; AND R. C. SoucY. "Simulating operating systems." IBM Systems J. 8, 4 (1969), 264-279.
• 48: William F. Sharpe, The Economics of Computers, Columbia University Press, New York, NY, 1972
• 49: SMITH, J. L. "An analysis of time-sharing computer systems using Markov models." Proc. AFIPS 1966 SJCC, Vol. 28, Spartan Books, New York, pp. 87-95.
• 50: STANG, HAROLD; AND PETER SOUTHGATE. "Performance evaluation of third-generation computing systems." Datamation 15, 11 (Nov. 1969), 181-190.
• 51: STANLEY, W. I. "Measurement of system operational statistics." IBM Systems J. 8, 4 (1969), 299-3O8.
• 52: ___, AND H. F. HERTEL. "Statistics gathering and simulation for the Apollo real-time operating system." IBM Systems J. 7, 2 (1968), 85-102.
• 53: STATLAND, NOaMAN. "Methods of evaluating computer systems performance." Computers and Automation 13, 2 (Feb. 1964), 18--23.
• 54: S. Stimler, A methodology for calculating and optimizing real-time system performance, Communications of the ACM, v.11 n.7, p.509-516, July 1968 [doi>10.1145/363397.363560]
• 55: TOTARO, J. B. "Real-time processing power: a standardized evaluation." Computers and Automation 17, 4 (April 1967), 16-19.
• 56: Edward S. Walter , Victor L. Wallace, Further analysis of a computing center environment, Communications of the ACM, v.10 n.5, p.266-272, May 1967 [doi>10.1145/363282.363294]
• 57: Peter White, Relative effects of central processor and input-output speeds upon throughput on the large computer, Communications of the ACM, v.7 n.12, p.711-714, Dec. 1964 [doi>10.1145/355588.365120]
• 58: WILLIAMS, 0.; ET AL. "A methodology for computer selection studies." Computers and Automation 12, 5 (May 1963), 18--23.
• 59: YOUCHAH, M. I.; ET AL. "The data processing system simulator (DPSS)." Proc. AFIPS 1964 FJCC, Vol. 26, Pt. 1, Spartan Books, New York, pp. 251-276.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)