An empirical evaluation of several methods to select the best system

ABSTRACT

Simulation is an important tool for comparing the performance of several alternative systems. There is therefore significant interest in procedures that efficiently select the best system, where best is defined by the maximum or minimum expected simulation output. In this paper, we examine both two-stage and sequential procedures that represent three structurally different modeling methodologies for allocating simulation replications to identify the best system, and we evaluate them empirically with respect to several measures of effectiveness. Empirical evidence suggests that sequential procedures perform better than their two-stage counterparts, including a heuristic sequential variation on Rinott's procedure. Further, there appears to be significant benefit to using procedures based on a Bayesian, average-case analysis as opposed to the statistically-conservative indifference-zone formulation.

REFERENCES

• 1: BANKS, J., CARSON, J. S., AND NELSON, B. L. 1996. Discrete-Event System Simulation. 2nd. ed. Prentice-Hall, Inc., Upper Saddle River, NJ.
• 2: BECHHOFER, R., SANTNER, T., AND GOLDSMAN, D. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. John Wiley and Sons, Inc., New York, NY.
• 3: BERGER, J. O. 1988. A Bayesian approach to ranking and selection of related means with alternatives to analysis-of-variance methodology. J. Am. Stat. Assoc. 83, 402, 364-373.
• 4: BERNARDO, J. M. AND SMITH, A. F. M. 1994. Bayesian Theory. John Wiley and Sons Ltd., Chichester, UK.
• 5: BRACKEN, J. AND SCHLEIFER, A. 1964. Tables for normal sampling with unknown variance. Tech. Rep., Graduate School of Business Admin.. Harvard Univ., Cambridge, MA.
• 6: CHEN, C. -H. 1996. A lower bound for the correct subset-selection probability and its application to discrete event simulations. IEEE Trans. Automat. Contr. 41, 8, 1227-1231.
• 7: Chun-Hung Chen , Hsiao-Chang Chen , Liyi Dai, A gradient approach for smartly allocating computing budget for discrete event simulation, Proceedings of the 28th conference on Winter simulation, p.398-405, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256696]
• 8: Hsiao-Chang Chen , Liyi Dai , Chun-Hung Chen , Enver Yücesan, New development of optimal computing budget allocation for discrete event simulation, Proceedings of the 29th conference on Winter simulation, p.334-341, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268501]
• 9: Stephen E. Chick , Koichiro Inoue, Sequential allocations that reduce risk for multiple comparisons, Proceedings of the 30th conference on Winter simulation, p.669-676, December 13-16, 1998, Washington, D.C., United States
• 10: CHICK, S. E. AND INOUE, K. 1999. New two-stage and sequential procedures for selecting the best simulated system. Tech. Rep., Dept. of Industrial Operations Engineering. University of Michigan, Ann Arbor, MI.
• 11: DE GROOT, M. H. 1970. Optimal Statistical Decisions. McGraw-Hill, Inc., New York, NY.
• 12: DUDEWICZ, E. J. AND DALAL, S. R. 1975. Allocation of observations in ranking and selection with unequal variances. Sanhkya B37, 28-78.
• 13: David Goldsman , Barry L. Nelson, Statistical screening, selection, and multiple comparison procedures in computer simulation, Proceedings of the 30th conference on Winter simulation, p.159-166, December 13-16, 1998, Washington, D.C., United States
• 14: INOUE, K. 1999. Bayesian decision-theoretic selection procedures to identify the best simulated system. Ph.D. Dissertation. University of Michigan, Ann Arbor, MI.
• 15: Koichiro Inoue , Stephen E. Chick, Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system, Proceedings of the 30th conference on Winter simulation, p.727-734, December 13-16, 1998, Washington, D.C., United States
• 16: KOENIG, L. W. AND LAW, A. M. 1985. A procedure for selecting a subset of size m containing the g best of k independent normal populations, with applications to simulation. Commun. Stat.-Simul. Comput. 14, 3, 719-734.
• 17: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 18: MIESCKE, K. J. AND GUPTA, S 1996. Bayesian look ahead one-stage sampling allocations for selecting the best population. J. Stat. Plan. Inference 54, 229-244.
• 19: Barry L. Nelson , Frank J. Matejcik, Using common random numbers for indifference-zone selection and multiple comparisons in simulation, Management Science, v.41 n.12, p.1935-1945, Dec. 1995 [doi>10.1287/mnsc.41.12.1935]
• 20: NELSON, B. L., SWANN, J., GOLDSMAN, D., AND SONG, W. 1999. Simple procedures for selecting the best simulated system when the number of alternatives is large. Tech. Rep., Dept. of Industrial Engineering and Management Science. Northwestern University, Evanston, IL.
• 21: RINOTT, Y. 1978. On two-stage selection procedures and related probability-inequalities. Commun. Stat. A7, 799-811.
• 22: WELCH, B. L. 1938. The significance of the difference between two means when the population variances are unequal. Biometrika 25, 350-362.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Experimental design

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

General Terms:
Experimentation, Performance

Keywords:
discrete-event simulation, multiple selection procedures, ranking and selection