Statistical selection of the best system

ABSTRACT

This tutorial discusses some statistical procedures for selecting the best of a number of competing systems. The term "best" may refer to that simulated system having, say, the largest expected value or the greatest likelihood of yielding a large observation. We describe various procedures for finding the best, some of which assume that the underlying observations arise from competing normal distributions, and some of which are essentially nonparametric in nature. In each case, we comment on how to apply the above procedures for use in simulations.

REFERENCES

• 1: Bechhofer, R. E. 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics 25:16--39.
• 2: Bechhofer, R. E., S. Elmaghraby, and N. Morse. 1959. A single-sample multiple decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics 30:102--119.
• 3: Bechhofer, R. E., and D. M. Goldsman. 1986. Truncation of the Bechhofer-Kiefer-Sobel sequential procedure for selecting the multinomial event which has the largest probability (II): Extended tables and an improved procedure. Communications in Statistics---Simulation and Computation B15:829--851.
• 4: Bechhofer, R. E., T. J. Santner, and D. Goldsman. 1995. Design and Analysis of Experiments for Statistical Selection, Screening and Multiple Comparisons. New York: John Wiley and Sons.
• 5: Dunnett, C. W. 1989. Multivariate normal probability integrals with product correlation structure. Applied Statistics 38:564--579. Correction: 42:709.
• 6: Jean Dickinson Gibbons , Ingram Olkin , Milton Sobel, Selecting and ordering populations: a new statistical methodology, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1999
• 7: David Goldsman , Seong-Hee Kim , William S. Marshall , Barry L. Nelson, Ranking and Selection for Steady-State Simulation: Procedures and Perspectives, INFORMS Journal on Computing, v.14 n.1, p.2-19, January 2002 [doi>10.1287/ijoc.14.1.2.7710]
• 8: Goldsman, D., and B. L. Nelson. 1998. Comparing systems via simulation. Handbook of Simulation, ed. J. Banks, Chapter 8. New York: John Wiley and Sons.
• 9: David Goldsman , Barry L. Nelson, Statistical selection of the best system, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 10: Gupta, S. S. 1956. On a Decision Rule for a Problem in Ranking Means. Ph.D. Dissertation (Mimeo. Ser. No. 150). Institute of Statistics, University of North Carolina, Chapel Hill, North Carolina.
• 11: Gupta, S. S. 1965. On some multiple decision (selection and ranking) rules. Technometrics 7:225--245.
• 12: Y. Hochberg , A. C. Tamhane, Multiple comparison procedures, John Wiley & Sons, Inc., New York, NY, 1987
• 13: Hsu, J. C. 1984. Constrained simultaneous confidence intervals for multiple comparisons with the best. Annals of Statistics 12:1136--1144.
• 14: Hsu, J. C. 1996. Multiple Comparisons: Theory and Methods. New York: Chapman and Hall.
• 15: Seong-Hee Kim , Barry L. Nelson, A fully sequential procedure for indifference-zone selection in simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.11 n.3, p.251-273, July 2001 [doi>10.1145/502109.502111]
• 16: Seong-Hee Kim , Barry L. Nelson, Selecting the best system: selecting the best system: theory and methods, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
• 17: Kim, S.-H., and B. L. Nelson. 2005a. Selecting the best system. To appear in Handbooks in Operations Research and Management Science: Simulation, ed. S. G. Henderson and B. L. Nelson, Chapter 17. Oxford: Elsevier Science.
• 18: Kim, S.-H., and B. L. Nelson. 2005b. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. To appear in Operations Research.
• 19: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 20: Matejcik, F. J., and B. L. Nelson. 1995. Two-stage multiple comparisons with the best for computer simulation. Operations Research 43:633--640.
• 21: Miller, J. O., B. L. Nelson, and C. H. Reilly. Efficient multinomial selection in simulation. 1998. Naval Research Logistics 45:459--482.
• 22: Nakayama, M. K. 1997. Multiple-comparison procedures for steady-state simulations. Annals of Statistics 25:2433--2450.
• 23: Barry L. Nelson , David Goldsman, Comparisons with a Standard in Simulation Experiments, Management Science, v.47 n.3, p.449-463, March 2001 [doi>10.1287/mnsc.47.3.449.9778]
• 24: Barry L. Nelson , Frank J. Matejcik, Using common random numbers for indifference-zone selection and multiple comparisons in simulation, Management Science, v.41 n.12, p.1935-1945, Dec. 1995 [doi>10.1287/mnsc.41.12.1935]
• 25: Barry L. Nelson , Julie Swann , David Goldsman , Wheyming Song, Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large, Operations Research, v.49 n.6, p.950-963, November 2001 [doi>10.1287/opre.49.6.950.10019]
• 26: Paulson, E. 1964. A sequential procedure for selecting the population with the largest mean from k normal populations. Annals of Mathematical Statistics 35:174--180.
• 27: Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics---Theory and Methods A7:799--811.
• 28: Wald, A. 1947. Sequential Analysis. New York: John Wiley.
• 29: Wilcox, R. R. 1984. A table for Rinott's selection procedure. Journal of Quality Technology 16:97--100.