Selecting the best system: selecting the best system: theory and methods

ABSTRACT

This paper provides an advanced tutorial on the construction of ranking-and-selection procedures for selecting the best simulated system. We emphasize procedures that provide a guaranteed probability of correct selection, and the key theoretical results that are used to derive them.

REFERENCES

• 1: Banerjee, S. 1961. On confidence interval for two-means problem based on separate estimates of variances and tabulated values of t-table. Sankhyá A23:359--378.
• 2: Banks, J., J. S. Carson, B. L. Nelson, and D. Nicol. 2001. Discrete-Event System Simulation. Upper Saddle River, NJ: Prentice Hall.
• 3: Bechhofer, R. E. 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics 25:16--39.
• 4: Bechhofer, R. E., S. Elmaghraby, and N. Morse. 1959. A single-sample multiple-decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics 30:102--119.
• 5: Bechhofer, R. E., T. J. Santner, and D. Goldsman. 1995. Design and Analysis of Experiments for Statistical Selection, Screening and Multiple Comparisons. New York: John Wiley & Sons.
• 6: Justin Boesel , Barry L. Nelson , Seong-Hee Kim, Using Ranking and Selection to "Clean Up" after Simulation Optimization, Operations Research, v.51 n.5, p.814-825, September 2003 [doi>10.1287/opre.51.5.814.16751]
• 7: Chen, H. C., C. H. Chen, and E. Yücesan. 2000. Computing efforts allocation for ordinal optimization and discrete event simulation. IEEE Transactions on Automatic Control 45:960--964
• 8: Stephen E. Chick , Koichiro Inoue, New Two-Stage and Sequential Procedures for Selecting the Best Simulated System, Operations Research, v.49 n.5, p.732-743, September 2001 [doi>10.1287/opre.49.5.732.10615]
• 9: Edward J. Dudewicz, The heteroscedastic method: fifty+ years of progress 1945–2000, and Professor Minoru Siotani's award-winning contributions, American Journal of Mathematical and Management Sciences, v.15 n.3-4, p.179-197, 1995
• 10: Fabian, V. 1974. Note on Anderson's sequential procedures with triangular boundary. Annals of Statistics 2:170--176.
• 11: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 12: Goldsman, D., and B. L. Nelson. 1998. Comparing systems via simulation. In Handbook of Simulation, ed. J. Banks, 273--306. New York: John Wiley.
• 13: Gupta, S. S. 1956. On a decision rule for a problem in ranking means. Doctoral dissertation, Institute of Statistics, Univ. of North Carolina, Chapel Hill, NC.
• 14: Gupta, S. S. 1965. On some multiple decision (ranking and selection) rules. Technometrics 7:225--245.
• 15: Hartmann, M. 1988. An improvement on Paulson's sequential ranking procedure. Sequential Analysis 7:363--372.
• 16: Hartmann, M. 1991. An improvement on Paulson's procedure for selecting the population with the largest mean from k normal populations with a common unknown variance. Sequential Analysis 10:1--16.
• 17: Y. Hochberg , A. C. Tamhane, Multiple comparison procedures, John Wiley & Sons, Inc., New York, NY, 1987
• 18: Hong, L. J., and B. L. Nelson. 2003. The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. Technical Report, Dept. of IEMS, Northwestern Univ., Evanston, IL.
• 19: Hsu, J. C. 1996. Multiple Comparisons: Theory and Methods. New York: Chapman & Hall.
• 20: Jennison, C., I. M. Johnstone, and B. W. Turnbull. 1980. Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means. Technical Report, Dept. of ORIE, Cornell Univ., Ithaca, NY.
• 21: Jennison, C., I. M. Johnstone, and B. W. Turnbull. 1982. Asymptotically optimal procedures for sequential adaptive selection of the best of several normal means. In Statistical Decision Theory and Related Topics III, Vol. 2, ed. S. S. Gupta and J. O. Berger, 55--86. New York: Academic Press.
• 22: Jennison, C., and B. W. Turnbull. 2000. Group Sequential Methods with Applications to Clinical Trials. New York: Chapman & Hall.
• 23: Koenig, L. W., and A. M. Law. 1985. A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation. Communications in Statistics---Simulation and Computation B14:719--734.
• 24: Kim, S. -H. 2002. Comparison with a standard via fully sequential procedures. Technical Report, School of ISyE, Georgia Tech, Atlanta, GA.
• 25: Seong-Hee Kim , Barry L. Nelson, A fully sequential procedure for indifference-zone selection in simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.11 n.3, p.251-273, July 2001 [doi>10.1145/502109.502111]
• 26: Kim, S. -H., and B. L. Nelson. 2003. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Technical Report, Dept. of IEMS, Northwestern Univ., Evanston, IL.
• 27: Kimball, A. W. 1951. On dependent tests of significance in the analysis of variance. Annals of Mathematical Statistics 22:600--602.
• 28: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 29: Miller, J. O., B. L. Nelson, and C. H. Reilly. 1998. Efficient multinomial selection in simulation. Naval Research Logistics 45:459--482.
• 30: Barry L. Nelson , David Goldsman, Comparisons with a Standard in Simulation Experiments, Management Science, v.47 n.3, p.449-463, March 2001 [doi>10.1287/mnsc.47.3.449.9778]
• 31: Barry L. Nelson , Frank J. Matejcik, Using common random numbers for indifference-zone selection and multiple comparisons in simulation, Management Science, v.41 n.12, p.1935-1945, Dec. 1995 [doi>10.1287/mnsc.41.12.1935]
• 32: Barry L. Nelson , Julie Swann , David Goldsman , Wheyming Song, Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large, Operations Research, v.49 n.6, p.950-963, November 2001 [doi>10.1287/opre.49.6.950.10019]
• 33: Paulson, E. 1964. A sequential procedure for selecting the population with the largest mean from k normal populations. Annals of Mathematical Statistics 35:174--180.
• 34: Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics---Theory and Methods A7:799--811.
• 35: Siegmund, D. 1985. Sequential Analysis: Tests and Confidence Intervals. New York: Springer-Verlag.
• 36: Slepian, D. 1962. The one-sided barrier problem for Gaussian noise. Bell Systems Technical Journal 41:463--501.
• 37: Stein, C. 1945. A two-sample test for a linear hypothesis whose power is independent of the variance. Annals of Mathematical Statistics 16:243--258.
• 38: D. W. Sullivan , J. R. Wilson, Restricted subset selection procedures for simulation, Operations Research, v.37 n.1, p.52-71, Jan.-Feb. 1989 [doi>10.1287/opre.37.1.52]
• 39: Wald, A. 1947. Sequential Analysis. New York: John Wiley.
• 40: Welch, B. L. 1938. The significance of the difference between two means when the population variances are unequal. Biometrika 25:350--362.
• 41: Wilcox, R. R. 1984. A table for Rinott's selection procedure. Journal of Quality Technology 16:97--100.
• 42: James R. Wilson, A Multiplicative Decomposition Property of the Screening-and-Selection Procedures of Nelson et al., Operations Research, v.49 n.6, p.964-966, November 2001 [doi>10.1287/opre.49.6.964.10013]