New results on procedures that select the best system using CRN

ABSTRACT

One use of simulation is to inform decision makers that seek to select the best of several alternative systems. The system with the highest (or lowest) mean value for simulation output is often selected as best, and simulation output is used to infer the value of the unknown mean of each system. Statistical procedures that help to identify the best system by suggesting an appropriate number of replications for each system are therefore useful tools in simulation. This article explores the performance of representative procedures from two approaches to develop statistical procedures, with the goal of understanding tradeoffs involving the ease of use, computational requirements, and the range of applicability. The focus is primarily on procedures that use common random numbers to sharpen comparisons between systems.

REFERENCES

• 1: Anderson, T. W. 1957. Maximum likelihood estimates for a multivariate normal distribution when some observations are missing. Journal of the American Statistical Association 52: 200-203.
• 2: Banks, J., J. S. Carson, and B. L. Nelson. 1996. Discrete-event system simulation (2nd ed.). Upper Saddle River, NJ, USA: Prentice-Hall, Inc.
• 3: Bechhofer, R. E., T. J. Santner, and D. M. Goldsman. 1995. Design and analysis for statistical selection, screening, and multiple comparisons. New York: John Wiley & Sons, Inc.
• 4: Chen, C.-H. 1996. A lower bound for the correct subset-selection probability and its application to discrete event simulations. IEEE Transactions on Automatic Control 41(8): 1227-1231.
• 5: Hsiao-Chang Chen , Chun-Hung Chen , Jianwu Lin , Enver Yücesan, An asymptotic allocation for simultaneous simulation experiments, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.359-366, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324242]
• 6: Stephen E. Chick , Koichiro Inoue, A decision-theoretic approach to screening and selection with common random numbers, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.603-610, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324444]
• 7: Chick, S. E., and K. Inoue. 2000a. New procedures for identifying the best simulated system using common random numbers. in resubmission.
• 8: Stephen E. Chick , Koichiro Inoue, New Two-Stage and Sequential Procedures for Selecting the Best Simulated System, Operations Research, v.49 n.5, p.732-743, September 2001 [doi>10.1287/opre.49.5.732.10615]
• 9: Gordon M. Clark , Wei-ning Yang, A Bonferroni selection procedure when using commom random numbers with unknown variances, Proceedings of the 18th conference on Winter simulation, p.313-315, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318452]
• 10: David Goldsman , Barry L. Nelson, Statistical screening, selection, and multiple comparison procedures in computer simulation, Proceedings of the 30th conference on Winter simulation, p.159-166, December 13-16, 1998, Washington, D.C., United States
• 11: Inoue, K. 2000. Decision-theoretic comparison of alternative system configurations using stochastic simulation. Ph. D. thesis, The University of Michigan, Ann Arbor, MI. Dept. of Industrial and Operations Engineering.
• 12: Koichiro Inoue , Stephen E. Chick, Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system, Proceedings of the 30th conference on Winter simulation, p.727-734, December 13-16, 1998, Washington, D.C., United States
• 13: Koichiro Inoue , Stephen E. Chick , Chun-Hung Chen, An empirical evaluation of several methods to select the best system, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.9 n.4, p.381-407, Oct. 1999 [doi>10.1145/352222.352226]
• 14: Koenig, L. W., and A. M. Law. 1985. A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation. Commun. Statist.-Simulation and Computation 14(3): 719-734.
• 15: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 16: Little, R. J. A. 1976. Inference about means from incomplete multivariate data. Biometrika 63(3): 593-604.
• 17: Matejcik, F. J., and B. L. Nelson. 1995. Two-stage multiple comparisons with the best for computer simulation. Operations Research 43: 633-640.
• 18: Barry L. Nelson , Frank J. Matejcik, Using common random numbers for indifference-zone selection and multiple comparisons in simulation, Management Science, v.41 n.12, p.1935-1945, Dec. 1995 [doi>10.1287/mnsc.41.12.1935]
• 19: Nelson, B. L., J. Swann, D. Goldsman, and W. Song. 1999. Simple procedures for selecting the best simulated system when the number of alternatives is large. Technical report, Northwestern University, Department of Industrial Engineering and Management Science, Evanston, IL.
• 20: Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics A7: 799-811.