Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

This paper reviews statistical methods for analyzing output data from computer simulations. First, it focuses on the estimation of steady-state system parameters. The estimation techniques include the replication/deletion approach, the regenerative method, the batch means method, and methods based on standardized time series. Second, it reviews recent statistical procedures to find the best system among a set of competing alternatives.

REFERENCES

• 1: Alexopoulos, C., N. T. Argon, D. Goldsman, G. Tokol, and J. R. Wilson. 2005a. Overlapping variance estimators for simulation: I. Theory. Technical Report, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 2: Alexopoulos, C., N. T. Argon, D. Goldsman, N. M. Steiger, G. Tokol, and J. R. Wilson. 2005b. Overlapping variance estimators for simulation: II. Implementation and evaluation. Technical Report, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 3: Christos Alexopoulos , David Goldsman, To batch or not to batch?, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.14 n.1, p.76-114, January 2004 [doi>10.1145/974734.974738]
• 4: Alexopoulos, C., and A. F. Seila. 1998. Output data analysis. In Handbook of Simulation, ed. J. Banks, Chapter 7. New York: John Wiley & Sons.
• 5: Anderson, T. W. 1984. An Introduction to multivariate statistical analysis. New York: John Wiley & Sons.
• 6: Bechhofer, R. E., T. J. Santner, and D. Goldsman. 1995. Design and analysis of experiments for statistical selection, screening and multiple comparisons. New York: John Wiley & Sons.
• 7: Billingsley, P. 1968. Convergence of probability measures. New York: John Wiley & Sons.
• 8: Charnes, J. M. 1989. Statistical analysis of multivariate discrete-event simulation output. Ph.D. Thesis, Department of Operations and Management Science, University of Minnesota, Minneapolis, Minnesota.
• 9: John M. Charnes, Power comparisons for the multivariate batch-means method, Proceedings of the 22nd conference on Winter simulation, p.281-287, December 09-12, 1990, New Orleans, Louisiana, United States
• 10: John M. Charnes, Multivariate simulation output analysis, Proceedings of the 23rd conference on Winter simulation, p.187-193, December 08-11, 1991, Phoenix, Arizona, United States
• 11: Hsiao-Chang Chen , Liyi Dai , Chun-Hung Chen , Enver Yücesan, New development of optimal computing budget allocation for discrete event simulation, Proceedings of the 29th conference on Winter simulation, p.334-341, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268501]
• 12: Der-Fa R. Chen , Andrew F. Seila, Multivariate inference in stationary simulation using batch means, Proceedings of the 19th conference on Winter simulation, p.302-304, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318436]
• 13: Stephen E. Chick, Selecting the best system: a decision-theoretic approach, Proceedings of the 29th conference on Winter simulation, p.326-333, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268499]
• 14: Stephen E. Chick , Koichiro Inoue, New Two-Stage and Sequential Procedures for Selecting the Best Simulated System, Operations Research, v.49 n.5, p.732-743, September 2001 [doi>10.1287/opre.49.5.732.10615]
• 15: Stephen E. Chick , Koichiro Inoue, New Procedures to Select the Best Simulated System Using Common Random Numbers, Management Science, v.47 n.8, p.1133-1149, August 2001 [doi>10.1287/mnsc.47.8.1133.10229]
• 16: Chien, C.-H. 1989. Small sample theory for steady state confidence intervals. Technical Report No. 37, Department of Operations Research, Stanford University, Palo Alto, California.
• 17: Chiahon Chien , David Goldsman , Benjamin Melamed, Large-sample results for batch means, Management Science, v.43 n.9, p.1288-1295, Sept. 1997 [doi>10.1287/mnsc.43.9.1288]
• 18: Chow, Y. S., and H. Robbins. 1965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics 36:457--462.
• 19: Conway, R. W. 1963. Some tactical problems in digital simulation. Management Science 10:47--61.
• 20: Crane, M. A., and D. L. Iglehart. 1975. Simulating stable stochastic systems III: Regenerative processes and discrete-event simulations. Operations Research 23:33--45.
• 21: Halim Damerdji, Strong consistency of the variance estimator in steady-state simulation output analysis, Mathematics of Operations Research, v.19 n.2, p.494-512, May 1994 [doi>10.1287/moor.19.2.494]
• 22: Halim Damerdji, On the batch means and area variance estimators, Proceedings of the 26th conference on Winter simulation, p.340-344, December 11-14, 1994, Orlando, Florida, United States
• 23: Halim Damerdji , Marvin K. Nakayama, Two-stage multiple-comparison procedures for steady-state simulations, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.9 n.1, p.1-30, Jan. 1999 [doi>10.1145/301677.301679]
• 24: Durrett, R. 2005. Probability: Theory and examples, 3d Ed. Belmont, California: Brooks/Cole-Thomson Learning.
• 25: Fishman, G. S. 1973. Statistical analysis for queueing simulations. Management Science 20:363--369.
• 26: Fishman, G. S. 1974. Estimation of multiserver queueing simulations. Operations Research 22:72--78.
• 27: George S. Fishman, Principles of Discrete Event Simulation, John Wiley & Sons, Inc., New York, NY, 1978
• 28: George S. Fishman, Discrete-event simulation, Springer-Verlag, London, 2001
• 29: Fishman, G. S., and L. S. Yarberry. 1997. An implementation of the batch means method. INFORMS Journal on Computing 9:296--310.
• 30: Robert D. Foley , David Goldsman, Confidence intervals using orthonormally weighted standardized time series, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.9 n.4, p.297-325, Oct. 1999 [doi>10.1145/352222.352223]
• 31: Peter W. Glynn , Donald L. Iglehart, Large-sample theory for standardized time series: an overview, Proceedings of the 17th conference on Winter simulation, p.129-134, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253097]
• 32: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 33: Glynn, P. W., and W. Whitt. 1991. Estimating the asymptotic variance with batch means. Operations Research Letters 10:431--435.
• 34: David Goldsman , Keebom Kang , Andrew F. Seila, Cramer-Von Mises Variance Estimators for Simulations, Operations Research, v.47 n.2, p.299-309, February 1999 [doi>10.1287/opre.47.2.299]
• 35: David Goldsman , William S. Marshall , Seong-Hee Kim , Barry L. Nelson, Ranking and selection for steady-state simulation, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 36: David Goldsman , Seong-Hee Kim , William S. Marshall , Barry L. Nelson, Ranking and Selection for Steady-State Simulation: Procedures and Perspectives, INFORMS Journal on Computing, v.14 n.1, p.2-19, January 2002 [doi>10.1287/ijoc.14.1.2.7710]
• 37: D. Goldsman , M. Meketon , L. Schruben, Properties of standardized time series weighted area variance estimators, Management Science, v.36 n.5, p.602-612, May 1990 [doi>10.1287/mnsc.36.5.602]
• 38: Goldsman, D., and B. L. Nelson. 1998. Comparing systems via simulation. In Handbook of Simulation, ed. J. Banks, Chapter 8. New York: John Wiley & Sons.
• 39: D. Goldsman , L. Schruben, New confidence interval estimators using standardized time series, Management Science, v.36 n.3, p.393-397, Mar. 1990 [doi>10.1287/mnsc.36.3.393]
• 40: Gupta, S. S. 1965. On some multiple decision (selection and ranking) rules. Technometrics 7:225--245.
• 41: Gupta, S. S., and D.-Y. Huang. 1976. Subset selection procedures for the means and variances of normal populations: Unequal sample sizes case, Sankhyā B, 38:112--128.
• 42: Heidelberger, P., and P. A. W. Lewis. 1984. Quantile estimation in dependent sequences. Operations Research 32:185--209.
• 43: Hong, L. J., and B. L. Nelson. 2005. The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. IIE Transactions 37:623--634.
• 44: Iglehart, D. L. 1975. Simulating stable stochastic systems, V: Comparison of ratio estimators. Naval Research Logistics Quarterly 22:553--565.
• 45: Donald L. Iglehart, Simulating Stable Stochastic Systems, VI: Quantile Estimation, Journal of the ACM (JACM), v.23 n.2, p.347-360, April 1976 [doi>10.1145/321941.321954]
• 46: Koichiro Inoue , Stephen E. Chick , Chun-Hung Chen, An empirical evaluation of several methods to select the best system, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.9 n.4, p.381-407, Oct. 1999 [doi>10.1145/352222.352226]
• 47: Seong-Hee Kim , Barry L. Nelson, A fully sequential procedure for indifference-zone selection in simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.11 n.3, p.251-273, July 2001 [doi>10.1145/502109.502111]
• 48: Seong-Hee Kim , Barry L. Nelson, Selecting the best system: selecting the best system: theory and methods, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
• 49: Kim, S.-H., and B. L. Nelson. 2005. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research, forthcoming.
• 50: Koenig, L. W., and A. M. Law. 1985. A procedure for selecting a subset of size m containing the I best of k independent normal populations, with applications to simulation. Communications in Statistics---Simulation and Computation B14:719--734.
• 51: Law, A. M., and J. S. Carson. 1979. A sequential procedure for determining the length of a steady-state simulation. Operations Research 27:1011--1025.
• 52: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 53: Marc S. Meketon , Bruce Schmeiser, Overlapping batch means: something for nothing?, Proceedings of the 16th conference on Winter simulation, p.226-230, December 28-30, 1984, Dallas, TX
• 54: Moore, L. W. 1980. Quantile estimation in regenerative processes. Ph.D. Thesis, Curriculum in Operations Research and Systems Analysis, University of North Carolina, Chapel Hill, North Carolina.
• 55: Nakayama, M. K. 1997. Multiple-comparison procedures for steady-state simulations. Annals of Statistics 25:2433--2450.
• 56: Marvin K. Nakayama, Selecting the best system in steady-state simulations using batch means, Proceedings of the 27th conference on Winter simulation, p.362-366, December 03-06, 1995, Arlington, Virginia, United States [doi>10.1145/224401.224637]
• 57: Barry L. Nelson , Julie Swann , David Goldsman , Wheyming Song, Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large, Operations Research, v.49 n.6, p.950-963, November 2001 [doi>10.1287/opre.49.6.950.10019]
• 58: Ockerman, D. H. 1995. Initialization bias tests for stationary stochastic processes based upon standardized time series techniques. Ph.D. Thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 59: Philipp, W., and W. Stout. 1975. Almost sure invariance principles for partial sums of weakly dependent random variables. Memoirs of the American Mathematical Society 2.
• 60: Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics --- Theory and Methods A 7:799--811.
• 61: Robert G. Sargent , Keebom Kang , David Goldsman, An investigation of finite-sample behavior of confidence interval estimators, Operations Research, v.40 n.5, p.898-913, Sept.–Oct. 1992 [doi>10.1287/opre.40.5.898]
• 62: Schmeiser, B. W. 1982. Batch size effects in the analysis of simulation output. Operations Research 30:556--568.
• 63: Schmeiser, B. W. 1986. Personal communication.
• 64: Schruben, L. W. 1983. Confidence interval estimation using standardized time series. Operations Research 31:1090--1108.
• 65: Seila, A. F. 1982a. A batching approach to quantile estimation in regenerative simulations. Management Science 28:573--581.
• 66: Seila, A. F. 1982b. Percentile estimation in discrete event simulation. Simulation 39:193--200.
• 67: Wheyming Tina Song , Bruce W. Schmeiser, Optimal mean-squared-error batch sizes, Management Science, v.41 n.1, p.110-123, Jan. 1995 [doi>10.1287/mnsc.41.1.110]
• 68: Song, W.-M. T. 1996. On the estimation of optimal batch sizes in the analysis of simulation output analysis. European Journal of Operations Research 88:304--309.
• 69: Natalie M. Steiger , Emily K. Lada , James R. Wilson , Jeffrey A. Joines , Christos Alexopoulos , David Goldsman, ASAP3: a batch means procedure for steady-state simulation analysis, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.15 n.1, p.39-73, January 2005 [doi>10.1145/1044322.1044325]
• 70: von Neumann, J. 1941a. The mean square difference. Annals of Mathematical Statistics 12:153--162.
• 71: von Neumann, J. 1941b. Distribution of the ratio of the mean square successive difference and the variance. Annals of Mathematical Statistics 12:367--395.
• 72: Welch, P. D. 1983. The statistical analysis of simulation results. In The Computer Performance Modeling Handbook, ed. S. Lavenberg, 268--328. New York: Academic Press.
• 73: Peter D. Welch, On the relationship between batch means, overlapping means and spectral estimation, Proceedings of the 19th conference on Winter simulation, p.320-323, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318440]
• 74: Wilcox, R. R. 1984. A table for Rinott's selection procedure. Journal of Quality Technology 16:97--100.
• 75: Wilson, J. R., and A. A. B. Pritsker. 1978a. A survey of research on the simulation startup problem. Simulation 31:55--58.
• 76: Wilson, J. R., and A. A. B. Pritsker. 1978b. Evaluation of startup policies in simulation experiments. Simulation 31:79--89.
• 77: Young, L. C. 1941. On randomness of order sequences. Annals of Mathematical Statistics 12:293--300.