Statistical analysis of simulation output: output data analysis for simulations

ABSTRACT

This paper reviews statistical methods for analyzing output data from computer simulations. First, it focuses on the estimation of steady-state system parameters. The estimation techniques include the replication/deletion approach, the regenerative method, the batch means method, and the standardized time series method. Second, it reviews recent statistical procedures to find the best system among a set of competing alternatives.

REFERENCES

• 1: Alexopoulos, C., and D. Goldsman. 2002. To batch or not to batch? Technical Report, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 2: Christos Alexopoulos , David Goldsman , Gamze Tokol, Properties of Batched Quadratic-Form Variance Parameter Estimators for Simulations, INFORMS Journal on Computing, v.13 n.2, p.149-156, March 2001 [doi>10.1287/ijoc.13.2.149.10518]
• 3: Alexopoulos, C., and A. F. Seila. 1998. Output data analysis. In Handbook of Simulation, ed. J. Banks, Chapter 7. New York: John Wiley & Sons.
• 4: Christos Alexopoulos , George S. Fishman , Andrew F. Seila, Computational experience with the batch means method, Proceedings of the 29th conference on Winter simulation, p.194-201, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268477]
• 5: Bechhofer, R. E., T. J. Santner, and D. Goldsman. 1995. Design and analysis of experiments for statistical selection, screening and multiple comparisons. New York: John Wiley & Sons.
• 6: Billingsley, P. 1968. Convergence of probability measures. New York: John Wiley & Sons.
• 7: Justin Boesel , Barry L. Nelson , Seong-Hee Kim, Using Ranking and Selection to "Clean Up" after Simulation Optimization, Operations Research, v.51 n.5, p.814-825, September 2003 [doi>10.1287/opre.51.5.814.16751]
• 8: Paul Bratley , Bennett L. Fox , Linus E. Schrage, A guide to simulation (2nd ed.), Springer-Verlag New York, Inc., New York, NY, 1987
• 9: Chance, F., and L. W. Schruben. 1992. Establishing a truncation point in simulation output. Technical Report, School of Operations Research and Industrial Engineering, Cornell University, Ithaca, New York.
• 10: Charnes, J. M. 1989. Statistical analysis of multivariate discrete-event simulation output. PhD. Thesis, Department of Operations and Management Science, University of Minnesota, Minneapolis, Minnesota.
• 11: John M. Charnes, Power comparisons for the multivariate batch-means method, Proceedings of the 22nd conference on Winter simulation, p.281-287, December 09-12, 1990, New Orleans, Louisiana, United States
• 12: John M. Charnes, Multivariate simulation output analysis, Proceedings of the 23rd conference on Winter simulation, p.187-193, December 08-11, 1991, Phoenix, Arizona, United States
• 13: Hsiao-Chang Chen , Liyi Dai , Chun-Hung Chen , Enver Yücesan, New development of optimal computing budget allocation for discrete event simulation, Proceedings of the 29th conference on Winter simulation, p.334-341, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268501]
• 14: Der-Fa R. Chen , Andrew F. Seila, Multivariate inference in stationary simulation using batch means, Proceedings of the 19th conference on Winter simulation, p.302-304, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318436]
• 15: Stephen E. Chick, Selecting the best system: a decision-theoretic approach, Proceedings of the 29th conference on Winter simulation, p.326-333, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268499]
• 16: Stephen E. Chick , Koichiro Inoue, New Procedures to Select the Best Simulated System Using Common Random Numbers, Management Science, v.47 n.8, p.1133-1149, August 2001 [doi>10.1287/mnsc.47.8.1133.10229]
• 17: Stephen E. Chick , Koichiro Inoue, New Two-Stage and Sequential Procedures for Selecting the Best Simulated System, Operations Research, v.49 n.5, p.732-743, September 2001 [doi>10.1287/opre.49.5.732.10615]
• 18: Chien, C.-H. 1989. Small sample theory for steady state confidence intervals. Technical Report No. 37, Department of Operations Research, Stanford University, Palo Alto, California.
• 19: Chiahon Chien , David Goldsman , Benjamin Melamed, Large-sample results for batch means, Management Science, v.43 n.9, p.1288-1295, Sept. 1997 [doi>10.1287/mnsc.43.9.1288]
• 20: Chow, Y. S., and H. Robbins. 1965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics 36:457--462.
• 21: Conway, R. W. 1963. Some tactical problems in digital simulation. Management Science 10:47--61.
• 22: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, I: General Multiserver Queues, Journal of the ACM (JACM), v.21 n.1, p.103-113, Jan. 1974 [doi>10.1145/321796.321805]
• 23: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, II: Markov Chains, Journal of the ACM (JACM), v.21 n.1, p.114-123, Jan. 1974 [doi>10.1145/321796.321806]
• 24: Crane, M. A., and D. L. Iglehart. 1975. Simulating stable stochastic systems III: Regenerative processes and discrete-event simulations. Operations Research 23:33--45.
• 25: Halim Damerdji, Strong consistency of the variance estimator in steady-state simulation output analysis, Mathematics of Operations Research, v.19 n.2, p.494-512, May 1994 [doi>10.1287/moor.19.2.494]
• 26: Halim Damerdji , Marvin K. Nakayama, Two-stage multiple-comparison procedures for steady-state simulations, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.9 n.1, p.1-30, Jan. 1999 [doi>10.1145/301677.301679]
• 27: Fishman, G. S. 1973. Statistical analysis for queueing simulations. Management Science 20:363--369.
• 28: Fishman, G. S. 1974. Estimation of multiserver queueing simulations. Operations Research 22:72--78.
• 29: Fishman, G. S. 1996. Monte Carlo: Concepts, algorithms, and applications. New York: Springer Verlag.
• 30: George S. Fishman, Discrete-event simulation, Springer-Verlag, London, 2001
• 31: Fishman, G. S., and L. S. Yarberry. 1997. An implementation of the batch means method. INFORMS Journal on Computing 9:296--310.
• 32: Gafarian, A. V., C. J. Ancker, and F. Morisaku. 1978. Evaluation of commonly used rules for detecting steady-state in computer simulation. Naval Research Logistics Quarterly 25:511--529.
• 33: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 34: David Goldsman , William S. Marshall , Seong-Hee Kim , Barry L. Nelson, Ranking and selection for steady-state simulation, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 35: David Goldsman , Seong-Hee Kim , William S. Marshall , Barry L. Nelson, Ranking and Selection for Steady-State Simulation: Procedures and Perspectives, INFORMS Journal on Computing, v.14 n.1, p.2-19, January 2002 [doi>10.1287/ijoc.14.1.2.7710]
• 36: D. Goldsman , M. Meketon , L. Schruben, Properties of standardized time series weighted area variance estimators, Management Science, v.36 n.5, p.602-612, May 1990 [doi>10.1287/mnsc.36.5.602]
• 37: Goldsman, D., and B. L. Nelson. 1998. Comparing systems via simulation. In Handbook of Simulation, ed. J. Banks, Chapter 8. New York: John Wiley & Sons.
• 38: Goldsman, D., and L. W. Schruben. 1984. Asymptotic properties of some confidence interval estimators for simulation output. Management Science 30:1217--1225.
• 39: D. Goldsman , L. Schruben, New confidence interval estimators using standardized time series, Management Science, v.36 n.3, p.393-397, Mar. 1990 [doi>10.1287/mnsc.36.3.393]
• 40: Goldsman, D., L. W. Schruben, and J. J. Swain. 1994. Tests for transient means in simulated time series. Naval Research Logistics 41:171--187.
• 41: Gupta, S. S. 1965. On some multiple decision (selection and ranking) rules. Technometrics 7:225--245.
• 42: Gupta, S. S., and D.-Y. Huang. 1976. Subset selection procedures for the means and variances of normal populations: Unequal sample sizes case, Sankhyā B, 38:112--128.
• 43: Heidelberger, P., and P. A. W. Lewis. 1984. Quantile estimation in dependent sequences. Operations Research 32:185--209.
• 44: Iglehart, D. L. 1975. Simulating stable stochastic systems, V: Comparison of ratio estimators. Naval Research Logistics Quarterly 22:553--565.
• 45: Donald L. Iglehart, Simulating Stable Stochastic Systems, VI: Quantile Estimation, Journal of the ACM (JACM), v.23 n.2, p.347-360, April 1976 [doi>10.1145/321941.321954]
• 46: Iglehart, D. L. 1978. The regenerative method for simulation analysis. In Current Trends in Programming Methodology, Vol. III, eds. K. M. Chandy, and K. M. Yeh, 52--71. Prentice-Hall, Englewood Cliffs, New Jersey.
• 47: Koichiro Inoue , Stephen E. Chick , Chun-Hung Chen, An empirical evaluation of several methods to select the best system, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.9 n.4, p.381-407, Oct. 1999 [doi>10.1145/352222.352226]
• 48: Kelton, W. D. 1989. Random initialization methods in simulation. IIE Transactions 21:355--367.
• 49: Seong-Hee Kim , Barry L. Nelson, A fully sequential procedure for indifference-zone selection in simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.11 n.3, p.251-273, July 2001 [doi>10.1145/502109.502111]
• 50: Law, A. M., and J. S. Carson. 1979. A sequential procedure for determining the length of a steady-state simulation. Operations Research 27:1011--1025.
• 51: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 52: Malkovich, J. F., and A. A. Afifi. 1973. On tests for multivariate normality. Journal of the American Statistical Association 68:176--179.
• 53: Mechanic, H., and W. McKay. 1966. Confidence intervals for averages of dependent data in simulations II. Technical Report ASDD 17--202, IBM Corporation, Yorktown Heights, New York.
• 54: Marc S. Meketon , Bruce Schmeiser, Overlapping batch means: something for nothing?, Proceedings of the 16th conference on Winter simulation, p.226-230, December 28-30, 1984, Dallas, TX
• 55: Moore, L. W. 1980. Quantile estimation in regenerative processes. PhD. Thesis, Curriculum in Operations Research and Systems Analysis, University of North Carolina, Chapel Hill, North Carolina.
• 56: Marvin K. Nakayama, Selecting the best system in steady-state simulations using batch means, Proceedings of the 27th conference on Winter simulation, p.362-366, December 03-06, 1995, Arlington, Virginia, United States [doi>10.1145/224401.224637]
• 57: Barry L. Nelson , Frank J. Matejcik, Using common random numbers for indifference-zone selection and multiple comparisons in simulation, Management Science, v.41 n.12, p.1935-1945, Dec. 1995 [doi>10.1287/mnsc.41.12.1935]
• 58: Barry L. Nelson , Julie Swann , David Goldsman , Wheyming Song, Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large, Operations Research, v.49 n.6, p.950-963, November 2001 [doi>10.1287/opre.49.6.950.10019]
• 59: Ockerman, D. H. 1995. Initialization bias tests for stationary stochastic processes based upon standardized time series techniques. PhD. Thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 60: Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in Statistics --- Theory and Methods A 7:799--811.
• 61: Robert G. Sargent , Keebom Kang , David Goldsman, An investigation of finite-sample behavior of confidence interval estimators, Operations Research, v.40 n.5, p.898-913, Sept.–Oct. 1992 [doi>10.1287/opre.40.5.898]
• 62: Schmeiser, B. W. 1982. Batch size effects in the analysis of simulation output. Operations Research 30:556--568.
• 63: Thomas J. Schriber , Richard W. Andrews, Interactive analysis of simulation output by the method of batch means, Proceedings of the 11th conference on Winter simulation, p.513-526, December 03-05, 1979, San Diego, CA, United States
• 64: Schruben, L. W. 1982. Detecting initialization bias in simulation output. Operations Research 30:569--590.
• 65: Schruben, L. W. 1983. Confidence interval estimation using standardized time series. Operations Research 31:1090--1108.
• 66: Schruben, L. W., H. Singh, and L. Tierney. 1983. Optimal tests for initialization bias in simulation output. Operations Research 31:1167--1178.
• 67: Seila, A. F. 1982a. A batching approach to quantile estimation in regenerative simulations. Management Science 28:573--581.
• 68: Seila, A. F. 1982b. Percentile estimation in discrete event simulation. Simulation 39:193--200.
• 69: Wheyming Tina Song , Bruce W. Schmeiser, Optimal mean-squared-error batch sizes, Management Science, v.41 n.1, p.110-123, Jan. 1995 [doi>10.1287/mnsc.41.1.110]
• 70: Natalie M. Steiger , James R. Wilson, Convergence Properties of the Batch Means Method for Simulation Output Analysis, INFORMS Journal on Computing, v.13 n.4, p.277-293, September 2001 [doi>10.1287/ijoc.13.4.277.9737]
• 71: Steiger, N. M., and J. R. Wilson. 2002a. An improved batch means procedure for simulation output analysis. Technical Report, Department of Industrial Engineering, North Carolina State University, Raleigh, North Carolina.
• 72: Natalie M. Steiger , Christos Alexopoulos , David Goldsman , Emily K. Lada , James R. Wilson , Faker Zouaoui, Output analysis: ASAP2: an improved batch means procedure for simulation output analysis, Proceedings of the 34th conference on Winter simulation: exploring new frontiers, December 08-11, 2002, San Diego, California
• 73: von Neumann, J. 1941a. Distribution of the ratio of the mean square successive difference and the variance. Annals of Mathematical Statistics 12:367--395.
• 74: von Neumann, J. 1941b. The mean square difference. Annals of Mathematical Statistics 12:153--162.
• 75: Welch, P. D. 1983. The statistical analysis of simulation results. In The Computer Performance Modeling Handbook, ed. S. Lavenberg, 268--328. New York: Academic Press.
• 76: Peter D. Welch, On the relationship between batch means, overlapping means and spectral estimation, Proceedings of the 19th conference on Winter simulation, p.320-323, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318440]
• 77: Wilcox, R. R. 1984. A table for Rinott's selection procedure. Journal of Quality Technology 16:97--100.
• 78: Wilson, J. R., and A. A. B. Pritsker. 1978a. A survey of research on the simulation startup problem. Simulation 31:55--58.
• 79: Wilson, J. R., and A. A. B. Pritsker. 1978b. Evaluation of startup policies in simulation experiments. Simulation 31:79--89.
• 80: Yarberry, L. S. 1993. Incorporating a dynamic batch size selection mechanism in a fixed-sample-size batch means procedure. PhD. dissertation, Department of Operations Research, University of North Carolina, Chapel Hill, North Carolina.