Advanced methods for simulation output analysis

ABSTRACT

This paper reviews statistical methods for analyzing output data from computer simulations of single systems. In particular, it focuses on the problems of choosing initial conditions and estimating steady-state system parameters. The estimation techniques include the replication/deletion approach, the regenerative method, the batch means method, the standardized time series method, the autoregressive method, and the spectral estimation method.

REFERENCES

• 1: Christos Alexopoulos, Distribution-free confidence intervals for conditional probabilities and ratios of expectations, Management Science, v.40 n.12, p.1748-1763, Dec. 1994 [doi>10.1287/mnsc.40.12.1748]
• 2: Billingsley, P. 1968. Convergence of probability mensures, Wiley, New York.
• 3: Paul Bratley , Bennett L. Fox , Linus E. Schrage, A guide to simulation (2nd ed.), Springer-Verlag New York, Inc., New York, NY, 1987
• 4: Chance, F., and L. W. Schruben. 1992. Establishing a truncation point in simulation output. Technical Report, School of Operations Research and Industrial Engineering, Cornell University, Ithaca, New York.
• 5: Charnes, J. M. 1989. Statistical analysis of multivariate discrete-event simulation output. Ph.D. Thesis, Department of Operations and Management Science, University of Minnesota, Minneapolis, Minnesota.
• 6: John M. Charnes, Power comparisons for the multivariate batch-means method, Proceedings of the 22nd conference on Winter simulation, p.281-287, December 09-12, 1990, New Orleans, Louisiana, United States
• 7: John M. Charnes, Multivariate simulation output analysis, Proceedings of the 23rd conference on Winter simulation, p.187-193, December 08-11, 1991, Phoenix, Arizona, United States
• 8: Der-Fa R. Chen , Andrew F. Seila, Multivariate inference in stationary simulation using batch means, Proceedings of the 19th conference on Winter simulation, p.302-304, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318436]
• 9: Chien, C.-H. 1989. Small sample theory for steady state confidence intervals. Technical Report No. 37, Department of Operations Research, Stanford University, Palo Alto, California.
• 10: Chow, Y. S., and H. Robbins. t965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics 36:457-462.
• 11: Conway, R. W. 1963. Some tactical problems in digital simulation. Management Science 10:47-61.
• 12: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, I: General Multiserver Queues, Journal of the ACM (JACM), v.21 n.1, p.103-113, Jan. 1974 [doi>10.1145/321796.321805]
• 13: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, II: Markov Chains, Journal of the ACM (JACM), v.21 n.1, p.114-123, Jan. 1974 [doi>10.1145/321796.321806]
• 14: Crane, M. A., and D. L. Iglehart. 1975. Simulating stable stochastic systems IIi: Regenerative processes and discrete-event simulations. Operations Research 23:33-45.
• 15: A. A. Crane , J. Lemoine, An Introduction to the Regenerative Method for Simulation Analysis, Springer-Verlag New York, Inc., Secaucus, NJ, 1977
• 16: Halim Damerdji, Strong consistency and other properties of the spectral variance estimator, Management Science, v.37 n.11, p.1424-1440, November 1991 [doi>10.1287/mnsc.37.11.1424]
• 17: Halim Damerdji, Strong consistency of the variance estimator in steady-state simulation output analysis, Mathematics of Operations Research, v.19 n.2, p.494-512, May 1994 [doi>10.1287/moor.19.2.494]
• 18: Halim Damerdji, On the batch means and area variance estimators, Proceedings of the 26th conference on Winter simulation, p.340-344, December 11-14, 1994, Orlando, Florida, United States
• 19: Fishman, G. S. 1972. Bias considerations in simulation experiments. Operations Research 20:785- 790.
• 20: Fishman, G. S. 1973. Statistical analysis for queueing simulations. Management Science 20:363-369.
• 21: Fishman, G. S. 1974. Estimation of multiserver queueing simulations. Operations Research 22:72- 78.
• 22: Fishman, G. S. 1978a. Grouping observations in digital simulation. Management Science 24:510-521.
• 23: George S. Fishman, Principles of Discrete Event Simulation, John Wiley & Sons, Inc., New York, NY, 1978
• 24: Fishman, G. S., and L. S. Yarberry. 1993. An implementation of the batch means method. Technical Report UNC/OR/TR/93-1, Department of Operations Research, University of North Carolina, Chapel Hill, North Carolina.
• 25: Gafarian, A.V., C. J. Ancker, and F. Morisaku. 1978. Evaluation of commonly used rules for detecting steady-state in computer simulation. Naval Research Logistics Quarterly 25:511-529.
• 26: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 27: D. Goldsman , M. Meketon , L. Schruben, Properties of standardized time series weighted area variance estimators, Management Science, v.36 n.5, p.602-612, May 1990 [doi>10.1287/mnsc.36.5.602]
• 28: Goldsman, D., and L. W. Schruben. 1984. Asymptotic properties of some confidence interval estimators for simulation output. Management Science 30:1217-1225.
• 29: D. Goldsman , L. Schruben, New confidence interval estimators using standardized time series, Management Science, v.36 n.3, p.393-397, Mar. 1990 [doi>10.1287/mnsc.36.3.393]
• 30: Goldsman, D., L. W. Schruben, and J. J. Swain. 1993. Tests for transient means in simulated time series. Technical Report, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 31: Heidelberger, P., and P. A. W. Lewis. 1984. Quantile estimation in dependent sequences. Operations Research 32:185-209.
• 32: Philip Heidelberger , Peter D. Welch, A spectral method for confidence interval generation and run length control in simulations, Communications of the ACM, v.24 n.4, p.233-245, April 1981 [doi>10.1145/358598.358630]
• 33: Heidelberger, P., and P. D. Welch. 1981b. Adaptive spectral methods for simulation output analysis. IBM Journal of Research and Development 25:860-876.
• 34: Heidelberger, P., and P. D. Welch. 1983. Simulation run length control in the presence of an initial transient. Operations Research 31:1109-1144.
• 35: Iglehart, D. L. 1975. Simulating stable stochastic systems, V: Comparison of ratio estimators. Naval Research Logistics Quarterly 22:553-565.
• 36: Donald L. Iglehart, Simulating Stable Stochastic Systems, VI: Quantile Estimation, Journal of the ACM (JACM), v.23 n.2, p.347-360, April 1976 [doi>10.1145/321941.321954]
• 37: Iglehart, D. L. 1978. The regenerative method for simulation analysis. In Current Trends in Programming Methodology, Vol. III, eds. K. M. Chandy, and K. M. Yeh, 52-71. Prentice-Hall, Englewood Cliffs, New Jersey.
• 38: Kelton, W. D. 1989. Random initialization methods in simulation. IIE Transactions 21:355-367.
• 39: Kelton, W. D., and A. M. Law. 1983. A new approach for dealing with the startup problem in discrete event simulation. Naval Research Logistics Quarterly 30:641-658.
• 40: Kleijnen, J. P. C. 1974. Statistical techniques in simulation, part I, Marcel Dekker, New York, New York.
• 41: Kleijnen, j. P. C. 1975. Statistical techniques in simulation, part II, Marcel Dekker, New York, New York.
• 42: Law, A. M. 1980. Statistical analysis of the output data from terminating simulations. Naval Research Logistics Quarterly 27:131-143.
• 43: Law, A. M., and J. S. Carson. 1979. A sequential procedure for determining the length of a steady-state simulation. Operations Research 27:1011-1025.
• 44: Law, A. M., and W. D. Kelton. 1984. Confidence intervals for steady-state simulations, I: A survey of fixed sample size procedures. Operations Research 32:1221-1239.
• 45: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 46: Marc S. Meketon , Bruce Schmeiser, Overlapping batch means: something for nothing?, Proceedings of the 16th conference on Winter simulation, p.226-230, December 28-30, 1984, Dallas, TX
• 47: Moore, L. W. 1980. Quantile estimation in regenerative processes. Ph.D. Thesis, Curriculum in Operations Research and Systems Analysis, University of North Carolina, Chapel Hill, North Carolina.
• 48: Nadas, A. 1969. An extension of the theorem of Chow and Robbins on sequential confidence intervals for the mean. Annals of Mathematical Statistics 40:667-671.
• 49: Ockerman, D. H. 1995. Initialization bias tests for stationary stochastic processes based upon standardized time series techniques. Ph.D. Thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 50: Robert G. Sargent , Keebom Kang , David Goldsman, An investigation of finite-sample behavior of confidence interval estimators, Operations Research, v.40 n.5, p.898-913, Sept.–Oct. 1992 [doi>10.1287/opre.40.5.898]
• 51: Schm~iger, B. W. 1982. Batch size effects in the analysis of simulation output. Operations Research 30:556-568.
• 52: Wheyming Tina Song , Bruce W. Schmeiser, Optimal mean-squared-error batch sizes, Management Science, v.41 n.1, p.110-123, Jan. 1995 [doi>10.1287/mnsc.41.1.110]
• 53: Schriber, T. j., and R. W. Andrews. 1984. An ARMA-based confidence interval for the analysis of simulation output. American J. Math. Management Sci. 4:345-373.
• 54: Lee W. Shruben, Control of initialization bias in multivariate simulation response, Communications of the ACM, v.24 n.4, p.246-252, April 1981 [doi>10.1145/358598.358634]
• 55: Schruben, L. W. 1982. Detecting initialization bias in simulation output. Operations Research 30:569- 590.
• 56: Schruben, L. W. 1983. Confidence interval estimation using standardized time series. Operations Research 31:1090-1108.
• 57: Schruben, L. W., H. Singh, and L. Tierney. 1983. Optimal tests for initialization bias in simulation output. Operations Research 31:1167-1178.
• 58: Seila, A. F. 1982a. A batching approach to quantile estimation in regenerative simulations. Management Science 28:573-581.
• 59: Seila, A. F. 1982b. Percentile estimation in discrete event simulation. Simulation 39:193-200.
• 60: Michael Sherman, On batch means in the simulation and statistics communities, Proceedings of the 27th conference on Winter simulation, p.297-302, December 03-06, 1995, Arlington, Virginia, United States [doi>10.1145/224401.224618]
• 61: Welch, P. D. 1981. On the problem of the initial transient in steady state simulations. Technical Report, IBM Watson Research Center, Yorktown Heights, New York.
• 62: Welch, P. D. 1983. The statistical analysis of simulation results. In The computer performance modeling handbook, ed. S. Lavenberg, 268-328, Academic Press, New York.
• 63: Peter D. Welch, On the relationship between batch means, overlapping means and spectral estimation, Proceedings of the 19th conference on Winter simulation, p.320-323, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318440]
• 64: Wilson, J. R., and A. A. B. Pritsker. 1978a. A survey of research on the simulation startup problem. Simulation 31:55-58.
• 65: Wilson, J. R., and A. A. B. Pritsker. 1978b. Evaluation of startup policies in simulation experiments. Simulation 31:79-89.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.1 Simulation Theory
          Subjects: Systems theory

General Terms:
Design, Performance, Theory