Output analysis: output analysis for simulations

ABSTRACT

This paper reviews statistical methods for analyzing output data from computer simulations of single systems. In particular, it focuses on the estimation of steady-state system parameters. The estimation techniques include the replication/deletion approach, the regenerative method, the batch means method, and the standardized time series method.

REFERENCES

• 1: Alexopoulos, C., and A. F. Seila. 1998. Output data analysis. In Handbook of Simulation, ed. J. Banks, Chapter 7, New York: John Wiley & Sons.
• 2: Christos Alexopoulos , George S. Fishman , Andrew F. Seila, Computational experience with the batch means method, Proceedings of the 29th conference on Winter simulation, p.194-201, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268477]
• 3: Billingsley, P. 1968. Convergence of probability measures, Wiley, New York.
• 4: Paul Bratley , Bennett L. Fox , Linus E. Schrage, A guide to simulation (2nd ed.), Springer-Verlag New York, Inc., New York, NY, 1987
• 5: Chance, F., and L. W. Schruben. 1992. Establishing a truncation point in simulation output. Technical Report, School of Operations Research and Industrial Engineering, Cornell University, Ithaca, New York.
• 6: Charnes, J. M. 1989. Statistical analysis of multivariate discrete-event simulation output. Ph.D. Thesis, Department of Operations and Management Science, University of Minnesota, Minneapolis, Minnesota.
• 7: John M. Charnes, Power comparisons for the multivariate batch-means method, Proceedings of the 22nd conference on Winter simulation, p.281-287, December 09-12, 1990, New Orleans, Louisiana, United States
• 8: John M. Charnes, Multivariate simulation output analysis, Proceedings of the 23rd conference on Winter simulation, p.187-193, December 08-11, 1991, Phoenix, Arizona, United States
• 9: Der-Fa R. Chen , Andrew F. Seila, Multivariate inference in stationary simulation using batch means, Proceedings of the 19th conference on Winter simulation, p.302-304, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318436]
• 10: Chien, C.-H. 1989. Small sample theory for steady state confidence intervals. Technical Report No. 37, Department of Operations Research, Stanford University, Palo Alto, California.
• 11: Chiahon Chien , David Goldsman , Benjamin Melamed, Large-sample results for batch means, Management Science, v.43 n.9, p.1288-1295, Sept. 1997 [doi>10.1287/mnsc.43.9.1288]
• 12: Chow, Y. S., and H. Robbins. 1965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics 36:457-462.
• 13: Conway, R. W. 1963. Some tactical problems in digital simulation. Management Science 10:47-61.
• 14: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, I: General Multiserver Queues, Journal of the ACM (JACM), v.21 n.1, p.103-113, Jan. 1974 [doi>10.1145/321796.321805]
• 15: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, II: Markov Chains, Journal of the ACM (JACM), v.21 n.1, p.114-123, Jan. 1974 [doi>10.1145/321796.321806]
• 16: Crane, M. A., and D. L. Iglehart. 1975. Simulating stable stochastic systems III: Regenerative processes and discrete-event simulations. Operations Research 23:33-45.
• 17: Halim Damerdji, Strong consistency of the variance estimator in steady-state simulation output analysis, Mathematics of Operations Research, v.19 n.2, p.494-512, May 1994 [doi>10.1287/moor.19.2.494]
• 18: Fishman, G. S. 1973. Statistical analysis for queueing simulations. Management Science 20:363-369.
• 19: Fishman, G. S. 1974. Estimation of multiserver queueing simulations. Operations Research 22:72-78.
• 20: George S. Fishman, Principles of Discrete Event Simulation, John Wiley & Sons, Inc., New York, NY, 1978
• 21: Fishman, G. S. 1996. Monte Carlo: Concepts, algorithms, and applications. New York: Springer Verlag.
• 22: Fishman, G. S. 2000. Private communication.
• 23: Fishman, G. S., and L. S. Yarberry. 1997. An implementation of the batch means method. INFORMS Journal on Computing 9:296-310.
• 24: Gafarian, A. V., C. J. Ancker, and F. Morisaku. 1978. Evaluation of commonly used rules for detecting steady-state in computer simulation. Naval Research Logistics Quarterly 25:511-529.
• 25: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 26: D. Goldsman , M. Meketon , L. Schruben, Properties of standardized time series weighted area variance estimators, Management Science, v.36 n.5, p.602-612, May 1990 [doi>10.1287/mnsc.36.5.602]
• 27: Goldsman, D., and L. W. Schruben. 1984. Asymptotic properties of some confidence interval estimators for simulation output. Management Science 30:1217-1225.
• 28: D. Goldsman , L. Schruben, New confidence interval estimators using standardized time series, Management Science, v.36 n.3, p.393-397, Mar. 1990 [doi>10.1287/mnsc.36.3.393]
• 29: Goldsman, D., L. W. Schruben, and J. J. Swain. 1994. Tests for transient means in simulated time series. Naval Research Logistics 41:171-187.
• 30: Heidelberger, P., and P. A. W. Lewis. 1984. Quantile estimation in dependent sequences. Operations Research 32:185-209.
• 31: Philip Heidelberger , Peter D. Welch, A spectral method for confidence interval generation and run length control in simulations, Communications of the ACM, v.24 n.4, p.233-245, April 1981 [doi>10.1145/358598.358630]
• 32: Iglehart, D. L. 1975. Simulating stable stochastic systems, V: Comparison of ratio estimators. Naval Research Logistics Quarterly 22:553-565.
• 33: Donald L. Iglehart, Simulating Stable Stochastic Systems, VI: Quantile Estimation, Journal of the ACM (JACM), v.23 n.2, p.347-360, April 1976 [doi>10.1145/321941.321954]
• 34: Iglehart, D. L. 1978. The regenerative method for simulation analysis. In Current Trends in Programming Methodology, Vol. III, eds. K. M. Chandy, and K. M. Yeh, 52-71. Prentice-Hall, Englewood Cliffs, New Jersey.
• 35: Kelton, W. D. 1989. Random initialization methods in simulation. IIE Transactions 21:355-367.
• 36: Law, A. M., and J. S. Carson. 1979. A sequential procedure for determining the length of a steady-state simulation. Operations Research 27:1011-1025.
• 37: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 38: Mechanic, H., and W. McKay. 1966. Confidence intervals for averages of dependent data in simulations II. Technical Report ASDD 17-202, IBM Corporation, Yorktown Heights, New York.
• 39: Marc S. Meketon , Bruce Schmeiser, Overlapping batch means: something for nothing?, Proceedings of the 16th conference on Winter simulation, p.226-230, December 28-30, 1984, Dallas, TX
• 40: Moore, L. W. 1980. Quantile estimation in regenerative processes. Ph.D. Thesis, Curriculum in Operations Research and Systems Analysis, University of North Carolina, Chapel Hill, North Carolina.
• 41: Nadas, A. 1969. An extension of the theorem of Chow and Robbins on sequential confidence intervals for the mean. Annals of Mathematical Statistics 40:667-671.
• 42: Ockerman, D. H. 1995. Initialization bias tests for stationary stochastic processes based upon standardized time series techniques. Ph.D. Thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia.
• 43: Sidney I. Resnick, Adventures in stochastic processes, Birkhauser Verlag, Basel, Switzerland, 1992
• 44: Robert G. Sargent , Keebom Kang , David Goldsman, An investigation of finite-sample behavior of confidence interval estimators, Operations Research, v.40 n.5, p.898-913, Sept.–Oct. 1992 [doi>10.1287/opre.40.5.898]
• 45: Schmeiser, B. W. 1982. Batch size effects in the analysis of simulation output. Operations Research 30:556-568.
• 46: Thomas J. Schriber , Richard W. Andrews, Interactive analysis of simulation output by the method of batch means, Proceedings of the 11th conference on Winter simulation, p.513-526, December 03-05, 1979, San Diego, CA, United States
• 47: Schruben, L. W. 1982. Detecting initialization bias in simulation output. Operations Research 30:569-590.
• 48: Schruben, L. W. 1983. Confidence interval estimation using standardized time series. Operations Research 31:1090-1108.
• 49: Schruben, L. W., H. Singh, and L. Tierney. 1983. Optimal tests for initialization bias in simulation output. Operations Research 31:1167-1178.
• 50: Seila, A. F. 1982a. A batching approach to quantile estimation in regenerative simulations. Management Science 28:573-581.
• 51: Seila, A. F. 1982b. Percentile estimation in discrete event simulation. Simulation 39:193-200.
• 52: Wheyming Tina Song , Bruce W. Schmeiser, Optimal mean-squared-error batch sizes, Management Science, v.41 n.1, p.110-123, Jan. 1995 [doi>10.1287/mnsc.41.1.110]
• 53: von Neumann, J. 1941a. The mean square difference. Annals of Mathematical Statistics 12:153-162.
• 54: von Neumann, J. 1941b. Distribution of the ratio of the mean square successive difference and the variance. Annals of Mathematical Statistics 12:367-395.
• 55: Welch, P. D. 1983. The statistical analysis of simulation results. In The computer performance modeling handbook, ed. S. Lavenberg, 268-328. New York: Academic Press.
• 56: Peter D. Welch, On the relationship between batch means, overlapping means and spectral estimation, Proceedings of the 19th conference on Winter simulation, p.320-323, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318440]
• 57: Wilson, J. R., and A. A. B. Pritsker. 1978a. A survey of research on the simulation startup problem. Simulation 31:55-58.
• 58: Wilson, J. R., and A. A. B. Pritsker. 1978b. Evaluation of startup policies in simulation experiments. Simulation 31:79-89.
• 59: Yarberry, L. S. 1993. Incorporating a dynamic batch size selection mechanism in a fixed-sample-size batch means procedure. Ph.D. dissertation, Department of Operations Research, University of North Carolina, Chapel Hill, North Carolina.
• 60: Young, L. C. 1941. On randomness of order sequences. Annals of Mathematical Statistics 12:293-300.