Special topics on simulation analysis: to batch or not to batch

ABSTRACT

When designing steady-state computer simulation experiments, one is often faced with the choice of batching observations in one long run or replicating a number of smaller runs. Both methods are potentially useful in simulation output analysis. In its simplest form, the choice boils down to: Should we divide one long run into <i>b</i> adjacent, nonoverlapping batches, each of size <i>m</i>? Or should we conduct <i>b</i> independent replications, each of length <i>m?</i>

We give results and examples to lend insight as to when one method might be preferred over the other. In the steady state case, batching and replication perform about the same in terms of estimating the mean and variance parameter, though replication tends to do better than batching when it comes to the performance of confidence intervals for the mean. On the other hand, batching can often do better than replication when it comes to point and confidence-interval estimation of the steady-state mean in the presence of an initial transient. This is not particularly surprising, and is a common rule ofthumb in the folklore.

REFERENCES

• 1: Alexopoulos, C., and D. Goldsman. 2003. To batch or not to batch? Technical Report, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA.
• 2: Alexopoulos, C., and A. F. Seila. 1998. Output data analysis. In Handbook of Simulation, ed. J. Banks, Chapter.7. New York: John Wiley and Sons.
• 3: Billingsley, P. 1968. Convergence of Probability Measures. New York: John Wiley and Sons
• 4: Carlstein, E. 1986. The use of subseries for estimating the variance of a general statistic from a stationary sequence. Annals of Statistics 14:1171--1179.
• 5: Chiahon Chien , David Goldsman , Benjamin Melamed, Large-sample results for batch means, Management Science, v.43 n.9, p.1288-1295, Sept. 1997 [doi>10.1287/mnsc.43.9.1288]
• 6: Evans, M., N. Hastings, and B. Peacock. 2000. Statistical Distributions, 3rd ed. New York: John Wiley and Sons.
• 7: George S. Fishman, Discrete-event simulation, Springer-Verlag, London, 2001
• 8: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 9: Glynn, P. W., and W. Whitt. 1991. Estimating the asymp-totic variance with batch means. Operations Research Letters 10:431--435.
• 10: Goldsman, D., and M. S. Meketon. 1986. A comparison of several variance estimators. Technical Report #J- 85--12, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA.
• 11: Goldsman, D., L. W. Schruben, and J. J. Swain. 1994. Tests for transient means in simulated time series. Naval Research Logistics 41:171--187.
• 12: Kang, K., and D. Goldsman. 1990. The correlation between mean and variance estimators in computer simulation. IIE Transactions 22:15--23.
• 13: Krishman, M. 1968. Series representation of the doubly non-central t distribution. Journal of the American Statistical Association 63(323):1004--1012.
• 14: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 15: Robert G. Sargent , Keebom Kang , David Goldsman, An investigation of finite-sample behavior of confidence interval estimators, Operations Research, v.40 n.5, p.898-913, Sept.–Oct. 1992 [doi>10.1287/opre.40.5.898]
• 16: Wheyming Tina Song , Bruce W. Schmeiser, Optimal mean-squared-error batch sizes, Management Science, v.41 n.1, p.110-123, Jan. 1995 [doi>10.1287/mnsc.41.1.110]
• 17: Titus, B. D. 1985. Modified confidence intervals for the mean of an autoregressive process. Technical Report No. 34, Department of Operations Research, Stanford University, Stanford, CA.
• 18: Ward Whitt, The efficiency of one long run versus independent replication in steady-state simulation, Management Science, v.37 n.6, p.645-666, June 1991 [doi>10.1287/mnsc.37.6.645]