Bootstrap methods in computer simulation experiments

ABSTRACT

We critically review the work that has been done in applying basic, smoothed and parametric bootstrap methods to simulation experiments. We develop a framework to classify bootstrap methods in this context and use it to compare various bootstrap schemes. Most bootstrap methods are hard to analyse theoretically. An exception is the parametric case for which a detailed analysis can be carried out. An interesting result in this case is that, whereas in standard statistical experiments bootstrap samples give only information about the variance of a statistic and not its mean, this turns out not to be so in simulation experiments. Thus parametric bootstrap samples can be advantageously included in estimates of the responses of interest.

REFERENCES

• 1: Banks, D.L. 1989. Improving the Bayesian bootstrap. Unpublished paper. Dept of Pure Mathematics and Mathematical Statistics, Cambridge University.
• 2: Russell R. Barton , Lee W. Schruben, Uniform and bootstrap resampling of empirical distributions, Proceedings of the 25th conference on Winter simulation, p.503-508, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256699]
• 3: Paul Bratley , Bennett L Fox , Linus E Schrage, A guide to simulation (2nd ed.), Springer-Verlag New York, Inc., New York, NY, 1986
• 4: Russell C. H. Cheng, Selecting input models, Proceedings of the 26th conference on Winter simulation, p.184-191, December 11-14, 1994, Orlando, Florida, United States
• 5: Cheng, R.C.H. and Holland, W. 1995a. The effect of input parameters on the variability of simulation output. In Proceedings of UKSS'95, The Second Conference of the U.K. Simulation Society (ed. R.C.H. Cheng and R.J. Pooley), UK Simulation Society, EUCS Reprographics, Edinburgh Univ., 29- 36.
• 6: Cheng, R.C.H. and Holland, W. I995b. The sensitivity of computer simulation experiments to errors in input data. Accepted for SAMO95 International Symposium, Sept. 1995, Belgirate, Italy.
• 7: DiCiccio, T.J. and Romano, J.P. 1988. A review of bootstrap confidence intervals. J. R. Statist. ~ooc. B, 50, 338-354.
• 8: Efron, B. 1982. The jackknife, the bootstrap and other resampling plans. Volume 38 of CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM.
• 9: Efron, B. and Tibshirani, R.J. 1993. An Introduction to the Bootstrap. New York and London: Chapman and Hall.
• 10: Hinkley D.J. 1988. Bootstrap methods. J. R. S~a~ist. Soc. B, 50, 321-337.
• 11: Kim, Y. B., Haddock, J. and Willemain, T. R. 1993a. The binary bootstrap: inference with correlated data. Commun. oCtatist.-Szmula., 22, 205-216.
• 12: Y. B. Kim , T. R. Willemain , J. Haddock , G. C. Runger, The threshold bootstrap: a new approach to simulation output analysis, Proceedings of the 25th conference on Winter simulation, p.498-502, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256697]
• 13: Barry L. Nelson , D. Richards, Control variate remedies, Operations Research, v.38 n.6, p.974-992, Nov./Dec. 1990 [doi>10.1287/opre.38.6.974]
• 14: Rubin, D.B. 1981. The Bayesian bootstrap. Ann. Statist. 9, 130-134.
• 15: Shiue, W.-K., Xu, C.-W. and Rea, C.B. 1993. Bootstrap confidence intervals for simulation outputs. J. Statzst. Compu~. Simul., 45, 249-255.
• 16: Shanker, A. and KeIton, W. D. 1994. Measuring output error due to input error in simulation: analysis of fitted vs. mixed empirical distributions for queues. To appear.
• 17: Silverman, B.W. and Young, G.A. 1987. The bootstrap: to smooth or not to smooth? Biome~rika, 74, 469-479.
• 18: Young, G.A. 1990. Alternative smoothed bootstraps. j.R. Sta~s~. Soc. B, 52,477-484.
• 19: Enver Yücesan, Nonparametric techniques in simulation analysis: a tutorial, Proceedings of the 26th conference on Winter simulation, p.99-105, December 11-14, 1994, Orlando, Florida, United States

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

Additional Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

General Terms:
Experimentation, Theory