Calculation of confidence intervals for simulation output

ABSTRACT

This article is concerned with the calculation of confidence intervals for simulation output that is dependent on two sources of variability. One, referred to as <i>simulation variability</i>, arises from the use of random numbers in the simulation itself; and the other, referred to as <i>parameter variability</i>, arises when the input parameters are unknown and have to be estimated from observed data. Three approaches to the calculation of confidence intervals are presented--the traditional asymptotic normality theory approach, a bootstrap approach and a new method which produces a conservative approximation based on performing just two simulation runs at carefully selected parameter settings. It is demonstrated that the traditional and bootstrap approaches provide similar degrees of accuracy and that whilst the new method may sometimes be very conservative, it can be calculated in a small fraction of the computational time of the exact methods.

REFERENCES

• 1: Apostol, T. M. 1974. Mathematical Analysis. Addison-Wesley, Reading, Mass.
• 2: Cheng-Shang Chang , Philip Heidelberger , Perwez Shahabuddin, Fast simulation of packet loss rates in a shared buffer communications switch, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.5 n.4, p.306-325, Oct. 1995 [doi>10.1145/226275.226277]
• 3: Cheng, R. C. H. and Holland, W. 1997. Sensitivity of computer simulation experiments to errors in input data. J. Statist. Comput. Simul. 57, 219--241.
• 4: Cheng, R. C. H. and Holland, W. 1998. Two-point methods for assessing variability in simulation output. J. Statist. Comput. Simul. 60, 183--205.
• 5: Cox, D. R. and Hinkley, D. V. 1974. Theoretical Statistics. Chapman and Hall, London, England.
• 6: Davison, A. C. and Hinkley, D. V. 1997. Bootstrap Methods and Their Application. Cambridge University Press, Cambridge, England.
• 7: Efron, B. 1982. The Jacknife, the Bootstrap and Other Resampling Plans. CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM.
• 8: Efron, B. and Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Chapman & Hall, New York.
• 9: Kleinrock, L. 1976. Queueing Systems : Volume 2. Computer Applications. Wiley, New York.
• 10: L'Ecuyer, P. and Perron, G. 1994. On the convergence rates of IPA and FDC derivative estimators. Oper. Res. 42, 463--656.
• 11: Lütkepohl, H. 1993. Introduction to Multiple Time Series Analysis. Springer-Verlag, Berlin, Germany.
• 12: Rao, C. R. 1973. Linear Statistical Inference and Its Applications. Wiley, New York.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.6 Simulation Output Analysis

General Terms:
Design, Experimentation

Keywords:
δ-Method, parameter variability, simulation variability, two-point method, uncertainty analysis