Five-stage procedure for the evaluation of simulation models through statistical techniques

ABSTRACT

This paper recommends the following sequence for the evaluation of simulation models. 1) Validation: the availability of data on the real system determines the proper type of statistical technique. 2) Screening: in the simulation's pilot phase the important inputs are identified through a novel technique, namely sequential bifurcation, which uses aggregation and sequential experimentation. 3) Sensitivity or what-if analysis: the important inputs are analyzed in more detail, including interactions between inputs; relevant techniques are design of experiments (DOE) and regression analysis. 4) Uncertainty or risk analysis: important environmental inputs may have values not precisely known, so the resulting uncertainties in the model outputs are quantified; techniques are Monte Carlo and Latin hypercube sampling. 5) Optimization: policy variables may be controlled, applying Response Surface Methodology (RSM), which combines DOE, regression analysis, and steepest-ascent hill-climbing. This paper summarizes case studies for each stage.

REFERENCES

• 1: Andres, T.H. (1996), Sampling methods and sensitivity analysis for large parameter sets. Journal Statistical Computation and Simulation (accepted).
• 2: Avramidis, A.N. and J.R. Wilson (1996), Integrated Variance Reduction Strategies for Simulation. Operations Research 44: 327-346.
• 3: Bettonvil, B. and J.P.C. Kleijnen, (1996), Searching for important factors in simulation models with many factors: sequential bifurcation. European Journal of Operational Research (accepted).
• 4: Cheng, R.C.H. and W. Holland (1996), The sensitivity of computer simulation experiments to errors in input data. Journal Statistical Computation and Simulation (accepted).
• 5: Russell C. H. Cheng , Jack P. C. Kleijnen, Improved Design of Queueing Simulation Experiments with Highly Heteroscedastic Responses, Operations Research, v.47 n.5, p.762-777, May 1999 [doi>10.1287/opre.47.5.762]
• 6: Cooke, R.M. (1995), UNICORN: methods and code for uncertainty analysis. The SRD Association, AEA Technology, Thomson House, Risley, Warrington WA3 6AT, UK.
• 7: De Wit, M.S. (1995), Uncertainty analysis in building thermal modelling. SAM095, Office for Official Publications of the European Communities, Luxembourg.
• 8: Draper, D. (1995), Assessment and propagation of model uncertainty, Journal Royal Statistical Society. Series B. 57: 45-97.
• 9: Eschenbach, T.G. (1992), Spider plots versus tornado diagrams for sensitivity analysis. Inter?laces, 22: 40-46.
• 10: Ffrbringer, J.-M. and C.A. Roulet (1996), Comparison and combination of factorial and Monte-Carlo design in sensitivity analysis. Building and Environment, 30: 505-519.
• 11: Haverkort, B.R. and A.M.H. Meeuwissen (1995), Sensitivity and uncertainty analysis of Markovreward models. IEEE Transactions on Reliability, 44: 147-154.
• 12: Helton, J.C. (1996), Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. Journal Statistical Computation and Simulation (accepted).
• 13: Helton, J.C. et al. (1995), Effect of alternative conceptual models in a preliminary performance assessment for the waste isolation pilot plant. Nuclear Engineering and Design" 251-344.
• 14: Ho,Y. and X. Cao. 1991. Perturbation analysis of discrete event systems. Dordrecht: Kluwer.
• 15: Jack P. C. Kleijnen, Risk analysis and sensitivity analysis: antithesis or synthesis?, ACM SIGSIM Simulation Digest, v.14 n.1-4, p.64-72, 1983 [doi>10.1145/1102863.1102865]
• 16: Jack P C Kleijnen, Statistical tools for simulation practitioners, Marcel Dekker, Inc., New York, NY, 1986
• 17: Kleijnen, J.P.C. (1994), Sensitivity analysis versus uncertainty analysis: when to use what'? Predictability and Nonlinear Modelling in Natural Sciences and Economics, edited by J. Grasman and G. van Straten, Kluwer, Dordrecht, The Netherlands, 1994, pp.322-333. (Preprinted in Kwantitatieve Methoden, 15, 1994: 3-15.)
• 18: Kleijnen, J.P.C. (1995a), Verification and validation of simulation models. European Journal of Operational Research 82:145-162.
• 19: Jack P. C. Kleijnen, Sensitivity analysis and optimization in simulation: design of experiments and case studies, Proceedings of the 27th conference on Winter simulation, p.133-140, December 03-06, 1995, Arlington, Virginia, United States [doi>10.1145/224401.224454]
• 20: Kleijnen, J.P.C. (1995c), Case-study" statistical validation of simulation models. European Journal of Operational Research, 87:21-34.
• 21: Kleijnen, J.P.C. (1995d), Sensitivity analysis and optimization of system dynamics models" regression analysis and statistical design of experiments. System Dynamics Review, 11" 1-14.
• 22: Kleijnen, J.P.C. (1996), Sensitivity analysis and related analyses: a survey of statistical techniques. Journal Statistical Computation and Simulation (accepted)
• 23: Kleijnen, J.P.C. and R.Y. Rubinstein (1996) Optimization and sensitivity analysis of computer simulation models by the score function method. European Journal of Operational Research, 88: 1-15.
• 24: Kleijnen, J.P.C., B. Bettonvil, and W. Van Groenendaal (1996), Validation of simulation models: a novel regression test. Management Science (accepted).
• 25: Jack P. C. Kleijnen , Willen van Groenendaal, Simulation: a statistical perspective, John Wiley & Sons, Inc., New York, NY, 1992
• 26: Kleijnen, J.P.C. and R. Sargent (1996), Metamodeling methodology. Tilburg University, Tilburg, Netherlands.
• 27: Michael D. McKay, Latin hypercube sampling as a tool in uncertainty analysis of computer models, Proceedings of the 24th conference on Winter simulation, p.557-564, December 13-16, 1992, Arlington, Virginia, United States [doi>10.1145/167293.167637]
• 28: McKay, M.D. (1995), Evaluating prediction uncertainty. Los Alamos National Laboratory, NUREG/CR- 6311 (LA-12915-MS).
• 29: Myers, R.H. and D.C. Montgomery (1995), Response Surface Methodology.
• 30: Max D. Morris, Factorial sampling plans for preliminary computational experiments, Technometrics, v.33 n.2, p.161-174, May 1991 [doi>10.2307/1269043]
• 31: A. Saltelli , T. H. Andres , T. Homma, Sensitivity analysis of model output: performance of the iterated fractional factorial design method, Computational Statistics & Data Analysis, v.20 n.4, p.387-407, Oct. 1995 [doi>10.1016/0167-9473(95)92843-M]
• 32: Saltelli, A. and I.M. Sobol (1995), About the use of rank transformation in sensitivity analysis of model output. Reliability Engineering and System Safety, 50: 225-239.
• 33: Sobol, I.M. (1996), Sensitivity analysis of nonlinear models using sensitivity indices. Journal Statistical Computation and Simulation (accepted).
• 34: Winkler, R.L. (1996), Uncertainty in probabilistic risk assessment. Reliability Engineering and System Safety, Special Issue, edited by J.Helton and D. Burmaster.
• 35: Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984