Output modeling: abc's of output analysis

ABSTRACT

We present a brief overview of several of the basic output analysis techniques for evaluating stochastic dynamic simulations. This tutorial is intended for those with little previous exposure to the topic, for those in need of a refresher course, and especially for those who have never heard of output analysis. We discuss the reasons why simulation output analysis differs from that taught in basic statistics courses and point out how to avoid common pitfalls that may lead to erroneous results and faulty conclusions.

REFERENCES

• 1: Alexopoulos, C. and A. F. Seila. 1998. Output data analysis. Chapter 7 in Handbook of Simulation, ed. J. Banks. New York: John Wiley and Sons.
• 2: Banks, J., J. S. Carson, B. L. Nelson and D. Nichol. 2000. Discrete-event system simulation, 3d ed. Upper Saddle River, New Jersey: Prentice-Hall.
• 3: Russell R. Barton, Simulation metamodels, Proceedings of the 30th conference on Winter simulation, p.167-176, December 13-16, 1998, Washington, D.C., United States
• 4: Paul Bratley , Bennett L. Fox , Linus E. Schrage, A guide to simulation (2nd ed.), Springer-Verlag New York, Inc., New York, NY, 1987
• 5: John M. Charnes, Analyzing multivariate output, Proceedings of the 27th conference on Winter simulation, p.201-208, December 03-06, 1995, Arlington, Virginia, United States [doi>10.1145/224401.224463]
• 6: Russel C. H. Cheng, Bayesian model selection when the number of components is unknown, Proceedings of the 30th conference on Winter simulation, p.653-660, December 13-16, 1998, Washington, D.C., United States
• 7: Russell C. H. Cheng , John D. Lamb, Interactive implementation of optimal simulation experiment designs, Proceedings of the 30th conference on Winter simulation, p.707-712, December 13-16, 1998, Washington, D.C., United States
• 8: Stephen E. Chick, Bayesian methods: bayesian methods for simulation, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 9: George S. Fishman, Principles of Discrete Event Simulation, John Wiley & Sons, Inc., New York, NY, 1978
• 10: Michael C. Fu, A tutorial review of techniques for simulation optimization, Proceedings of the 26th conference on Winter simulation, p.149-156, December 11-14, 1994, Orlando, Florida, United States
• 11: Fu, M. C. and J-Q Hu. 1997. Conditional Monte Carlo, Gradient Estimation and Optimization Applications. Boston, Massachusetts: Kluwer Academic Publishers.
• 12: David Goldsman , Barry L. Nelson, Statistical screening, selection, and multiple comparison procedures in computer simulation, Proceedings of the 30th conference on Winter simulation, p.159-166, December 13-16, 1998, Washington, D.C., United States
• 13: David Alan Grier, Graphical techniques for output analysis, Proceedings of the 24th conference on Winter simulation, p.314-319, December 13-16, 1992, Arlington, Virginia, United States [doi>10.1145/167293.167356]
• 14: Sarah J. Hood , Peter D. Welch, Response surface methodology and its application in simulation, Proceedings of the 25th conference on Winter simulation, p.115-122, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256598]
• 15: W. David Kelton, Statistical analysis of simulation output, Proceedings of the 29th conference on Winter simulation, p.23-30, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268443]
• 16: Khoshnevis, B. 1994. Discrete systems simulation. New York: McGraw-Hill.
• 17: Jack P C Kleijnen, Statistical tools for simulation practitioners, Marcel Dekker, Inc., New York, NY, 1986
• 18: Kleijnen, J. P. C. 1998. Experimental design for sensitivity analysis, optimization, and validation of simulation models. Handbook of simulation, ed. J. Banks. New York: John Wiley and Sons.
• 19: Pierre L'Ecuyer, Efficiency improvement and variance reduction, Proceedings of the 26th conference on Winter simulation, p.122-132, December 11-14, 1994, Orlando, Florida, United States
• 20: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 21: P. A. W. Lewis , E. J. Orav, Simulation methodology for statisticians, operations analysts, and engineers: vol. 1, Wadsworth Publ. Co., Belmont, CA, 1989
• 22: Matejcik, F. J. and B. L. Nelson. 1995. Two-stage multiple comparisons with the best for computer simulation. Operations Research 43(4):633-640.
• 23: Douglas J. Morrice , Lee W. Schruben, Simulation factor screening using harmonic analysis, Management Science, v.39 n.12, p.1459-1476, Dec. 1993 [doi>10.1287/mnsc.39.12.1459]
• 24: Nakayama, M. 1997. Multiple comparison procedures for steady-state simulations. Annals of Statistics 25: 2433-2450.
• 25: Barry L. Nelson, Designing efficient simulation experiments, Proceedings of the 24th conference on Winter simulation, p.126-132, December 13-16, 1992, Arlington, Virginia, United States [doi>10.1145/167293.167317]
• 26: Nelson, B. L. 1995. Stochastic modeling: analysis and simulation. New York: McGraw-Hill.
• 27: Brian D. Ripley, Stochastic simulation, John Wiley & Sons, Inc., New York, NY, 1987
• 28: A. Saltelli , S. Tarantola , K. P.-S. Chan, A quantitative model-independent method for global sensitivity analysis of model output, Technometrics, v.41 n.1, p.39-56, Feb. 1999 [doi>10.2307/1270993]
• 29: Susan M. Sanchez, ABC's of output analysis, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.24-32, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324144]
• 30: Susan M. Sanchez, Design of experiments: robust design: seeking the best of all possible worlds, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 31: Sanchez, S. M., P. J. Sanchez, J. S. Ramberg and F. Moeeni. 1996. Effective engineering design through simulation?" International Transactions on Operational Research 3 (2): 169-185.
• 32: Sanchez, S. M., P. J. Sanchez and J. S. Ramberg. 1998. "A simulation framework for robust system design." Chapter 12 in Concurrent Design of Products, Manufacturing Processes and Systems, ed. B. Wang, 279-314. New York: Gordon and Breach.
• 33: Lee W. Schruben , V. James Cogliano, An experimental procedure for simulation response surface model identification, Communications of the ACM, v.30 n.8, p.716-730, Aug. 1987 [doi>10.1145/27651.27656]
• 34: Tew, J. D. and Wilson, J. R. 1994. Estimating Simulation Metamodels Using Combined Correlation-Based Reduction Techniques. IIE Transactions, 26 (3): 2-16.
• 35: Arne Thesen , Laurel E. Travis , Robert J. Gordon, Simulation for Decision Making, West Publishing Co., St. Paul, MN, 1992

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.6 Simulation Output Analysis

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Monte Carlo

General Terms:
Algorithms, Performance, Verification