Verifying and validating simulation models

ABSTRACT

This paper discusses verification and validation of simulation models. The different approaches to deciding model validity am presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined, conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; and a recommended procedure is presented.

REFERENCES

• 1: Anderson, H.A. and R.G. Sargent. 1974. An Investigation into Scheduling for an Interactive Computer System, IBM Journal of Research and Development, 18, 2, pp. 125-137.
• 2: O. Balci, How to assess the acceptability and credibility of simulation results, Proceedings of the 21st conference on Winter simulation, p.62-71, December 04-06, 1989, Washington, D.C., United States [doi>10.1145/76738.76746]
• 3: Osman Balci , Robert G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, v.24 n.4, p.190-197, April 1981 [doi>10.1145/358598.358609]
• 4: Balci, O. and R.G. Sargent. 1982a. Validation of Multivariate Response Simulation Models by Using HoteUing's Two-Sample T2 Test, Simulation, Vol. 39, No. 6, pp. 185-192.
• 5: Osman Balci , Robert G. Sargent, Some examples of simulation model validation using hypothesis testing, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
• 6: Osman Balci , Robert G. Sargent, Validation fo Multivariate Response Trace-driven Simulation Models, Proceedings of the 9th International Symposium on Computer Performance Modelling, Measurement and Evaluation, p.309-323, May 25-27, 1983
• 7: Balci, O. and R.G. Sargent. 1984a. A Bibliography on the Credibility Assessment and Validation of Simulation and Mathematical Models, Simuletter, 15, 3, pp. 15-27.
• 8: Balci, O. and R.G. Sargent. 1984b. Validation of Simulation Models via Simultaneous Confidence Intervals, American Journal of Mathematical and Management Science, pp. 375-406.
• 9: Banks, J., J.S. Carson II, and B.L. Nelson. 1996. Discrete-Event System Simulation, 2nd Edition Prentice-Hall, Englewood Cliffs, NJ.
• 10: Banks, J., D. Gerstein, and S.P. Searles. 1988. Modeling Processes, Validation, and Verification of Complex Simulations: A Survey, Methodology and Validation, Simulation Series, Vol. 19, No. 1, The Society for Computer Simulation, pp. 13-18.
• 11: Chattergy, R. and V.W. Pooch. 1977. Integrated Design and Verification of Simulation Programs, Computer, Vol. 10, No. 4, pp. 40-46.
• 12: Edward A. Davis, Use of seminar gaming to specify and validate simulation models, Proceedings of the 18th conference on Winter simulation, p.242-247, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318313]
• 13: DOD Simulations: Improved Assessment Procedures Would Increase the Credibility of Results 1987. United States General Accounting Office, PEMD-88-3.
• 14: Fairley, R.E. 1976. Dynamic Testing of Simulation Software, Proceedings of the 1976 Summer Computer Simulation Conference, Washington, D.C., pp. 40-46.
• 15: Gass, S.I. 1977. A Procedure for Evaluation of Complex Models, Proceedings of the First International Conference on Mathematical Modeling, University of Missouri, pp. 247-257.
• 16: Gass, S.I. 1983. Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis, Operations Research, 31, 4, pp. 601-663.
• 17: Gass, S.I. 1984. Documenting a Computer-Based Model, Interfaces, Vol. 14, No. 3, pp. 84-93.
• 18: Gass, S.I. and L. Joel. 1987. Concepts of Model Confidence, Computers and Operations Research, 8,4, pp. 341-346.
• 19: Gass, S.I. and B.W. Thompson. 1980. Guidelines for Model Evaluation: An Abridged Version of the U.S. General Accounting Office Exposure Draft, Operations Research, 28, 2, pp. 431-479.
• 20: Johnson, R.A. 1994. Miller and Freund's Probability and Statistics for Engineers, 5th edition, Prentice-Hall, Englewood Cliffs, N.J.
• 21: Jack P C Kleijnen, Statistical tools for simulation practitioners, Marcel Dekker, Inc., New York, NY, 1986
• 22: George B. Kleindorfear , Ram Geneshan, The philosophy of science and validation in simulation, Proceedings of the 25th conference on Winter simulation, p.50-57, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256582]
• 23: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 24: Naylor, T.H. and J.M. Finger. 1967. Verification of Computer Simulation Models, Management Science, 14, 2, pp. B92-B101.
• 25: Tuncer I. Ören, Concepts and criteria to assess acceptability of simulation studies: a frame of reference, Communications of the ACM, v.24 n.4, p.180-189, April 1981 [doi>10.1145/358598.358605]
• 26: Rao, M.J. and R.G. Sargent. 1988. An Advisory System for Operational Validity, Artificial Intelligence and Simulation: The Diversity of Applications, edited by T. Hensen, Society for Computer Simulation, San Diego, CA, pp. 245-250.
• 27: Robert G. Sargent, Validation of simulation models, Proceedings of the 11th conference on Winter simulation, p.497-503, December 03-05, 1979, San Diego, CA, United States
• 28: Sargent, R.G. 198 i. An Assessment Procedure and a Set of Criteria for Use in the Evaluation of Computerized Models and Computer-Based Modelling Tools, Final Technical Report RADC-TR-80-409.
• 29: Sargent, R.G. 1982. Verification and Validation of Simulation Models, Chapter IX in Progress in Modelling and Simulation, edited by F.E. Cellier, Academic Press, London, pp. 159-169.
• 30: Sargent, R.G. 1984. Simulation Model Validation, Simulation and Model-Based Methodologies: An Integrative View, edited by Oren, et al., Springer- Verlag.
• 31: Robert G. Sargent, An expository on verification and validation of simulation models, Proceedings of the 17th conference on Winter simulation, p.15-22, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253058]
• 32: Robert G. Sargent, The use of graphical models in model validation, Proceedings of the 18th conference on Winter simulation, p.237-241, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318312]
• 33: Robert G. Sargent, A tutorial on validation and verification of simulation models, Proceedings of the 20th conference on Winter simulation, p.33-39, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318150]
• 34: Sargent, R.G. 1990. Validation of Mathematical Models, Proceedings of GEOVAL-90: Symposium on Validation of Geosphere Flow and Transport Models, Stockholm, Sweden, pp. 571-579.
• 35: Robert G. Sargent, Simulation model verification and validation, Proceedings of the 23rd conference on Winter simulation, p.37-47, December 08-11, 1991, Phoenix, Arizona, United States
• 36: Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 24th conference on Winter simulation, p.104-114, December 13-16, 1992, Arlington, Virginia, United States [doi>10.1145/167293.167311]
• 37: Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 26th conference on Winter simulation, p.77-87, December 11-14, 1994, Orlando, Florida, United States
• 38: Robert G. Sargent, Some subjective validation methods using graphical displays of data, Proceedings of the 28th conference on Winter simulation, p.345-351, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256648]
• 39: Schlesinger, et al. 1979. Terminology for Model Credibility, Simulation, 32, 3, pp. 103-104.
• 40: Schruben, L.W. 1980. Establishing the Credibility of Simulations, Simulation, 34, 3, pp. 101-105.
• 41: Shannon, R.E. 1975. Systems Simulation: The Art and the Science, Prentice-Hall.
• 42: Robert E. Shannon, Tests for the verification and validation of computer simulation models, Proceedings of the 13th conference on Winter simulation, p.573-577, December 09-11, 1981, Atlanta, Georgia, United States
• 43: Smith, D.E. and C.A. Mauro. 1982. Factor Screening in Computer Simulation, Simulation, 38, 2, pp. 49- 54.
• 44: R. B. Whitner , O. Balci, Guidelines for selecting and using simulation model verification techniques, Proceedings of the 21st conference on Winter simulation, p.559-568, December 04-06, 1989, Washington, D.C., United States [doi>10.1145/76738.76811]
• 45: David O. Wood, MIT model analysis program: what we have learned about policy model review, Proceedings of the 18th conference on Winter simulation, p.248-252, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318314]
• 46: Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984