Verification, validation, and accreditation: verification, validation, and accreditation of simulation models

ABSTRACT

This paper discusses verification, validation, and accreditation of simulation models. The different approaches to deciding model validity are presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; a recommended procedure is presented; and accreditation is briefly discussed.

REFERENCES

• 1: Anderson, H. A. and R. G. Sargent. 1974. An investigation into scheduling for an interactive computer system, IBM Journal of Research and Development, 18 (2):125-137.
• 2: O. Balci, How to assess the acceptability and credibility of simulation results, Proceedings of the 21st conference on Winter simulation, p.62-71, December 04-06, 1989, Washington, D.C., United States [doi>10.1145/76738.76746]
• 3: Osman Balci, Verification, validation, and accreditation, Proceedings of the 30th conference on Winter simulation, p.41-4, December 13-16, 1998, Washington, D.C., United States
• 4: Osman Balci , Robert G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, v.24 n.4, p.190-197, April 1981 [doi>10.1145/358598.358609]
• 5: Balci, O. and R. G. Sargent. 1982a. Validation of multivariate response simulation models by using hotelling's two-sample T2 test, Simulation, 39(6):185-192.
• 6: Osman Balci , Robert G. Sargent, Some examples of simulation model validation using hypothesis testing, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
• 7: Osman Balci , Robert G. Sargent, Validation fo Multivariate Response Trace-driven Simulation Models, Proceedings of the 9th International Symposium on Computer Performance Modelling, Measurement and Evaluation, p.309-323, May 25-27, 1983
• 8: Balci, O. and R. G. Sargent. 1984a. A bibliography on the credibility assessment and validation of simulation and mathematical models, Simuletter, 15(3):15-27.
• 9: Balci, O. and R. G. Sargent. 1984b. Validation of simulation models via simultaneous confidence intervals, American Journal of Mathematical and Management Science, 4(3):375-406.
• 10: Banks, J., J. S. Carson II, and B. L. Nelson. 1996. Discrete-event system simulation, 2nd Ed., Prentice-Hall, Englewood Cliffs, N.J.
• 11: Banks, J., D. Gerstein, and S. P. Searles. 1988. Modeling processes, validation, and verification of complex simulations: A survey, Methodology and Validation, Simulation Series, Vol. 19, No. 1, The Society for Computer Simulation, 13-18.
• 12: DOD simulations: Improved assessment procedures would increase the credibility of results. 1987. U. S. General Accounting Office, PEMD-88-3.
• 13: Fairley, R. E. 1976. Dynamic testing of simulation software, Proc. of the 1976 Summer Computer Simulation Conf., Washington, D.C., 40-46.
• 14: Gass, S. I. 1983. Decision-aiding models: Validation, assessment, and related issues for policy analysis, Operations Research, 31(4):601-663.
• 15: Gass, S. I. 1984. Documenting a computer-based model, Interfaces, 14(3):84-93.
• 16: Gass, S. I. 1993. Model accreditation: A rationale and process for determining a numerical rating, European Journal of Operational Research, 66(2):250-258.
• 17: Gass, S. I. and L. Joel. 1987. Concepts of model confidence, Computers and Operations Research, 8, 4, pp.341-346.
• 18: Gass, S. I. and B. W. Thompson. 1980. Guidelines for model evaluation: An abridged version of the U.S. general accounting office exposure draft, Operations Research, 28(2):431-479.
• 19: Johnson, R. A. 1994. Miller and Freund's probability and statistics for engineers, 5th edition, Prentice-Hall, Englewood Cliffs, N.J.
• 20: Jack P C Kleijnen, Statistical tools for simulation practitioners, Marcel Dekker, Inc., New York, NY, 1986
• 21: Jack P. C. Kleijnen, Validation of models: statistical techniques and data availability, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.647-654, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324450]
• 22: George B. Kleindorfear , Ram Geneshan, The philosophy of science and validation in simulation, Proceedings of the 25th conference on Winter simulation, p.50-57, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256582]
• 23: Peter L. Knepell , Deborah Arangno, Simulation Validation - a Confidence Assessment Methodology, IEEE Computer Society Press, Los Alamitos, CA, 1993
• 24: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 25: Naylor, T. H. and J. M. Finger. 1967. Verification of computer simulation models, Management Science, 14(2):B92-B101.
• 26: Tuncer I. Ören, Concepts and criteria to assess acceptability of simulation studies: a frame of reference, Communications of the ACM, v.24 n.4, p.180-189, April 1981 [doi>10.1145/358598.358605]
• 27: Rao, M. J. and R. G. Sargent. 1988. An advisory system for operational validity, Artificial Intelligence and Simulation: The Diversity of Applications, ed. T. Hensen, Society for Computer Simulation, San Diego, CA, 245-2250.
• 28: Robert G. Sargent, Validation of simulation models, Proceedings of the 11th conference on Winter simulation, p.497-503, December 03-05, 1979, San Diego, CA, United States
• 29: Sargent, R. G. 1981. An assessment procedure and a set of criteria for use in the evaluation of computerized models and computer-based modeling tools, Final Technical Report RADC-TR-80-409.
• 30: Sargent, R. G. 1982. Verification and validation of simulation models, Chapter IX in Progress in Modelling and Simulation, ed. F. E. Cellier, Academic Press, London,159-169.
• 31: Sargent, R. G. 1984. Simulation model validation, Simulation and Model-Based Methodologies: An Integrative View, ed. Oren, et al., Springer-Verlag.
• 32: Robert G. Sargent, An expository on verification and validation of simulation models, Proceedings of the 17th conference on Winter simulation, p.15-22, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253058]
• 33: Robert G. Sargent, The use of graphical models in model validation, Proceedings of the 18th conference on Winter simulation, p.237-241, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318312]
• 34: Robert G. Sargent, A tutorial on validation and verification of simulation models, Proceedings of the 20th conference on Winter simulation, p.33-39, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318150]
• 35: Sargent, R. G. 1990. Validation of mathematical models, Proc. of Geoval-90: Symposium on Validation of Geosphere Flow and Transport Models, Stockholm, Sweden, pp. 571-579.
• 36: Robert G. Sargent, Simulation model verification and validation, Proceedings of the 23rd conference on Winter simulation, p.37-47, December 08-11, 1991, Phoenix, Arizona, United States
• 37: Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 26th conference on Winter simulation, p.77-87, December 11-14, 1994, Orlando, Florida, United States
• 38: Robert G. Sargent, Some subjective validation methods using graphical displays of data, Proceedings of the 28th conference on Winter simulation, p.345-351, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256648]
• 39: Robert G. Sargent, Verifying and validating simulation models, Proceedings of the 28th conference on Winter simulation, p.55-64, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256572]
• 40: Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.39-48, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324148]
• 41: Schlesinger, et al. 1979. Terminology for model credibility, Simulation, 32(3):103-104.
• 42: Schruben, L. W. 1980. Establishing the credibility of simulations, Simulation, 34(3):101-105.
• 43: Shannon, R. E. 1975. Systems simulation: The art and the science, Prentice-Hall.
• 44: R. B. Whitner , O. Balci, Guidelines for selecting and using simulation model verification techniques, Proceedings of the 21st conference on Winter simulation, p.559-568, December 04-06, 1989, Washington, D.C., United States [doi>10.1145/76738.76811]
• 45: David O. Wood, MIT model analysis program: what we have learned about policy model review, Proceedings of the 18th conference on Winter simulation, p.248-252, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318314]
• 46: Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984