Verification and validation: verification and validation of simulation models

ABSTRACT

In this paper we discuss verification and validation of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate verification and validation to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.

REFERENCES

• 1: Anderson, H. A. and R. G. Sargent. 1974. An investigation into scheduling for an interactive computer system. IBM Journal of Research and Development, 18 (2): 125--137.
• 2: O. Balci, How to assess the acceptability and credibility of simulation results, Proceedings of the 21st conference on Winter simulation, p.62-71, December 04-06, 1989, Washington, D.C., United States [doi>10.1145/76738.76746]
• 3: Osman Balci, Verification, validation, and accreditation, Proceedings of the 30th conference on Winter simulation, p.41-4, December 13-16, 1998, Washington, D.C., United States
• 4: Osman Balci , Robert G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, v.24 n.4, p.190-197, April 1981 [doi>10.1145/358598.358609]
• 5: Balci, O. and R. G. Sargent. 1982a. Validation of multivariate response simulation models by using Hotelling's two-sample T2 test. Simulation 39(6):185--192.
• 6: Osman Balci , Robert G. Sargent, Some examples of simulation model validation using hypothesis testing, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
• 7: Osman Balci , Robert G. Sargent, Validation fo Multivariate Response Trace-driven Simulation Models, Proceedings of the 9th International Symposium on Computer Performance Modelling, Measurement and Evaluation, p.309-323, May 25-27, 1983
• 8: Balci, O. and R. G. Sargent. 1984a. A bibliography on the credibility assessment and validation of simulation and mathematical models. Simuletter 15(3):15--27.
• 9: Balci, O. and R. G. Sargent. 1984b. Validation of simulation models via simultaneous confidence intervals. American Journal of Mathematical and Management Science 4(3):375--406.
• 10: Banks, J., J. S. Carson II, B. L. Nelson, and D. Nicol. 2000. Discrete-event system simulaton. 3d. Ed. Englewood Cliffs, NJ: Prentice-Hall.
• 11: Banks, J., D. Gerstein, and S. P. Searles. 1988. Modeling Processes, Validation, and Verification of Complex Simulations: A Survey. In Methodology and Validation, Simulation Series, Vol. 19, No. 1, The Society for Computer Simulation, 13--18. San Diego, CA: Society for Modeling and Simulation International.
• 12: John S. Carson, II, Verification validation: model verification and validation, Proceedings of the 34th conference on Winter simulation: exploring new frontiers, December 08-11, 2002, San Diego, California
• 13: DOD simulations: improved assessment procedures would increase the credibility of results. 1987. U. S. General Accounting Office, PEMD-88--3.
• 14: Fairley, R. E. 1976. Dynamic testing of simulation software. In Proc. 1976 Summer Computer Simulation Conf., 40--46, Washington, D.C.
• 15: Gass, S. I. 1983. Decision-aiding models: validation, assessment, and related issues for policy analysis. Operations Research 31(4):601--663.
• 16: Gass, S. I. 1984. Documenting a computer-based model. Interfaces, 14(3):84--93.
• 17: Gass, S. I. 1993. Model Accreditation: A Rationale and Process for Determining a Numerical Rating. European Journal of Operational Research, 66 (2): 250--258.
• 18: Gass, S. I. and L. Joel. 1987. Concepts of Model Confidence. Computers and Operations Research, 8 (4): 341--346.
• 19: Gass, S. I. And B. W. Thompson. 1980. Guidelines for model evaluation: An abridged version of the U. S. general accounting office exposure draft. Operations Research 28(2):431--479.
• 20: James Lee Johnson, Probability and Statistics for Computer Science, John Wiley & Sons, Inc., New York, NY, 2003
• 21: Jack P C Kleijnen, Statistical tools for simulation practitioners, Marcel Dekker, Inc., New York, NY, 1986
• 22: Jack P. C. Kleijnen, Validation of models: statistical techniques and data availability, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.647-654, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324450]
• 23: George B. Kleindorfear , Ram Geneshan, The philosophy of science and validation in simulation, Proceedings of the 25th conference on Winter simulation, p.50-57, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256582]
• 24: Peter L. Knepell , Deborah Arangno, Simulation Validation - a Confidence Assessment Methodology, IEEE Computer Society Press, Los Alamitos, CA, 1993
• 25: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 26: Averill M. Law , Michael G. McComas, Building valid models: how to build valid and credible simulation models, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 27: Julie C. Lowery, Design of hospital admissions scheduling system using simulation, Proceedings of the 28th conference on Winter simulation, p.1199-1204, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256932]
• 28: Nayor, T. H. and J. M. Finger. 1967. Verification of computer simulation models. Management Science 14(2):B92-B101.
• 29: Tuncer I. Ören, Concepts and criteria to assess acceptability of simulation studies: a frame of reference, Communications of the ACM, v.24 n.4, p.180-189, April 1981 [doi>10.1145/358598.358605]
• 30: Rao, M. J. and R. G. Sargent. 1988. An advisory system for operational validity. In Artificial Intelligence and Simulation: the Diversity of Applications, ed. T. Hensen, 245--250. San Diego, CA: Society for Computer Simulation.
• 31: Robert G. Sargent, Validation of simulation models, Proceedings of the 11th conference on Winter simulation, p.497-503, December 03-05, 1979, San Diego, CA, United States
• 32: Sargent, R. G. 1981. An assessment procedure and a set of criteria for use in the evaluation of computerized models and computer-based modeling tools. Final Technical Report RADC-TR-80--409.
• 33: Sargent, R. G. 1982. Verification and Validation of Simulation Models. Chapter IX in Progress in Modelling and Simulation, ed. F. E. Cellier, 159--169. London: Academic Press.
• 34: Sargent, R. G. 1984. Simulation Model Validation. Chapter 19 in Simulation and Model-Based Methodologies: An Integrative View, ed. T. I. Oren, B. P. Zeigler, and M. S. Elzas, 537--555. Heidelberg, Germany: Springer-Verlag.
• 35: Robert G. Sargent, An expository on verification and validation of simulation models, Proceedings of the 17th conference on Winter simulation, p.15-22, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253058]
• 36: Robert G. Sargent, The use of graphical models in model validation, Proceedings of the 18th conference on Winter simulation, p.237-241, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318312]
• 37: Sargent, R. G. 1990. Validation of mathematical models. In Proc. of Geoval-90: Symposium on Validation of Geosphere Flow and Transport Models, 571--579. Stockholm, Sweden.
• 38: Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 26th conference on Winter simulation, p.77-87, December 11-14, 1994, Orlando, Florida, United States
• 39: Robert G. Sargent, Some subjective validation methods using graphical displays of data, Proceedings of the 28th conference on Winter simulation, p.345-351, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256648]
• 40: Robert G. Sargent, Verifying and validating simulation models, Proceedings of the 28th conference on Winter simulation, p.55-64, December 08-11, 1996, Coronado, California, United States [doi>10.1145/256562.256572]
• 41: Robert G. Sargent, Verification, validation, and accreditation: verification, validation, and accreditation of simulation models, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 42: Sargent, R. G. 2001a. Graphical displays of simulation model data as statistical references. In Simulation 2001 (Proc. of the 4th St. Petersburg Workshop on Simulation), ed. S. M. Ermakor, Yu. N. Kashtanov, and V. B. Melas, 109--118. Publisher: Chemistry Research Institute of St. Petersburg University.
• 43: Robert G. Sargent, Verification and validation: some approaches and paradigms for verifying and validating simulation models, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 44: Schlesinger, et al. 1979. Terminology for model credibility. Simulation, 32(3):103--104.
• 45: Schruben, L. W. 1980. Establishing the credibility of simulation models. Simulation 34(3):101--105.
• 46: R. B. Whitner , O. Balci, Guidelines for selecting and using simulation model verification techniques, Proceedings of the 21st conference on Winter simulation, p.559-568, December 04-06, 1989, Washington, D.C., United States [doi>10.1145/76738.76811]
• 47: David O. Wood, MIT model analysis program: what we have learned about policy model review, Proceedings of the 18th conference on Winter simulation, p.248-252, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318314]
• 48: Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984