A tutorial on validation and verification of simulation models

ABSTRACT

We give a general introduction to validation and verification of simulation models, define the various validation techniques, and present a recommended model validation procedure in this tutorial paper.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Anderson, H.A. and R.G. Sargent (1974). An Investigation into Scheduling for an Interactive Computer System, IBM Journal of Research and Development, 18, 2, pp. 125-137.
	2	Bogdan Czejdo , Marek Rusinkiewicz, Query generation in an instructional database management system, ACM SIGCSE Bulletin, v.19 n.1, p.19-25, Feb. 1987
	3	Osman Balci , Robert G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, v.24 n.4, p.190-197, April 1981 [doi>10.1145/358598.358609]
	4	Balci, O. and R.G. Sargent (1982a). Validation of Multivariate Response Simulation Models by Using Hotelling's Two-Sample T2 Test, Simulation, Vol. 39, No. 6, pp. 185-192.
	5	Osman Balci , Robert G. Sargent, Some examples of simulation model validation using hypothesis testing, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
	6	Osman Balci , Robert G. Sargent, Validation fo Multivariate Response Trace-driven Simulation Models, Proceedings of the 9th International Symposium on Computer Performance Modelling, Measurement and Evaluation, p.309-323, May 25-27, 1983
	7	Balci, O. and R.G. Sargent (1984a). A Bibliography on the Credibility, Assessment and Validation of Simulation and Mathematical Models, Simuletter, Vol. 15, No. 3, pp. 15-2"7.
	8	Balci, O. and R.G. Sargent (1984b). Validation of Simulation Models via Simultaneous Confidence Intervals, American Journal of Mathematical and Management Science, Vol. 4, No. 3 and 4, pp. 375- 406.
	9	Banks, J. and J.S. Carson Ii (1984). Discrete-Event System Simulation, Prentice-Hall, New Jersey.
	10	Banks, J., D. Gerstein, and S.P. Sear/es (1987). Modeling Processes, Validation, and Verification of Complex Simulations: A Survey, Methodology and Validation, Simulation Series, Vol. 19, No. 1. The Society for Computer Simulation, pp. 13-18.
	11	Chattergy, R. and V.W. Pooch (1977). Integrated Design and Verification of Simulation Programs, Computer, Vol. 10, No. 4, pp. 40-46.
	12	B. C. Chen , R. G. Sargent, Using standardized time series to estimate confidence intervals for the difference between two stationary stochastic processes, Operations Research, v.35 n.3, p.428-436, May-June 1987 [doi>10.1287/opre.35.3.428]
	13	Edward A. Davis, Use of seminar gaming to specify and validate simulation models, Proceedings of the 18th conference on Winter simulation, p.242-247, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318313]
	14	DOD Simulations: Improved Assessment Procedures Would Increase the Credibility of Results (1987). United States General Accounting Office, PEMD-88-3.
	15	Fairley, R.E. (1976). Dynamic Testing of Simulation Software, Proceedings of the 1976 Summer Computer Simulation Conference, Washington, D.C., pp. 40-46.
	16	Gass, S,I. (1983). Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis, Operations Research, 3I, 4, pp. 601-663.
	17	Gass, S.I. (1984). Documenting a Computer-Based Model, Inte,faces, Vol. 14, No. 3, pp. 84-93.
	18	Gass, S.I. and B.W. Thompson (1980). Guidelines for Model Evaluation: An Abridged Version of the U.SI General Accounting Office Exposure Draft, Operations Research, 28, 2, pp. 431-479.
	19	Kleijnen, J.P.C. (1975). Statistical Techniques in Simulation, Part II, Marcel Dekker, Nev. k'ork.
	20	Kleijnen, J.P.C. (1982). Experimentation with Models: Statistical Design and Analysis Techniques, Progress in Modelling and Simulation, F.E. Cellier, Editor, Academic Press, London.
	21	Law, A.M. (I983). Statistical Analysis of Simulation Output Data, Operations Research, Vol. 31, No. 6, pp. 983-1029.
	22	Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
	23	Miller, i. and J.E. Feund (1985). Probability and Statistics for Engineers, Prentice-Hall, Englewood Cliffs, New Jersey.
	24	Naylor, T.H:. and J.M. Finger (1967). Verification of Computer Simulation Models, Management Science, 14, 2, pp. B92-B101.
	25	Tuncer I. Ören, Concepts and criteria to assess acceptability of simulation studies: a frame of reference, Communications of the ACM, v.24 n.4, p.180-189, April 1981 [doi>10.1145/358598.358605]
	26	Rao, M.J. and R.G. Sargent (1988). An Advisory System for Operational Validity, Artificial Intelligence and Simulation: The Diversity of Applications (Proceedings of the 1988 Multic~nference)' edited by T. Hensen, San Diego, California, pp. 245-250.
	27	Robert G. Sargent, Validation of simulation models, Proceedings of the 11th conference on Winter simulation, p.497-503, December 03-05, 1979, San Diego, CA, United States
	28	Sargent, R.G. (1981). An Assessment Procedure and a Set of Criteria for Use in the Evaluation of Computerized Models and Computer-Based Modelling Tools, Final Technical Report RADC-TR-80-409.
	29	Sargent, R.G. (1982), Verification and Validation of Simulation Models, Chapter IX in Progress in Modelling and Simulation, edited by F.E. CeUier, Academic Press, London, pp. 159-169.
	30	Sargent, R.G. (1984). Simulation Model Yalidation, Simulat}pn and Model-Based Methodologies: An Integrative View, edited by Oren, et al., Springer-Verlag.
	31	Robert G. Sargent, An expository on verification and validation of simulation models, Proceedings of the 17th conference on Winter simulation, p.15-22, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253058]
	32	Robert G. Sargent, The use of graphical models in model validation, Proceedings of the 18th conference on Winter simulation, p.237-241, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318312]
	33	Robert G. Sargent, An overview of verification and validation of simulation models, Proceedings of the 19th conference on Winter simulation, p.33-39, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318379]
	34	Schlesinger, et aI. (1979). Terminology for Model Credibility, Simulation, 32, 3, pp. 103-104.
	35	Schruben, L.W. (1980). Establishing the Credibility of Simulations, Simulation, 34, 3, pp. 101-105.
	36	Shannon, R.E. (1975). Systems Simulation: The Art and the Science, Prentice-Hall.
	37	Robert E. Shannon, Tests for the verification and validation of computer simulation models, Proceedings of the 13th conference on Winter simulation, p.573-577, December 09-11, 1981, Atlanta, Georgia, United States
	38	Smith, D.E. and C.A. Mauro (1982). Factor Screening in Computer Simulation, Simulation, VoI. 38, No. 2, pp. 49-54.
	39	David O. Wood, MIT model analysis program: what we have learned about policy model review, Proceedings of the 18th conference on Winter simulation, p.248-252, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318314]
	40	Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984