How to assess the acceptability and credibility of simulation results

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

How to assess the acceptability and credibility of simulation results

Full text

Pdf (1.14 MB)

Source

Winter Simulation Conference archive
Proceedings of the 21st conference on Winter simulation table of contents

Washington, D.C., United States

Pages: 62 - 71

Year of Publication: 1989

ISBN:0-911801-58-8

Author

O. Balci

Sponsors

IIE : Institute of Industrial Engineers
NIST : National Institue of Standards & Technology
SES : SES
TIMS/CS :
IEEE-CS : Computer Society
ORSA : Operations Research Society of America
SIGSIM: ACM Special Interest Group on Simulation and Modeling

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 14, Downloads (12 Months): 64, Citation Count: 28

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/76738.76746 What is a DOI?

ABSTRACT

The purpose of this paper is to present a comprehensive life cycle of a simulation study and guide the simulationist in conducting 10 processes, 10 phases, and 13 credibility assessment stages of the life cycle. The guidelines assist the simulation practitioners in: formulating the problem; investigating solution techniques and the system under study; formulating, representing, and programming the simulation model; designing experiments; experimenting; redefining the model; and presenting the simulation results. The guidelines also assist the practitioners in: formulated problem verification, feasibility assessment of simulation, system and objectives definition verification, model qualification, communicative model verification, programmed model verification, experiment design verification, data validation, model validation, and presentation verification. The practitioners can follow the guidelines presented herein and significantly increase their chance of being successful in conducting a simulation study.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Aigner, D. J. (1972). A note on verification of computer simulation models. Management Science 18, 11 (Nov.), 615--619.
	2	Annino, J. S. and Russell, E. C. (1979). The ten most frequent causes of simulation analysis failure - and how to avoid them!. Simulation 32, 6 (June), 137--140.
	3	Balci, O. (1986). Requirements for model development environments. Computers & Operations Research 13, 1 (Jan.-Feb.), 53--67.
	4	Balci, O. (1987a). Guidelines for successful simulation studies. Technical Report TR-85-2, Department of Computer Science, VPI&SU, Blacksburg, Va., Mar.
	5	Balci, O. (1987b). The implementation of four conceptual frameworks for simulation modeling in high-level languages. In Proceedings of the 1988 Winter Simulation Conference (San Diego, Calif., Dec. 12--14). IEEE, Piscataway, N. J., 287--295.
	6	Balci, O. and Nance, R. E. (1985). Formulated problem verification as an explicit requirement of model credibility. Simulation 45, 2 (Aug.), 76--86.
	7	Balci, O. and Nance, R. E. (1987). Simulation model development environments: a research prototype. Journal of Operational Research Society 38, 8 (Aug.), 753--763.
	8	Balci, O. and Sargent, R. G. (1981). A methodology for cost-risk analysis in the statistical validation of simulation models. Communications of the ACM 24, 4 (Apr.), 190--197.
	9	Balci, O. and Sargent, R. G. (1982a). Some examples of simulation model validation using hypothesis testing. In Proceedings of the 1982 Winter Simulation Conference (San Diego, Calif., Dec. 6--8). IEEE, New Jersey, 620--629.
	10	Balci, O. and Sargent, R. G. (1982b). Validation of multivariate response models using Hotelling's two-sample T² test. Simulation 39, 6 (Dec.), 185--192.
	11	Balci, O. and Sargent, R. G. (1983). Validation of multivariate response trace-driven simulation models. In Performance '83 (A. K. Agrawala and S. K. Tripathi, eds.). North-Holland, Amsterdam, 309--323.
	12	Balci, O. and Sargent, R. G. (1984a). A bibliography on the credibility assessment and validation of simulation and mathematical models. Simuletter 15, 3 (July), 15--27.
	13	Balci, O. and Sargent, R. G. (1984b). Validation of simulation models via simultaneous confidence intervals. American Journal of Mathematical and Management Sciences 4, 3&4, 375--406.
	14	Banks, J. and Carson, J. S. (1984). Discrete-Event System Simulation. Prentice-Hall, Englewood Cliffs, N.J.
	15	Boehm, B. W., Brown, J. R., and Lipow, M. (1976). Quantitative evaluation of software quality. In Proceedings of the 2nd International Conference on Software Engineering (San Francisco, Calif.), 592--605.
	16	Cohen, K. J. and Cyert, R. M. (1961). Computer models in dynamic economics. Quarterly Journal of Economics 75, 1 (Feb.), 112--127.
	17	Cyert, R. M. (1966). A description and evaluation of some firm simulations. In Proceedings of the IBM Scientific Computing Symposium on Simulation Models and Gaming (White Plains, N.Y.), IBM, White Plains, N.Y., 3--22.
	18	Damborg, M. J. and Fuller, L. F. (1976). Model validation using time and frequency domain error measures. ERDA Report 76--152, available from NTIS, Springfield, Va.
	19	Elmaghraby, S. E. (1968). The role of modeling in IE design. Industrial Engineering 19, 6 (June), 292--305.
	20	Emshoff, J. R. and Sisson, R. L. (1970). Design and Use of Computer Simulation Models. MacMillan, New York.
	21	Federal Computer Performance Evaluation and Simulation Center (1981). Computer Model Documentation Guide. Special Publication 500-73, Nat. Bur. of Standards, Washington, D.C., Jan.
	22	Fishman, G. S. (1978). Principles of Discrete Event Simulation. Wiley-Interscience, New York.
	23	Fishman, G. S. and Kiviat, P. J. (1967). The analysis of simulation generated time series. Management Science 13, 7 (July), 525--557.
	24	Forrester, J. W. (1961). Industrial Dynamics. MIT Press, Cambridge, Mass.
	25	Gafarian, A. V. and Walsh, J. E. (1969). Statistical approach for validating simulation models by comparison with operational systems. In Proceedings of the 4th International Conference on Operations Research, John Wiley & Sons, New York, 702--705.
	26	Gallant, A. R., Gerig, T. M., and Evans, J. W. (1974). Time series realizations obtained according to an experimental design. J. American Statistical Association 69, 347 (Sept.), 639--645.
	27	Garratt, M. (1974). Statistical validation of simulation models. In Proceedings of the 1974 Summer Computer Simulation Conference (Houston, Tex., July 9--11). Simulation Councils, La Jolla, Calif., 915--926.
	28	Gass, S. I. (1979). Computer Model Documentation: A Review and an Approach. Special Publication 500--39, Nat. Bur. of Standards, Washington, D.C., Feb.
	29	Gass, S. I. (1983). Decision-aiding models: validation, assessment, and related issues for policy analysis. Operations Research 31, 4 (July-Aug.), 603--631.
	30	Gass, S. I., Hoffman, K. L., Jackson, R. H. F., Joel, L. S., and Saunders, P. B. (1981). Documentation for a model: a hierarchical approach. Communications of the ACM 24, 11 (Nov.), 728--733.
	31	Hermann, C. F. (1967). Validation problems in games and simulations with special reference to models of international politics. Behavioral Science 12, 3 (May), 216--231.
	32	Howrey, P. and Kelejian, H. H. (1969). Simulation versus analytical solutions. In The Design of Computer Simulation Experiments (T. H. Naylor, ed.). Duke University Press, Durham, N.C., 207--231.
	33	Hunt, A. W. (1970). Statistical evaluation and verification of digital simulation models through spectral analysis. Ph.D. Dissertation, University of Texas at Austin, Austin, Tex.
	34	Kheir, N. A. and Holmes, W. M. (1978). On validating simulation models of missile systems. Simulation 30, 4 (Apr.), 117--128.
	35	Law, A. M. (1983). Statistical analysis of simulation output data. Operations Research 31, 6 (Nov.-Dec.), 983--1029.
	36	Law, A. M. and Kelton, W. D. (1982). Simulation Modeling and Analysis. McGraw-Hill, New York.
	37	Martin, J. and McClure, C. (1985). Diagramming Techniques for Analysts and Programmers. Prentice-Hall, Englewood Cliffs, N.J.
	38	Miller, D. K. (1975). Validation of computer simulations in the social sciences. In Proceedings of the Sixth Annual Conference on Modeling and Simulation (Pittsburg, Pa.), 743--746.
	39	Miller, D. R. (1974a). Model validation through sensitivity analysis. In Proceedings of the 1974 Summer Computer Simulation Conference (Houston, Tex., July 9--11). Simulation Councils, La Jolla, Calif., 911--914.
	40	Miller, D. R. (1974b). Sensitivity analysis and validation of simulation models. J. Theoretical Biology 48, 2 (Dec.), 345--360.
	41	Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review 63, 2 (Mar.), 81--97.
	42	Mitroff, I. I. (1969). Fundamental issues in the simulation of human behavior: a case in the strategy of behavioral science. Management Science 15, 12 (Dec.), B635-B649.
	43	Nance, R. E. (1977). The feasibility of and methodology for developing federal documentation standards for simulation models: final report to the National Bureau of Standards. Department of Computer Science, VPI&SU, Blacksburg, Va., June.
	44	Nance, R. E. (1981). Model representation in discrete event simulation: the Conical Methodology. Technical Report CS81003-R, Department of Computer Science, VPI&SU, Blacksburg, Va., Mar.
	45	Nance, R. E. (1987). The Conical Methodology: a framework for simulation model development. In Proceedings of the Conference on Simulation Methodology and Validation (ESC, Orlando, Fla., Apr. 6--9). Published as Simulation Series 19 (1) on Jan. 1988 by SCS, San Diego, Calif., 38--43.
	46	Naylor, T. H. and Finger, J. M. (1967). Verification of computer simulation models. Management Science 14, 2 (Feb.), B92--B101.
	47	Newton, O. L. and Weatherbee, J. E. (1980). Guidelines for documenting simulation models: a review and procedures. In Proceedings of the 1980 Winter Simulation Conference (Orlando, Fla., Dec. 3--5). IEEE, New Jersey, 243--258.
	48	Overstreet, C. M. and Nance, R. E. (1985). A specification language to assist in analysis of discrete event simulation models. Communications of the ACM 28, 2 (Feb.), 190--201.
	49	Richels, R, (1981). Building good models is not enough. Interfaces 11, 4 (Aug.), 48--54.
	50	Rowland, J. R. and Holmes, W. M. (1978). Simulation validation with sparse random data. Computers and Electrical Engineering 5, 3 (Mar.), 37--49.
	51	Schellenberger, R. E. (1974). Criteria for assessing model validity for managerial purposes. Decision Sciences 5, 4 (Apr.), 644--653.
	52	Schlesinger, S., et al. (1979). Terminology for model credibility. Simulation 32, 3 (Mar.), 103--104.
	53	Schmeiser, B. (1981). Random variate generation. In Proceedings of the 1981 Winter Simulation Conference (Atlanta, Ga., Dec. 9--11). IEEE, New Jersey, 227--242.
	54	Schruben, L. W. (1980). Establishing the credibility of simulations. Simulation 34, 3 (Mar.), 101--105.
	55	Shannon, R. E. (1975). Systems Simulation: The Art and Science. Prentice-Hall, Englewood Cliffs, N.J.
	56	Shannon, R. E. (1986). Intelligent simulation environments. In Proceedings of the Conference on Intelligent Simulation Environments (San Diego, Calif., Jan. 23--25). Published as Simulation Series 17 (1) on Jan. 1986 by SCS, San Diego, Calif., 150--156.
	57	Teorey, T. J. (1975). Validation criteria for computer system simulations. Simuletter 6, 4 (July), 9--20.
	58	Theil, H. (1961). Economic Forecasts and Policy. North-Holland, Amsterdam.
	59	Turing, A. M. (1963). Computing machinery and intelligence. In Computers and Thought (E. A. Feigenbaum and J. Feldman, eds.). McGraw-Hill, New York, 11--15.
	60	Tytula, T. P. (1978). A method for validating missile system simulation models. Technical Report E-78-11, U.S. Army Missile R&D Command, Redstone Arsenal, Ala., June.
	61	U. S. General Accounting Office (1979). Guidelines for model evaluation. PAD-79--17, U.S. G.A.O., Washington, D.C., Jan.
	62	Van Horn, R. L. (1971). Validation of simulation results. Management Science 17, 5 (May), 247--258.
	63	Watt, K. F. (1977). Why won't anyone believe us?. Simulation 28, 1 (Jan.), 1--3.
	64	Watts, D. (1969). Time series analysis. In The Design of Computer Simulation Experiments (T. H. Naylor, ed.). Duke University Press, Durham, N.C., 165--179.
	65	Whitner, R. B. and Balci, O. (1989). Guidelines for selecting and using simulation model verification techniques. In Proceedings of the 1989 Winter Simulation Conference (Washington, D.C., Dec. 4--6). IEEE, Piscataway, N.J., to appear.
	66	Wilson, J. R. and Pritsker, A. A. B. (1978). A survey of research on the simulation startup problem. Simulation 31, 2 (Aug.), 55--58.
	67	Woolley, R. N. and Pidd, M. (1981). Problem structuring - a literature review. Journal of Operational Research Society 32, 3 (Mar.), 197--206.
	68	Wright, R. D. (1972). Validating dynamic models: an evaluation of tests of predictive power. In Proceedings of the 1972 Summer Computer Simulation Conference (San Diego, Calif., July 14--16). Simulation Councils, La Jolla, Calif., 1286--1296.

CITED BY 28

	David A. Diener , Heston R. Hicks , Lawrence L. Long, Comparison of models: ex post facto validation/acceptance?, Proceedings of the 24th conference on Winter simulation, p.1095-1103, December 13-16, 1992, Arlington, Virginia, United States
	Michael L. Carpenter , Kenneth W. Bauer, Jr. , Thomas F. Schuppe , Michael A. Vidulich, Animation: what's essential for effective communication of military simulation model operation?, Proceedings of the 25th conference on Winter simulation, p.1081-1088, December 12-15, 1993, Los Angeles, California, United States
	Dana L. Wyatt, A framework for reusability using graph-based models, Proceedings of the 22nd conference on Winter simulation, p.472-476, December 09-12, 1990, New Orleans, Louisiana, United States
	Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.39-48, December 05-08, 1999, Phoenix, Arizona, United States
	Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 24th conference on Winter simulation, p.104-114, December 13-16, 1992, Arlington, Virginia, United States
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 26th conference on Winter simulation, p.77-87, December 11-14, 1994, Orlando, Florida, United States
	Robert G. Sargent, Verifying and validating simulation models, Proceedings of the 28th conference on Winter simulation, p.55-64, December 08-11, 1996, Coronado, California, United States
	Osman Balci , Anders I. Bertelrud , Chuck M. Esterbrook , Richard E. Nance, The Visual Simulation Environment technology transfer, Proceedings of the 29th conference on Winter simulation, p.1323-1329, December 07-10, 1997, Atlanta, Georgia, United States
	Michael Rooks, A unified framework for visual interactive simulation, Proceedings of the 23rd conference on Winter simulation, p.1146-1155, December 08-11, 1991, Phoenix, Arizona, United States
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 30th conference on Winter simulation, p.121-130, December 13-16, 1998, Washington, D.C., United States
	Robert G. Sargent, Simulation model verification and validation, Proceedings of the 23rd conference on Winter simulation, p.37-47, December 08-11, 1991, Phoenix, Arizona, United States
	Osman Balci , Richard E. Nance, The simulation model development environment: an overview, Proceedings of the 24th conference on Winter simulation, p.726-736, December 13-16, 1992, Arlington, Virginia, United States
	Marc Abrams , Ernest H. Page , Richard E. Nance, Simulation program development by stepwise refinement in UNITY, Proceedings of the 23rd conference on Winter simulation, p.233-242, December 08-11, 1991, Phoenix, Arizona, United States
	Robert G. Sargent, Verification, validation, and accreditation: verification, validation, and accreditation of simulation models, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
	Robert G. Sargent, Verification and validation: verification and validation of simulation models, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
	Robert G. Sargent, Verification and validation: some approaches and paradigms for verifying and validating simulation models, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
	Husam Hamad, A new metric for measuring metamodels quality-of-fit for deterministic simulations, Proceedings of the 37th conference on Winter simulation, December 03-06, 2006, Monterey, California
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 37th conference on Winter simulation, December 04-07, 2005, Orlando, Florida
	Sk Ahad Ali , Hamid Seifoddini , Hong Sun, Intelligent modeling and simulation of flexible assembly systems, Proceedings of the 37th conference on Winter simulation, December 04-07, 2005, Orlando, Florida
	Hengjie Song , Ping Ma , Ming Yang, The application of evaluation method based on HMM for results validity of complex simulation system, Proceedings of the 37th conference on Winter simulation, December 04-07, 2005, Orlando, Florida
	Husam Hamad , Sami Al-Hamdan, Two new subjective validation methods using data displays, Proceedings of the 37th conference on Winter simulation, December 04-07, 2005, Orlando, Florida
	Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 36th conference on Winter simulation, December 05-08, 2004, Washington, D.C.
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come, December 09-12, 2007, Washington D.C.
	Dimosthenis Anagnostopoulos , Mara Nikolaidou, Data Organization and Data Comparison for Model Validation in Faster-than-Real-Time Simulation, Proceedings of the 39th annual Symposium on Simulation, p.252-260, April 02-06, 2006
	Feiyan Min , Ping Ma , Ming Yang, A knowledge-based method for the validation of military simulation, Proceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come, December 09-12, 2007, Washington D.C.
	Brian F. Tivnan, Modeling organizational adaptation: a replication of Levinthal's model of emergent order, Proceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come, December 09-12, 2007, Washington D.C.
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 40th Conference on Winter Simulation, December 07-10, 2008, Miami, Florida
	Lenrick Johnston , Lee Schruben , Arden Yang , David Zhang, Establishing the credibility of a biotech simulation model, Proceedings of the 40th Conference on Winter Simulation, December 07-10, 2008, Miami, Florida