Credibility assessment of simulation results

ABSTRACT

The purpose of this paper is to provide some guidelines for assessing the credibility of simulation results. The life cycle of a simulation study is characterized in terms of 10 phases, 10 processes, and 13 credibility assessment stages (CASs). The credibility of simulation results is assessed by integrating ten CASs: formulated problem verification, feasibility assessment, system and objectives definition verification, model qualification, communicative model verification, programmed model verification, experiment design verification, data validation, model validation, and quality assurance of experimental model. Indicators are identified for evaluating credibility in most of the CASs. The guidelines provided herein are essential for the success of a simulation study.

REFERENCES

• 1: Annino, J.S. and Russell, E.C. (1979). The ten most frequent causes of simulation analysis failure- and how to avoid them!. Simulation 32, 6 (June), 137-140.
• 2: Osman Balci, Requirements for model development environments, Computers and Operations Research, v.13 n.1, p.53-67, Jan. 1986 [doi>10.1016/0305-0548(86)90064-X]
• 3: Balci, O. (1986b). Guidelines for successful simulation studies. Technical Report TR-85-2, Department of Computer Science, Virginia Tech, Blacksburg, Va., May
• 4: B~lci, O. ~rtd Nance, R.E. (1985). Formulated problem verification as an explicit requirement of model credibility. Simulation 45, 2 (Aug.), 76-86.
• 5: Osman Balci , Robert G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, v.24 n.4, p.190-197, April 1981 [doi>10.1145/358598.358609]
• 6: Balci, O. and Sargent, R.G. (1984). A bibliography on the credibility assessment and validation of simulation and mathematical models. SimuIetter 15, 3 (July), 15-27.
• 7: B. W. Boehm , J. R. Brown , M. Lipow, Quantitative evaluation of software quality, Proceedings of the 2nd international conference on Software engineering, p.592-605, October 13-15, 1976, San Francisco, California, United States
• 8: Elmaghraby, S.E. (1968). The role of modeling ia IE design. Industrial Engineering lt?, 6 (June), 292-305.
• 9: Law, A.M. (1983). Feature article: statistical analysis of simulation output d~ta. Operations Research 31, 6 (Nov.-Dec.), 983-1029.
• 10: Monahan, J.F. (1977). The accuracy of stochastic algorithms. Tech. Rep. 23502, Brookhuven N~tional L~boratory, Upton, N.Y.
• 11: Nance, R.E. (1981). Model representation in discrete event simulation: the conical methodology. Technical Report CS81003-R, Department of Computer Science, Virginia Tech, Blacksburg, Va., Mar.
• 12: Richard E. Nance, A tutorial view of simulation model development, Proceedings of the 15th conference on Winter simulation, p.325-332, December 12-14, 1983, Arlington, Virginia, United States
• 13: Nance, R.E. and Balci, O. (1986). The objectives and requirements of model management, in Systems and Control Encyclopedia: Theory, Technology, and Applications. (M. Singh, Ed.) Pergamon Press, Oxford. In press.
• 14: Richard E. Nance , Ahmed L. Mezaache , C. Michael Overstreet, Simulation model management: Resolving the technological gaps, Proceedings of the 13th conference on Winter simulation, p.173-179, December 09-11, 1981, Atlanta, Georgia
• 15: Paul F. Roth , Saul I. Gass , Austin J. Lemoine, Some considerations for improving federal modeling, Proceedings of the 10th conference on Winter simulation, p.213-218, December 04-06, 1978, Miami Beach, FL
• 16: Schlesinger, S., et al. (1979). Terminology for model credibility. Simulation 32, 3 (Mar.), 103-t04.
• 17: Bruce Schmeiser, Random variate generation, Proceedings of the 13th conference on Winter simulation, p.227-242, December 09-11, 1981, Atlanta, Georgia
• 18: Wilson, J.R. and Pritsker, A.A.B. (1978). A survey of research on the simulation startup problem. Simulation 81, 2 (Aug.), 55-58.
• 19: Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984
• 20: Bernard P. Zeigler, Multifacetted modelling and discrete event simulation, Academic Press Professional, Inc., San Diego, CA, 1984

INDEX TERMS

Primary Classification:
  A. General Literature

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

General Terms:
Design, Measurement, Performance, Reliability, Theory, Verification