Concepts and criteria to assess acceptability of simulation studies

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Concepts and criteria to assess acceptability of simulation studies: a frame of reference

Full text

Pdf (893 KB)

Source

Communications of the ACM archive
Volume 24 , Issue 4 (April 1981) table of contents
Special issue on simulation modeling and statistical computing

Pages: 180 - 189

Year of Publication: 1981

ISSN:0001-0782

Author

Tuncer I. Ören

Univ. of Ottawa, Ontario, Canada

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 7, Downloads (12 Months): 48, Citation Count: 26

Additional Information:

references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/358598.358605 What is a DOI?

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Allen, B. and Clema, J.K. Independent Verification/Validation Support Software. Proc. Nat'l Aerospace and Electron. Conf., Dayton, Ohio, March 1976, pp. 276-281.
	2	Annino, J.S. and Russell, E.C. The ten most frequent causes of simulation analysis failure-and how to avoid them. Simulation Today, 60, (June 1979), pp. 137-140.
	3	Dieter Bahr, A note on figures of merit, ACM SIGMETRICS Performance Evaluation Review, v.4 n.3, p.1-3, July 1975 [doi>10.1145/1041707.1041708]
	4	Balci, O. and Sargent, R.G. Bibliography on Validation of Simulation Models. Newsletter-TIMS College on Simulation and Gaming, Spring 1980, pp. 11-15.
	5	Beer, S. Platform for Change. Wiley, New York, 1975.
	6	Belady, L.A. and Leavenworth, B. Program Modifiability. IBM, Thomas J. Watson Research Center, Res. Rept. RC 8147, 1980.
	7	Boulton, J. and Gibson, A.R. First Ann. Rept. Canadian Nuclear Fuel Waste Management Program, Rept AECL-6443, Atomic Energy of Canada Ltd., Pinawa, Manitoba, 1979.
	8	Britting, K.R. Backward integration of system dynamics models- A useful validation test? In Worm Modeling: A Dialogue. C.W. Churchman and R.O. Mason (Eds.) Elsevier, New York, 1976, pp. 141-149.
	9	Cellier, F.E. Combined continuous/discrete system simulation by use of digital computers: Techniques and tools, Ph.D. Th.,Diss ETH No. 6483, Zurich, Switzerland. 1979, (Chapter 6-Software Robustness, pp. 121-159).
	10	Cellier, F.E. How to enhance the robustness of simulation software. In Progress in Modelling and Simulation. F.E. CeUier (Ed.) Academic (London), England, (In press).
	11	Cellier, F.E. and Moebius, P.J. Towards robust general purpose simulation software. ACM S1GNUM 14, 2, (June 1979).
	12	Cutler, M.M. Proving properties of simulation programs for system verification and validation. Proc., Summer Comptr Simulation Conf. Toronto, Canada, 1979, pp. 610-616.
	13	Melvin M. Cutler, A formal program model for discrete event simulation and its use in the verification and validation of system models and implementations., 1980
	14	Deeter, C.R. and Hoffman, A.A.J. An Annotated Bibliography of Mathematical Models Classified by Validation and Error Analysis Methods. Texas Christian University, Fort Worth, TX 1978.
	15	Richard A. De Millo , Richard J. Lipton , Alan J. Perlis, Social processes and proofs of theorems and programs, Communications of the ACM, v.22 n.5, p.271-280, May 1979 [doi>10.1145/359104.359106]
	16	Dickhoven, S. Software requirements for an improvement of transfer and adaptability of models. In Utility and Use of Large-Scale Mathematical Models, S. Gass (Ed.). Nat'l Bur Stand. Special Publication No. 534, U.S. Dept. of Commerce, Washington, D.C., May 1979, pp. 195-205.
	17	Elzas, M.S. Whither simulation? Proc., UKSC Conf. on Comptr Simulation, IPC Science and Technology Press, Chester, England, April 1978, pp. 494-501.
	18	Elzas, M.S. Results of a simulation: Can they be "trusted"? Proc. Seminar on Modelling in Business of IFIP Applied Information Processing Group, Amsterdam, The Netherlands, Jan. 1978, pp. 170- 183.
	19	Elzas, M.S. What is needed for robust simulation? In Methodology in Systems Modelling and Simulation, B.P. Zeigler et al. (Eds.) North-Holland, Amsterdam, 1979, pp. 57-91.
	20	Emshoff, J.R. and Sisson, R.L. Design and Use of Computer Simulation Models, Collier-MacMillan, New York, 1970.
	21	Gass, S.I. Evaluation of complex models. Comptrs and Oper. Res. 4. I (March, 1977), pp. 27-35.
	22	Gass, S.I. A procedure for the evaluation of complex models. Proc. First Int. Conf. in Mathematical Models, Univ. of Missouri, 1977, pp. 247-258,
	23	Gass, S.I. Computer model documentation: A review and an approach. Nat. Bur. Stand. Special Publication No. 500--39, U.S. Dept. of Commerce, Washington, D.C. Feb. 1979.
	24	Gass, S.I. (Ed.). Utility and use of large-scale mathematical models. Proc. of a Workshop held at the Nat'l. Bur. Stand., Gaithersburg, Maryland, Apr. 28-29, 1977, National Bureau of Standards, Special Publication 534, Washington, D.C., U.S. Dept. of Commerce, May 1979.
	25	Gass, S.I. Bibliography--Model evaluation and assessment. In Utility and Use of Large-Scale Mathematical Models, S.I. Gass (Ed.). Nat'l. Bur. Stand. Special Publication No. 534, U.S. Dept. of Commerce, Washington, D.C., May 1979, pp. 215-217.
	26	Gass, S.I. Validation and Assessment Issues of Energy Models. Prepared for the NATO Advanced Research Institute, The Application of Systems Science to Energy Policy Planning. Nov. 12- 16, Brookhaven National Laboratory, Upton, New York, 1979.
	27	Greenberger, M. and Richels, R. Assessing energy policy models: Current state and future directions. Ann. Rev. Energy, 4, (1979).
	28	Henize, J. Evaluating policy models. Proc. Summer Comptr Simulation Conf., Toronto, Canada, 1979, pp. 550-556.
	29	Holling, C.S. Adaptive Environmental Assessment and Management, John Wiley, New Yok, 1978.
	30	House, P.W. and McLeod, J. Large-Scale Models for Policy Evaluation, Wiley-Interscience, New York, 1977.
	31	W. E. Howden, Applicability of Software Validation Techniques to Scientific Programs, ACM Transactions on Programming Languages and Systems (TOPLAS), v.2 n.3, p.307-320, July 1980 [doi>10.1145/357103.357107]
	32	Hunt, R. Large-scale systems theories: Some paradigms from software systems engineering. In Cyberne.tics and Modelling and Simulation of Large-Scale Systems. T.I. Oren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978, pp. 141-150.
	33	Kahne, S. Model credibility for large-scale systems. IEEE Trans. SMC, (Aug. 1976), pp. 586-590.
	34	Karplus, W.J. The spectrum of mathematical modelling and systems simulation. In Simulation of Systems, L. Dekker (Ed.) North- Holland, Amsterdam, 1976, pp. 5-13.
	35	Kheir, N.A. and W.M. Homes. On validating simulation models of missile systems. Simulation, 30, 4, (Apr. 1978), pp. 117-128.
	36	Kleijnen, J.P.C. Statistical design and analysis of simulation experiments. Proc. Simulation '75, Ziirich, Switzerland Acta Press, 1975.
	37	Kleijnen, J.P.C. The role of statistical methodology in simulation. In Methodology in Systems Modelling and Simulation, B.P. Zeigler, M.S. Elzas, G.J. Klir, and T.I. Oren (Eds.) North-Holland, Amsterdam, 1979, pp. 425-445.
	38	Lee, D.B. Requiem for large-scale models. J. Am. Inst. of Planners, 39, 3 (May 1973), pp. 163-178.
	39	Marmor-Squires, A.B. Software validation: An annotated bibliography. In Current Trends in Programming Methodology. Vol. II. Program Validation, R.T. Yeh (Ed.) Prentice-Hall, Englewood Cliffs, N.J., 1977, pp. 269-319.
	40	McLeod, J. Terms and definitions-'Play it Again, Stew'. Simulation Today, 29, 4 (Oct. 1977) pp. vii-ix.
	41	Mihram, G.A. Simulation--Statistical Foundations and Methodology. Academic Press, New York, 1972.
	42	Mihram, G.A. Some practical aspects of the verification and validation of simulation models. Operational Res. Quart., 23, (1972) pp. 17-29.
	43	Mihram, G.A. Four questions regarding the credibility of simulation. Proc. 7th Ann. Pittsburgh Conf., Part 2-Modeling and Simulation, Pittsburgh, PA, Instrum. Soc. Am. Apr 1976, pp. 1225- 1230.
	44	Mihram, G.A., G. Innis, E.J. Mastascusa, S. Schlesinger, and T.J. Teorey. Bibliography on simular credibility. Simulation, 23, 5, (Dec. 1974) p. vi.
	45	Miller, E., and Howden, W.E. Tutorial: Software Testing and Validation Techniques. IEEE Comptr Soc., New York, NY, 1978.
	46	Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
	47	Nance, R.E. The feasibility of and methodology for developing federal documentation standards for simulation models. Rept. Dept. of Comptr Sci. VP1 and State University, Blacksburg, VA, 1977.
	48	Newton, O.L. and Weatherbee, J.E. Guidelines for documenting computer simulation models. Project Memorandum MV-068-060- USAF, Federal Computer Performance Evaluation and Simulation Center, Washington, DC, 1979.
	49	Newton, O.L. and Whetherbee, J.E. Guidelines for documenting simulation models: A review and procedures. In Simulation with Discrete Models: A State-of-the-Art View, T.I. Oren, C.M. Shub, and P.F. Roth (Eds.). IEEE, New York, 1980 pp. 243-258.
	50	Oren, T.I. Deductive general systems theories and simulation of large-scale systems. Proc., Summer Comptr Simulation Conf., Houston, Texas, July 1974 pp. 13-16.
	51	Oren, T.I. Simulation-as it has been, is, and should be. Simulation, 29, 5 (Nov. 1977) pp. 182-183.
	52	Oren, T.I. Modelling, model manipulation and programming concepts in simulation. In Modelling, Identification, and Control in Environmental Systems, G.C. van Steenkiste (Ed.) North-Holland, Amsterdam, 1978 pp. 833-84 Z
	53	Oren, T.I. Rationale for large-scale systems simulation software based on cybernetics and general system., theories. In Cybernetics and Modelling of Large-Scale Systems, T.I. Oren (Ed.) Int. Assoc. for Cybe.rnetics, Namur, Belgium, 1978 pp. 151-179.
	54	Oren, T.I. Reference manual of GEST 78-Level 1. (A modelling and simulation language for combined systems). Tech. Rept. 78-02, Univ. of Ottawa, Comptr Sci. Dept., Ottawa, Ontario, Canada, 1978.
	55	Oren, T.I. Epilogue. In Cybernetics and Modelling of Large-Scale Systems. T.I. Oren (Ed.) Int. Assoc. for Cybernetics, Namur, Belgium, 1978, pp. 181-189.
	56	Oren, T.I. Concepts for advanced computer-assisted modelling. In Methodology in Systems Mod..elling and Simulation, B.P. Zeigler, M.S. Elzas, G.J. Klir, and T.I. Oren (Eds.). North-Holland, Amsterdam, 1979, pp. 29-55.
	57	Oren, T.I. Computer-aided modelling systems. In Progress in Modelling and Simulation F.E. Cellier (Ed.), Academic (London), Engl. and (In press).
	58	Tuncer I. Ören , Brian Collie, Design of SEMA: A software system for computer-aided modelling and simulation of sequential machines, Proceedings of the 12th conference on Winter simulation, p.113-123, December 03-05, 1980, Orlando, FL
	59	Oren, T.I. and den Dulk, J. Ecological models expressed in GEST 78, Rept prepared for the Dept. of Theoretical Production Ecology, Dutch Agricultural Univ., Wageningen, the Netherlands, 1978.
	60	Oren, T.I. and Zeigler, B.P. Concepts for advanced simulation methodologies. Simulation, 32, 3 (March 1979) pp. 69-82.
	61	O'Shaugnessy, J. Inquiry and Decisions. Allen & Unwin, London, 1972.
	62	Robert E. Pugh, Evaluation of Policy Simulation Models: A Conceptual Approach and Case Study, Information Resources Press, Arlington, VA, 1977
	63	Paul F. Roth , Saul I. Gass , Austin J. Lemoine, Some considerations for improving federal modeling, Proceedings of the 10th conference on Winter simulation, p.213-218, December 04-06, 1978, Miami Beach, FL
	64	Salaway, G. and Siler, K.F. Validation considerations during parameter development for simulation models. Proc., Summer Comptr Simulation Conf., Toronto, Canada, July 1979 pp. 624-628.
	65	Sargent, R.G. Validation of discrete-event simulation models. Proc. NSF Workshop on Validation of Mathematical Models in Energy Related Research and Development. Forth Worth, Texas, 1978.
	66	Robert G. Sargent, Validation of simulation models, Proceedings of the 11th conference on Winter simulation, p.497-503, December 03-05, 1979, San Diego, CA, United States
	67	Sargent, R.G. Verification and validation of simulation models. In Progress in Modelling and Simulation. F.E. Cellier (Ed.), Academic (London), England, (In Press).
	68	Saridis, G.N. Comparison of six on-line identification algorithms. Automatica, 10, 1 (Jan. 1974) pp. 69-79.
	69	Schellenberger, R.E. Criteria for assessing model validity for managerial purposes. Decision Science, 5, 4 (Oct. 1974) pp. 644-653.
	70	Schlesinger, S., J.R. Buyan, E.D. Collender, W.K. Clarkson, and F.M. Perkins. Developing standard procedures for simulation validation and verification. Proc. Summer Comptr Simulation Conf., July 1974, pp. 927-933.
	71	Schlesinger, S. et al. Terminology for model credibility. Simulation, 32 3, (1979) pp. 103-104.
	72	Schruben, L.W. Establishing the credibility of simulations. Simulation,.34, 3 (March 1980) pp. 101-105.
	73	Toby J. Teorey, Validation criteria for computer system simulations, Proceedings of the 3rd symposium on Simulation of computer systems, p.161-173, August 12-14, 1975, Boulder, Colorado, United States
	74	Thissen, W. Investigations into the Club of Rome's World 3 Model - Lessons for understanding complicated models. Ph.D. Th., Dept. of Technical Physics, Eindhoven University of Technology, The Netherlands, 1978.
	75	Thissen, W. Investigations into the World 3 Model: Lessons for understanding complicated models. IEEE Trans. on Systems, Man, and Cybernetics, SMC-8, 3 ( March 1978) pp. 183-193.
	76	Toffler, A. The Eco-spasm Report. Bantam Books, New York, 1975.
	77	U.S. Dep't Justice. Descriptions and assessments of the model evaluation program projects, Stock Number 27-000-00797-0, U.S. Govt Printing Office, Washington, DC., 1979.
	78	Waller, J.D., Scanlon, J.W., Kemp, D.M., and Nalley, P.G. Developing useful evaluation capability: Lessons from the model evaluation program, Stock Number 27-000-00798-8, U.S. Govt Printing Office, Washington, DC., 1979.
	79	Watt, K.F. Why won't anyone believe us? Simulation, 28 (Jan. 1977), pp. 1-3.
	80	Wigan, M.R. The fitting, calibration, and validation of simulation models. Simulation, 18, 5, (May 1972) pp. 188-192.
	81	Wong, C.S.Y. Criteria sensitivity analysis: A new approach to parameter sensitivity. Applied Mathematical Modelling. 4, (Feb. 1980) pp. 7-15.
	82	Wright, R.D. Retrodictive tests of dynamic models. Proc. Summer Comptr Simulation Conf., Montreal, Canada, July 1973.
	83	Wymore, A.W. Systems Engineering Methodology for Interdisciplinary Teams. Wiley, New York, 1974.
	84	Yeh, R.T. (Ed.) Current Trends in Programming Methodoloy, Vol. ll--Program Validation. Prentice-Hall, Englewood Cliffs, N J, 1977.
	85	Bernard P. Zeigler, Theory of Modelling and Simulation, Krieger Publishing Co., Inc., Melbourne, FL, 1984
	86	Zeigler, B.P. Structuring the Organization of Partial..Models. In Cybernetics and Modelling of Large-Scale Systems. T.1. Oren (Ed.). Int. Assoc. for Cybernetics, Namur, Belgium, 1978 pp. 127-139.

CITED BY 26

	Bernard P. Zeigler , Tuncer I. Ören, Multifaceted, multiparadigm modeling perspectives: tools for the 90's, Proceedings of the 18th conference on Winter simulation, p.708-712, December 08-10, 1986, Washington, D.C., United States
	Osman Balci, Principles and techniques of simulation validation, verification, and testing, Proceedings of the 27th conference on Winter simulation, p.147-154, December 03-06, 1995, Arlington, Virginia, United States
	Tuncer I. Ören, Quality assurance in cognizant simulative design, Proceedings of the 18th conference on Winter simulation, p.850-852, December 08-10, 1986, Washington, D.C., United States
	Stewart Robinson , Michael Pidd, Service quality in the management of simulation projects, Proceedings of the 27th conference on Winter simulation, p.952-959, December 03-06, 1995, Arlington, Virginia, United States
	Robert G. Sargent, Verifying and validating simulation models, Proceedings of the 28th conference on Winter simulation, p.55-64, December 08-11, 1996, Coronado, California, United States
	Robert G. Sargent, An overview of verification and validation of simulation models, Proceedings of the 19th conference on Winter simulation, p.33-39, December 14-16, 1987, Atlanta, Georgia, United States
	Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.39-48, December 05-08, 1999, Phoenix, Arizona, United States
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 26th conference on Winter simulation, p.77-87, December 11-14, 1994, Orlando, Florida, United States
	Robert G. Sargent, Verification, validation, and accreditation: verification, validation, and accreditation of simulation models, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
	Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 24th conference on Winter simulation, p.104-114, December 13-16, 1992, Arlington, Virginia, United States
	Robert G. Sargent, An expository on verification and validation of simulation models, Proceedings of the 17th conference on Winter simulation, p.15-22, December 11-13, 1985, San Francisco, California, United States
	Robert G. Sargent, Validating simulation models, Proceedings of the 15th conference on Winter simulation, p.333-338, December 12-14, 1983, Arlington, Virginia, United States
	Robert G. Sargent, Simulation model verification and validation, Proceedings of the 23rd conference on Winter simulation, p.37-47, December 08-11, 1991, Phoenix, Arizona, United States
	Nur E. Ozdemirel, Measuring the user acceptance of generic manufacturing simulation models by review of modeling assumptions, Proceedings of the 23rd conference on Winter simulation, p.419-427, December 08-11, 1991, Phoenix, Arizona, United States
	Bernard P. Zeigler, Impact of general systems orientation: Present and future, Proceedings of the 13th conference on Winter simulation, p.223-225, December 09-11, 1981, Atlanta, Georgia
	Robert G. Sargent, A tutorial on verification and validation of simulation models, Proceedings of the 16th conference on Winter simulation, p.114-121, December 28-30, 1984, Dallas, TX
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 30th conference on Winter simulation, p.121-130, December 13-16, 1998, Washington, D.C., United States
	Robert G. Sargent, A tutorial on validation and verification of simulation models, Proceedings of the 20th conference on Winter simulation, p.33-39, December 12-14, 1988, San Diego, California, United States
	Berth Eklundh, Queueing structure modelling, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
	Robert G. Sargent, Verification and validation: verification and validation of simulation models, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
	Tuncer I. Ören , Maurlce S. Elzas , Grant Sheng, Model reliability and software quality assurance in simulation of nuclear fuel waste management systems, ACM SIGSIM Simulation Digest, v.16 n.4, p.4-19, October 1985
	Tuncer I. Ören, Foundations for an information technology, Proceedings of the 13th conference on Winter simulation, p.201-208, December 09-11, 1981, Atlanta, Georgia
	Osman Balci, Validation, verification, and testing techniques throughout the life cycle of a simulation study, Proceedings of the 26th conference on Winter simulation, p.215-220, December 11-14, 1994, Orlando, Florida, United States
	Robert G. Sargent, Verification and validation of simulation models, Proceedings of the 37th conference on Winter simulation, December 04-07, 2005, Orlando, Florida
	Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 36th conference on Winter simulation, December 05-08, 2004, Washington, D.C.
	Robert G. Sargent, A tutorial on verification and validation of simulation models (1984), Proceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come, December 09-12, 2007, Washington D.C.