Philosophical foundations of computer simulation validation

ABSTRACT

Innovations in information technology (IT) have been responsible for the rapid growth of computer simulation, which in turn has encouraged IT innovation. However, there is a striking contrast between the technological sophistication that drives and is driven by computer simulation and the rudimentary, underdeveloped state of theory underpinning the simulation endeavor. This article attempts to provide computer simulation with a theoretical framework by the application of philosophy of science to simulation validation. The authors discuss the key elements of Popper's theory of falsification and explore how simulation validation is in fact anchored to Popper's theory. Moreover, this article suggests that a Popperian perspective encourages the review of a validated model's continuing validity, thereby potentially leading to the construction of a new, improved model. Finally, we contrast Popper's falsificationist philosophy of science with Quine's philosophical system of holism and early-period Putnam's scientific realism, both of which discourage the building of better models.

REFERENCES

• 1: Tarek K. Abdel-Hamid, The Dynamics of Software Project Staffing: A System Dynamics Based Simulation Approach, IEEE Transactions on Software Engineering, v.15 n.2, p.109-119, February 1989 [doi>10.1109/32.21738]
• 2: Tarek K. Abdel-Hamid , Stuart E. Madnick, Lessons learned from modeling the dynamics of software development, Communications of the ACM, v.32 n.12, p.1426-1438, Dec. 1989 [doi>10.1145/76380.76383]
• 3: Alaa R. Alameldeen , Milo M. K. Martin , Carl J. Mauer , Kevin E. Moore , Min Xu , Mark D. Hill , David A. Wood , Daniel J. Sorin, Simulating a $2M Commercial Server on a $2K PC, Computer, v.36 n.2, p.50-57, February 2003 [doi>10.1109/MC.2003.1178046]
• 4: Ambrus, V. (1999). Is Putnam's causal theory of meaning compatible with internal realism?Journal for General Philosophy of Science, 30, 1-16.
• 5: Anderson, R. J. (1987). The reality problem in games and simulations. In D. Crookall, C. S. Greenblat, A. Coote, J. H. G. Klabbers,& D. R. Watson (Eds.), Simulation-gaming in the late 1980s: Proceedings of the International Simulation and Gaming Association's 17th International Conference (pp. 43-50). Elmsford, NY: Pergamon.
• 6: Angell, M. (1996). Science on trial: The clash of medical evidence and the law in the breast implant case. New York: Norton.
• 7: Bradford J. A. Appleton , John Patra , Yasser Mohamed , Simaan AbouRizk, CEPM 1: special purpose simulation modeling of tower cranes, Proceedings of the 34th conference on Winter simulation: exploring new frontiers, December 08-11, 2002, San Diego, California
• 8: Osman Balci , Robert G. Sargent, A methodology for cost-risk analysis in the statistical validation of simulation models, Communications of the ACM, v.24 n.4, p.190-197, April 1981 [doi>10.1145/358598.358609]
• 9: Barlas, Y., & Carpenter, S. (1990). Philosophical roots of model validation: Two paradigms. Systems Dynamics Review, 6, 148-166.
• 10: Berenson, M. L., Levine, D. M., & Goldstein, M. (1983). Intermediate statistical methods and applications. Englewood Cliffs, NJ: Prentice Hall.
• 11: Bernstein, D. E. (1994). The admissibility of scientific evidence after Daubert v. Merrell Dow Pharmaceuticals, Inc.Cardozo Law Review, 15, 2139-2181.
• 12: Louis G. Birta , F. Nur Özmizrak, A knowledge-based approach for the validation of simulation models: the foundation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.6 n.1, p.76-98, Jan. 1996 [doi>10.1145/229493.229511]
• 13: Bohlin, T. (1994). Derivation of a designer's guide for interactive grey-box identification of nonlinear stochastic objects. International Journal of Control, 59, 1505-1524.
• 14: Bornstein, B. H. (1999). The ecological validity of jury simulations: Is the jury still out?Law and Human Behavior, 23, 75-91.
• 15: Borradori, G. (1994). The American philosopher: Conversations with Quine, Davidson, Putnam, Nozick, Danto, Rorty, Cavell, MacIntyre, and Kuhn (R. Crocitto, Trans.). Chicago: University of Chicago Press. (Original work published 1991)
• 16: Pradip Bose , Thomas M. Conte, Performance Analysis and Its Impact on Design, Computer, v.31 n.5, p.41-49, May 1998 [doi>10.1109/2.675632]
• 17: Boyd, R. (1983). On the current status of the issue of scientific realism. Erkenntnis, 19, 45-90.
• 18: Boyd, R. (1984). The current status of scientific realism. In J. Leplin (Ed.), Scientific realism (pp. 41-82). Berkeley: University of California Press.
• 19: Boyd, R. (2002). Scientific realism. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Summer 2002 ed.). Retrieved January 14, 2004, from http://plato.stanford.edu/archives/sum2002/entries/scientific-realism/
• 20: Broad, W., & Wade, N. (1982). Betrayers of the truth. New York: Simon & Schuster.
• 21: Buttà, A. (2004, June). Merger analysis in dynamic markets: The importance of innovation. Competition Law Insight, 22-23.
• 22: Carson, J. S. (1986). Convincing users of model's validity is challenging aspect of modeler's job. Industrial Engineering, 18(1), 74-78.
• 23: Caudill, D. S. (2002). Ethnography and the idealized accounts of science in law. San Diego Law Review, 39, 269-305.
• 24: Martha A. Centeno , Manuel Carrillo, Simulation in practice: challenges of introducing simulation as a decision making tool, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 25: Timothy R. Colburn, Philosophy and Computer Science: Problems and Applications, Sharpe, M.e., Inc., 1999
• 26: Conway, R. W. (1963). Some tactical problems in digital simulation. Management Science, 10(1), 47-61.
• 27: Cooper, D. R., & Schindler, P. S. (2001). Business research methods (7th ed.). New York: McGraw-Hill.
• 28: Crookall, D., Martin, A., Coote, A., & Saunders, D. (1986). Control and interaction in computerized simulations. Social Science Microcomputer Review, 4, 149-164.
• 29: David Crookall , Allan Martin , Danny Saunders , Alan Coote, Human and computer involvement in simulation, Simulation and Gaming, v.17 n.3, p.345-375, Sept. 1986 [doi>10.1177/0037550086173005]
• 30: Crookall, D., Oxford, R., & Saunders, D. (1987). Towards a reconceptualization of simulation: From representation to reality. Simulation/Games for Learning, 17, 147-171.
• 31: Crump, D. (2003). The trouble with Daubert-Kumho: Reconsidering the Supreme Court's philosophy of science. Missouri Law Review, 68, 1-42.
• 32: Curd, M., & Cover, J. A. (Eds.). (1998). Philosophy of science: The central issues. New York: Norton.
• 33: Daniels, A. M. (1996, November 8). Midas in reverse {Review of the book Science on trial}. Times Literary Supplement, p. 31.
• 34: Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993).
• 35: Davis, L. C. (2004). Effect of adaptive cruise control systems on traffic flow. Physical Review E, 69, 0661101-0661108.
• 36: Dijkum, C.van, DeTombe, D., & Kuijk, E. van. (Eds.).(1999). Validation of simulation models. Amsterdam: SISWO.
• 37: Dijkum, C. van, & Kuijk, E. van. (1999). Validation of simulation models in the social sciences. In C. van Dijkum, D. DeTombe, & E. van Kuijk (Eds.), Validation of simulation models (pp. 7-29). Amsterdam: SISWO.
• 38: Dobbin, S. A., & Gatowski, S. I. (1999). A judge's deskbook on the basic philosophies and methods of science. Reno: University of Nevada, State Justice Institute.
• 39: Dogan, B. (2002, October/November). What's new in philosophy of science?Philosophy Now, pp.14-15.
• 40: Duhem, P. (1954). The aim and structure of physical theory (P. P. Wiener, Trans.). Princeton, NJ: Princeton University Press. (Original work published 1914)
• 41: Edmond, G. (2003). After objectivity: Expert evidence and procedural reform. Sydney Law Review, 25, 131-163.
• 42: Edmond, G., & Mercer, D. (1998). Trashing "junk science."Stanford Technology Law Review, 1998. Retrieved December 19, 2004, from http://stlr.stanford.edu/STLR/Articles/98_STLR_3
• 43: Edmond, G., & Mercer, D. (2002). Conjectures and exhumations: Citations of history, philosophy and sociology of science in US federal courts. Law and Literature, 14, 309-366.
• 44: Edwards, P. N. (1999). Global climate science, uncertainty and politics: Data-laden models, model-filtered data. Science as Culture, 8, 437-472.
• 45: Edwards, R. C. (2000-2001). Who said nothing in this world is free? A&M Records, Inc. v. Napster, Inc.: Problems presented, solutions explored, and answers posed. Kentucky Law Journal, 89, 835-880.
• 46: Eisenberg, R. S. (1989). Patents and the progress of science: Exclusive rights and experimental use. University of Chicago Law Review, 56, 1017-1086.
• 47: Epstein, L. A. (2004). The technology challenge: Lawyers have finally entered the race but will ethical hurdles slow the pace?Nova Law Review, 28, 721-743.
• 48: Faigman, D. L. (2000). The law's scientific revolution: Reflections and ruminations on the law's use of experts in year seven of the revolution. Washington & Lee Law Review, 57, 661-684.
• 49: Anthony J. Faria , William J. Wellington, A Survey of Simulation Game Users, Former-Users, and Never-Users, Simulation and Gaming, v.35 n.2, p.178-207, June 2004 [doi>10.1177/1046878104263543]
• 50: Farrell, M. G. (1994). Daubert v. Merrell Dow Pharmaceuticals, Inc.: Epistemology and legal process. Cardozo Law Review, 15, 2183-2217.
• 51: Andrew Hale Feinstein , Hugh M. Cannon, A hermeneutical approach to external validation of simulation models, Simulation and Gaming, v.34 n.2, p.186-197, June 2003 [doi>10.1177/1046878103252885]
• 52: Feldman, H. L. (1995). Science and uncertainty in mass exposure litigation. Texas Law Review, 74, 1-48.
• 53: Feldman, R. (2001). Naturalized epistemology. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Fall 2001 ed.). Retrieved January 22, 2004, from http://plato.stanford.edu/archives/fall2001/entries/epistemology-naturalized/
• 54: Fenves, G. L. (2000). Advances in earthquake engineering with information technology. Proceedings of the China-U.S. Millennium Symposium of Earthquake Engineering: Earthquake Engineering in the New Millennium. Retrieved June 12, 2003, from http://cee.uiuc.edu/sstl/Beijing_Symposium/p-56/IV-4.Fenves.pdf
• 55: Fiedler, B. S. (2003-2004). Are your eyes deceiving you? The evidentiary crisis regarding the admissibility of computer generated evidence. New York Law School Law Review, 48, 295-321.
• 56: Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, A222, 309-368.
• 57: Fisher, R. A. (1925). Statistical method for research workers. Edinburgh, UK: Oliver and Boyd.
• 58: Fisher, R. A. (1935). The design of experiments. Edinburgh, UK: Oliver and Boyd.
• 59: Flew, A. (1979). A dictionary of philosophy (Rev. 2nd ed.). New York: St. Martin's.
• 60: Foster, K. R., & Huber, P. W. (1999). Judging science: Scientific knowledge and the federal courts. Cambridge, MA: MIT Press.
• 61: Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (1995). Bayesian data analysis. London: Chapman & Hall.
• 62: P. P. Gelsinger , P. A. Gargini , G. H. Parker , A. Y. C. Yu, Micro 2000, IEEE Spectrum, v.26 n.10, p.43-47, Oct. 1989 [doi>10.1109/6.40684]
• 63: Giaglis, G. M. (2001). A taxonomy of business process modelling and information systems modelling techniques. International Journal of Flexible Manufacturing Systems, 13, 209-228.
• 64: Giaglis, G. M., Paul, R. J., & O'Keefe, R. M. (1999). Integrating business and network simulation models for IT investment evaluation: A research note. Logistics Information Management, 12(1-2), 108-117.
• 65: Glymour, C. (1980). Theory and evidence. Princeton, NJ: Princeton University Press.
• 66: Golanski, A. (2001). Why legal scholars get Daubert wrong: A contextualist explanation of law's epistemology. Whittier Law Review, 22, 653-721.
• 67: Gregory, F. H. (1995). Soft systems models for knowledge elicitation and representation. Journal of the Operations Research Society, 46, 562-578.
• 68: Gregory, F. H. (1996). The need for "scientific" information systems. Proceedings of the 1996 Association for Information Systems Americas Conference, 534-536.
• 69: Grimes, A. (with Saranow, J., Vara, V., & Bank, D.) (2004, July 22). John Deere tractors meet videogames. The Wall Street Journal, p. B4.
• 70: Gross, N. (Page Ed.). (2003, June 23). Innovations {Developments to watch}. Business Week, p. 113.
• 71: Haack, S. (1993). Evidence and inquiry: Towards reconstruction in epistemology. Cambridge, MA: Blackwell.
• 72: Haack, S. (2001). An epistemologist in the bramble-bush: At the Supreme Court with Mr. Joiner. Journal of Health Politics, Policy and Law, 26, 217-248.
• 73: Haack, S. (2003). Truth, truths, "truth," and "truths" in the law. Harvard Journal of Law and Public Policy, 26, 17-21.
• 74: Haack, S. (in press). Trial and error: The Supreme Court's philosophy of science. American Journal of Public Health.
• 75: Haigh, T. (2001). Inventing information systems: The systems men and the computer, 1950-1968. Business History Review, 75(1), 15-61.
• 76: Hanfling, O. (1981). Logical positivism. New York: Columbia University Press.
• 77: Harmon, S. Y., Gross, D. C., & Youngblood, S. M. (1999). Why validation?Proceedings of the 1999Summer Computer Simulation Conference, 409-414.
• 78: Harney, D. (2002). Mickey Mousing the copyright clause of the U.S. Constitution: Eldred v. Reno. Dayton Law Review, 27, 291-311.
• 79: Dean S. Hartley, III, Verification & validation in military simulations, Proceedings of the 29th conference on Winter simulation, p.925-932, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268682]
• 80: Hempel, C. G. (1966). Philosophy of natural science. Englewood Cliffs, NJ: Prentice Hall.
• 81: Herskovitz, P. J. (1991). A theoretical framework for simulation validation: Popper's falsificationism. International Journal of Modelling and Simulation, 11, 56-58.
• 82: Himsworth, H. (1986). Scientific knowledge and philosophic thought. Baltimore: Johns Hopkins University Press.
• 83: Horwich, P. (1982). Probability and evidence. Cambridge, UK: Cambridge University Press.
• 84: Howie, D. (2002). Interpreting probability: Controversies and developments in the early twentieth century. Cambridge, UK: Cambridge University Press.
• 85: Huber, P. W. (1991). Galileo's revenge: Junk science in the courtroom. New York: Basic Books.
• 86: Hunt, S. D. (1991). Modern marketing theory: Critical issues in the philosophy of marketing science. Cincinnati, OH: South-Western.
• 87: Imwinkelried, E. J. (2003). The how and what of "appropriate validation" under Daubert: Reconsidering the treatment of Einstein and Freud. Missouri Law Review, 68, 43-52.
• 88: Sanjay Jain, Simulation in the next millennium, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.1478-1484, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324898.325306]
• 89: Jochims v. Isuzu Motors, Ltd., 145 F.R.D. 507 (S.D. Iowa 1992).
• 90: Joergensen, J. (1951). The development of logical empiricism. Chicago: Chicago University Press.
• 91: Johnson, G. (1995, May 21). Imaginary witnesses: O.J.'s blood and the big bang, together at last. The New York Times, sec. 4, p. 1.
• 92: David Kelton , Randall P. Sadowski , David T. Sturrock, Simulation with Arena, McGraw-Hill, Inc., New York, NY, 2003
• 93: Kilikauskas, M. L., Brade, D., Gravitz, R. M., Hall, D. H., Hoppus, M. L., Ketcham, R. L., et al. (2002). Estimating V&V resource requirements and scheduling impact. Retrieved July 20, 2004, from Clemson University Web site: http://www.cs.clemson.edu/found04/Foundations02/Session_Papers/B4.pdf
• 94: George B. Kleindorfear , Ram Geneshan, The philosophy of science and validation in simulation, Proceedings of the 25th conference on Winter simulation, p.50-57, December 12-15, 1993, Los Angeles, California, United States [doi>10.1145/256563.256582]
• 95: George B. Kleindorfer , Liam O'Neill , Ram Ganeshan, Validation in Simulation: Various Positions in the Philosophy of Science, Management Science, v.44 n.8, p.1087-1099, August 1998 [doi>10.1287/mnsc.44.8.1087]
• 96: Peter L. Knepell , Deborah Arangno, Simulation Validation - a Confidence Assessment Methodology, IEEE Computer Society Press, Los Alamitos, CA, 1993
• 97: Korobkin, R. B., & Ulen, T. S. (2000). Law and behavioral science: Removing the rationality assumption from law and economics. California Law Review, 88, 1051-1144.
• 98: Koslowski, B. (1996). Theory and evidence: The development of scientific reasoning. Cambridge, MA: MIT Press.
• 99: Kuhn, T. (1970a). Logic of discovery or psychology of research? In I. Lakatos & A. Musgrave (Eds.), Criticism and the growth of knowledge (pp. 1-23). Cambridge, UK: Cambridge University Press.
• 100: Kuhn, T. (1970b). The structure of scientific revolutions (2nd ed.). Chicago: University of Chicago Press.
• 101: Lakatos, I. (1970). Falsification and the methodology of scientific research programmes. In I. Lakatos & A. Musgrave (Eds.), Criticism and the growth of knowledge (pp. 91-196). Cambridge, UK: Cambridge University Press.
• 102: LaRue, L. H., & Caudill, D. S. (2004). A non-romantic view of expert testimony. Seton Hall Law Review, 35, 1-45.
• 103: Laudan, L. (1990). Science and relativism: Some key controversies in the philosophy of science. Chicago: University of Chicago Press.
• 104: Luciano Lavagno, Guest Editor's Introduction: Systems on a Chip--The Next Electronic Frontier, IEEE Micro, v.22 n.5, p.14-15, September 2002 [doi>10.1109/MM.2002.1044295]
• 105: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 106: Leavitt, H. J., & Whisler, T. L. (1958). Management in the 1980s. Harvard Business Review, 36(6), 41-48.
• 107: Byungtae Lee , Anitesh Barua , Andrew B. Whinston, Discovery and representation of causal relationships in MIS research: a methodological framework, MIS Quarterly, v.21 n.1, p.109-129, March 1997 [doi>10.2307/249744]
• 108: Leplin, J. (1984). Introduction. In J. Leplin (Ed.), Scientific realism (pp. 1-7). Berkeley: University of California Press.
• 109: Logan, R. W., & Nitta, C. K. (2003). Verification & validation: Goals, methods, levels, and metrics. Proceedings of the 2003 Summer Computer Simulation Conference, 702-707.
• 110: MacKenzie, G. R., Schulmeyer, G. G., & Yilmaz, L. (2002). Verification technology potential with different modeling and simulation development and implementation paradigms. Paper presented at the Foundations '02 Conference on V&V State of the Art. Retrieved December 19, 2004, from https://www.dmso.mil/public/library/projects/vva/found_02/sess_papers/a1.pdf
• 111: Magee, B. (1978). Men of ideas. New York: Viking.
• 112: Maitra, K. (2003). On Putnam. Boston: Wadsworth.
• 113: Marsden, G., Brackstone, M., & McDonald, M. (2001). Assessment of the stop and go function using real driving behaviour. Proceedings of the 2001 International Conference on Advanced Driver Assistance Systems, 76-80.
• 114: Mason, M. C. (2001). The scientific evidence problem: A philosophical approach. Arizona State Law Journal, 33, 887-907.
• 115: Mayo, D. G. (1996). Error and the growth of experimental knowledge. Chicago: University of Chicago Press.
• 116: Mayo, D. G. (2000). Experimental practice and an error statistical account of evidence. Philosophy of Science, 67(Proceedings), S193-S207.
• 117: McLean v. Arkansas, 529 F. Supp. 1255 (E. D. Ark. 1982).
• 118: McMullin, E. (1984). A case for scientific realism. In J. Leplin (Ed.), Scientific realism (pp. 8-40). Berkeley: University of California Press.
• 119: Meehl, P. E. (1986). What social scientists don't understand. In D. W. Fiske & R. A. Shweder (Eds.), Metatheory in social science (pp. 315-338). Chicago: University of Chicago Press.
• 120: Arvind Mehta , Ian Rawles, Business solutions using WITNESS, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.230-233, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324210]
• 121: Minium, E. W., King, B. M., & Bear, G. (1993). Statistical reasoning in psychology and education. New York: John Wiley.
• 122: Mitchell, G. (2002). Why law and economics' perfect rationality should not be traded for behavioral and economics' equal incompetence. Georgetown Law Journal, 91, 67-167.
• 123: Moenssens, A. A. (1997). Handwriting identification evidence in the post-Daubert world. University of Missouri Kansas City Law Review, 66, 251-343.
• 124: Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics, 38(8), 114-117.
• 125: Morrison, M., & Morgan, M. S. (1999). Models as mediating instruments. In M. S. Morgan & M. Morrison (Eds.), Models as mediators: Perspectives on natural and social science (pp. 10-37). Cambridge, UK: Cambridge University Press.
• 126: Nance, D. A. (2003). Reliability and the admissibility of experts. Seton Hall Law Review, 34, 191-253.
• 127: Naylor, T. H., Balintfy, J. L., Burdick, D. S., & Chu, K. (1966). Computer simulation techniques. New York: John Wiley.
• 128: Naylor, T. H., & Finger, J. M. (1967). Verification of computer simulation models. Management Science, 14, B92-B101.
• 129: Newberg, J. A., & Dunn, R. L. (2002). Keeping secrets in the campus lab: Law, values and rules of engagement for industry-university R&D partnerships. American Business Law Journal, 39, 187-240.
• 130: Neyman, J., & Pearson, E. S. (1928a). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20A, 175-240.
• 131: Neyman, J., & Pearson, E. S. (1928b). On the use and interpretation of certain test criteria for purposes of statistical inference: Part II. Biometrika, 20A, 263-294.
• 132: Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society, 231A, 289-337.
• 133: Norris, C. (2002). Hilary Putnam: Realism, reason and the uses of uncertainty. Manchester, UK: Manchester University Press.
• 134: Notturno, M. A. (1995, July 7). Karl Popper's sense of science {Letter to the editor}. The Times Literary Supplement, p. 17.
• 135: NXIVM Corp. v. Ross Inst., 364 F.3d 471 (2d Cir. 2004).
• 136: Upul Obeysekare , Fernando F. Grinstein , Gopal Patnaik, The Visual Interactive Desktop Laboratory, IEEE Computational Science & Engineering, v.4 n.1, p.63-71, January 1997 [doi>10.1109/99.590859]
• 137: O'Connor, S. (1995). The Supreme Court's philosophy of science: Will the real Karl Popper please stand up?Jurimetrics, 35, 263-276.
• 138: Oddi, S. (2002). The tragicomedy of the public domain in intellectual property law. Hastings Communications and Entertainment Law Journal, 23, 1-64.
• 139: Ernest H. Page , Bradford S. Canova , John A. Tufarolo, A case study of verification, validation, and accreditation for advanced distributed simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.7 n.3, p.393-424, July 1997 [doi>10.1145/259207.259375]
• 140: Heinz R. Pagels , H Pagels, Dreams of Reason: The Computer and the Rise of the Sciences of Complexity, Bantam Books, Inc., New York, NY, 1989
• 141: Passmore, J. (1985). Recent philosophers. La Salle, IL: Open Court.
• 142: Patterson, D. (2003). From postmodernism to law and truth. Harvard Journal of Law and Public Policy, 26, 49-65.
• 143: Popper, K. R. (1959). The logic of scientific discovery (K. R. Popper, with J. Freed & L. Freed, Trans.). New York: Basic Books. (Original work published 1934)
• 144: Popper, K. R. (1965). Conjectures and refutations: The growth of scientific knowledge (2nd ed.). New York: Basic Books.
• 145: Popper, K. R. (1982). The open universe: An argument for indeterminism. Totowa, NJ: Rowman and Littlefield.
• 146: Popper, K. R. (1983). Realism and the aim of science. Totowa, NJ: Rowman and Littlefield.
• 147: Poulter, S. R. (1998). Monte Carlo simulation in environmental risk assessment--Science, policy and legal issues. Risk: Health, Safety & Environment, 9, 7-26.
• 148: Prentice, R. (2003). Enron: A brief behavioral autopsy. American Business Law Journal, 40, 417-444.
• 149: Putnam, H. (1974). The "corroboration" of theories. In P. A. Schilpp (Ed.), The philosophy of Karl Popper (Vol. 1, pp. 221-239). La Salle, IL: Open Court.
• 150: Putnam, H. (1975a). Introduction: Philosophy of language and the rest of philosophy. In Mind, language and reality (pp. vii-xvii). Cambridge, UK: Cambridge University Press.
• 151: Putnam, H. (1975b). Introduction: Science as approximation to truth. In Mathematics, matter and method (pp. vii-xiv). Cambridge, UK: Cambridge University Press.
• 152: Putnam, H. (1975c). Language and reality. In Mind, language and reality (pp. 272-290). Cambridge, UK: Cambridge University Press.
• 153: Putnam, H. (1978). Meaning and the moral sciences. London: Routledge & Kegan Paul.
• 154: Putnam, H. (1981). Reason, truth and history. Cambridge, UK: Cambridge University Press.
• 155: Putnam, H. (1990). A defense of internal realism. In J. Conant (Ed.), Realism with a human face (pp. 30-42). Cambridge, MA: Harvard University Press.
• 156: Putnam, H. (1996). Pragmatism and realism. Cardozo Law Review, 18, 153-170.
• 157: Quinn, P. L. (1984). The philosopher of science as expert witness. In C. F. Delaney, J. T. Cushing, & G. Gutting (Eds.), Science and reality: Recent work in the philosophy of science (pp. 32-53). Notre Dame, IN: University of Notre Dame Press.
• 158: Quine, W. V. (1960). Word and object. Cambridge, MA: MIT Press.
• 159: Quine, W. V. (1964). Two dogmas of empiricism. In From a logical point of view (Rev. 2nd ed., pp. 20-46). Cambridge, MA: Harvard University Press.
• 160: Quine, W. V. (1969). Epistemology naturalized. In Ontological relativity and other essays (pp. 69-90). New York: Columbia University Press.
• 161: Quine, W. V. (1974). On Popper's negative methodology. In P. A. Schilpp (Ed.), The philosophy of Karl Popper (Vol. 1, pp. 218-220). La Salle, IL: Open Court.
• 162: Quine, W. V. (1975). On empirically equivalent systems of the world. Erkenntnis, 9, 313-328.
• 163: Quine, W. V. (1990). Pursuit of truth. Cambridge, MA: Harvard University Press.
• 164: Quine, W. V. (2004). Two dogmas in retrospect. In R. F. Gibson Jr. (Ed.), Quintessence: Basic readings from the philosophy of W. V. Quine (pp. 54-63). Cambridge, MA: Harvard University Press.
• 165: Quine, W. V., & Ullian, J. S. (1970). The web of belief. New York: Random House.
• 166: Rai, A. K. (1999). Regulating scientific research: Intellectual property rights and the norms of science. Northwestern University Law Review, 94, 77-152.
• 167: Lei Rao , Larry Owen , David Goldsman, Development and application of a validation framework for traffic simulation models, Proceedings of the 30th conference on Winter simulation, p.1079-1086, December 13-16, 1998, Washington, D.C., United States
• 168: Redman, D. A. (1991). Economics and the philosophy of science. New York: Oxford University Press.
• 169: Refsgaard, J. C., & Henriksen, H. J. (2002). Modelling guidelines--A theoretical framework (HarmoniQuA Report No. D-WP1-10). Retrieved January 25, 2004, from http://harmoniqua.wau.nl/Public%2520docs/soa/SOA-03-Framework-Final.pdf
• 170: Regis, E. (2003). The info mesa: Science, business, and new age alchemy on the SantaFe plateau. New York: Norton.
• 171: Matt Reilly , John Edmondson, Performance Simulation of an Alpha Microprocessor, Computer, v.31 n.5, p.50-58, May 1998 [doi>10.1109/2.675634]
• 172: Rhoads, B. L., & Thorn, C. E. (1996). Observation in geomorphology. In B. L. Rhoads & C. E. Thorn (Eds.), Proceedings of the 27th Binghamton Symposium in Geomorphology (pp.21-56).New York: John Wiley.
• 173: Rice, P. R. (2003). The quagmire of scientific expert testimony: Crumping the Supreme Court's style. Missouri Law Review, 68, 53-69.
• 174: Matthew W. Rohrer, Seeing is believing: the importance of visualization in manufacturing simulation, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 175: Roscoe Pound Foundation. (1998). Scientific evidence in the courts: Concepts and controversies. Washington, DC: Author.
• 176: Rosenthal, R., & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York: McGraw-Hill.
• 177: K. Guus C. de Ruiter , Joost M. Sluijs , Wilbert B. Stoutjesdijk, Simulation for recurring decisions, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 178: Ruse, M. (Ed.). (1996). But is it science? The philosophical question in the creation/evolution controversy. Amherst, NY: Prometheus Books.
• 179: Deborah A. Sadowski , Mark R. Grabau, Simulation practice: tips for successful practice of simulation, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 180: Santillana, G. de, & Zilsel, E. (1941). The development of rationalism and empiricism. Chicago: Chicago University Press.
• 181: Robert G. Sargent, Validation and verification of simulation models, Proceedings of the 31st conference on Winter simulation: Simulation---a bridge to the future, p.39-48, December 05-08, 1999, Phoenix, Arizona, United States [doi>10.1145/324138.324148]
• 182: Robert G. Sargent, Verification, validation, and accreditation: verification, validation, and accreditation of simulation models, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 183: Robert G. Sargent, Verification and validation: verification and validation of simulation models, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
• 184: Albert Semtner, Ocean and climate modeling, Communications of the ACM, v.43 n.4, p.80-89, April 2000 [doi>10.1145/332051.332077]
• 185: Serrano, A., Eatock, J., & Macredie, R. (2000). Integration of computer network and business process discrete event simulation models towards a generic reengineering approach. Proceedings of the 2000 Western Multiconference, 113-118.
• 186: Shannon, R. E. (1975). Systems simulation: The art and science. Englewood Cliffs, NJ: Prentice Hall.
• 187: Jonathan Jingsheng Shi, Practical approaches for validating a construction simulation, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 188: Simon, H. A. (1976). Administrative behavior (3rd ed.). New York: Free Press.
• 189: Simon, H. A. (1982). Models of bounded rationality (Vols. 1-2). Cambridge, MA: MIT Press.
• 190: Smart, J. J. C. (1969). Quine's philosophy of science. In D. Davidson & J. Hintikka (Eds.), Words and objections, essays on the work of W. V. Quine (pp. 3-13). Dordrecht, the Netherlands: D. Reidel.
• 191: Sosa, E. (1993). Putnam's pragmatic realism. Journal of Philosophy, 90, 605-626.
• 192: Mark Stanford , Robert Graham, Are business managers and non-technical consultants ready for low-cost discrete-event simulation? A survey of users, Proceedings of the 30th conference on Winter simulation, p.1333-1338, December 13-16, 1998, Washington, D.C., United States
• 193: Suppe, F. (1977). Afterword--1977. In F. Suppe (Ed.), The structure of scientific theories (2nd ed., pp. 614-730). Urbana: University of Illinois Press.
• 194: Precha Thavikulwat, The Architecture of Computerized Business Gaming Simulations, Simulation and Gaming, v.35 n.2, p.242-269, June 2004 [doi>10.1177/1046878104263545]
• 195: Trout, J. D. (1998). Measuring the intentional world: Realism, naturalism, and quantitative methods in the behavioral sciences. New York: Oxford University Press.
• 196: Ulrich, W. (1994). Critical heuristics of social planning: A new approach to practical philosophy. Chichester, UK: Wiley.
• 197: Ulrich, W. (2001). A philosophical staircase for information systems definition, design, and development: A discursive approach to reflective practice in ISD (part 1). Journal of Information Technology Theory & Application, 3(3), 55-84.
• 198: Vangheluwe, H., Lara, J. de, & Mosterman, P. J. (2002). An introduction to multi-paradigm modeling and simulation. Proceedings of the 2002 Conference on AI, Simulation and Planning in High Autonomy Systems, 9-20.
• 199: Vreede, G-J. de, & Eijck, D. T. T. van. (1998). Modeling and simulating organizational coordination. Simulation & Gaming, 29, 60-87.
• 200: Wallerstein, I. (2004). The uncertainties of knowledge. Philadelphia: Temple University Press.
• 201: Watkins, J. W. N. (1984). Science and skepticism. Princeton, NJ: Princeton University Press.
• 202: Werden, G. J., Froeb, L. M., & Scheffman, D. T. (2004). A Daubert discipline for merger simulation. Retrieved August 3, 2004, from Federal Trade Commission Web site: http://www.ftc.gov/be/daubertdiscipline.pdf
• 203: James R. Wilson, Conduct, misconduct, and cargo cult science (doctoral colloquium keynote address), Proceedings of the 29th conference on Winter simulation, p.1405-1413, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268790]
• 204: Yu, P. K. (2004). The escalating copyright wars. Hofstra Law Review, 32, 907-951.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.3 Applications

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis

General Terms:
Design, Documentation, Human Factors

Keywords:
computer simulation validation, evidence, falsificationism, holism, information technology, intellectual property law, model validation, model verification, philosophy of science, proof, scientific realism, theoretical framework