Two-phase screening procedure for simulation experiments

ABSTRACT

Analysts examining complex simulation models often conduct screening experiments to identify important factors. The controlled sequential bifurcation screening procedures CSB and CSB-X use a sequence of tests to classify factors as important or unimportant, while controlling Type I error and power. These procedures require analysts to identify the directions of the effects prior to experimentation, which can be problematic. We propose hybrid two-phase approaches, FFCSB and FFCSBX, as alternatives. Phase 1 uses an efficient fractional factorial to estimate factor effect directions; phase 2 uses CSB or CSB-X. Empirical investigations show these outperform CSB(X) in efficiency and effectiveness for many situations of practical interest.

REFERENCES

• 1: Bettonvil, B. and Kleijnen, J. P. C. 1997. Searching for important factors in simulation models with many factors: Sequential bifurcation. Eur. J. Oper. Res. 96, 180--194.
• 2: Beyer, W. H. 1987. CRC Handbook of Tables for Probability and Statistics, 2nd ed. Wiley, Boca Raton, FL.
• 3: Bosché, K. N. 2006. An empirical evaluation of a factor effects screening procedure for exploring complex simulation models. M.S. thesis, Operations Research Department, Naval Postgraduate School, Monterey, California.
• 4: Box, G. E. P., Hunter, W. G., and Hunter, J. S. 1978. Statistics for Experimenters: An Introduction to Design, Data Analysis and Model Building. Wiley, New York.
• 5: Campolongo, F., Kleijnen, J. P. C., and Andres, T. 2000. Screening methods. In Sensitivity Analysis, A. Saltelli et al., Eds. Wiley, New York, 65--89.
• 6: Russell C. H. Cheng, Searching for important factors: sequential bifurcation under uncertainty, Proceedings of the 29th conference on Winter simulation, p.275-280, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268491]
• 7: Thomas M. Cioppa , Thomas W. Lucas , Susan M. Sanchez, Military applications of agent-based simulations, Proceedings of the 36th conference on Winter simulation, December 05-08, 2004, Washington, D.C.
• 8: Dorfman, R. 1943. The detection of defective numbers of large populations. Ann. Math. Statist. 14, 436--440.
• 9: Amos Gilat, MATLAB: An Introduction with Applications, John Wiley & Sons, Inc., New York, NY, 2007
• 10: Hartman, M. 1991. An improvement of Paulson's procedure for selecting the population with the largest mean from k normal populations with a common unknown variance. Sequential Anal. 10, (1-2), 1--16.
• 11: Hedayat, A., Sloane, J., and Stufken, N. J. 1999. Orthogonal Arrays: Theory and Applications. Springer, New York.
• 12: Seong-Hee Kim, Comparison with a standard via fully sequential procedures, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.15 n.2, p.155-174, April 2005 [doi>10.1145/1060576.1060579]
• 13: Klejnen, J. P. C., Bettonvil, B., and Persson, F. 2005a. Finding the important factors in large discrete-event simulation: Sequential bifurcation and its applications. In Screening, A. M. Dean, and S. M. Lewis, Eds. Springer, New York, 287--301.
• 14: Jack P. C. Kleijnen , Susan M. Sanchez , Thomas W. Lucas , Thomas M. Cioppa, State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments, INFORMS Journal on Computing, v.17 n.3, p.263-289, Summer 2005 [doi>10.1287/ijoc.1050.0136]
• 15: Law, A. M. and Kelton, W. D. 2000. Simulation Modeling and Analysis, 3rd ed. McGraw-Hill, New York.
• 16: Lucas, T. W., Sanchez, S. M., Brown, L., and Vinyard, W. Better designs for high-dimensional explorations of simulations. In Maneuver Warfare Science 2002, G. Horne, and S. Johnson, Eds. Quantico, Virginia: USMC Project Albert. 17--46. http://www.projectalbert.org/files/MWS2002On-line.pdf {accessed July 1, 2005}.
• 17: Douglas C. Montgomery, Design and Analysis of Experiments, John Wiley & Sons, 2006
• 18: Nist/Sematech. 2005. e-Handbook of Statistical Methods. http://www.itl.nist.gov/div898/handbook/{accessed July 1, 2005}.
• 19: Oh, R. P. T. 2007. Fractional factorial controlled sequential bifurcation: Efficient factor screening through divide and discard. M.S. thesis, Operations Research Department, Naval Postgraduate School, Monterey, California.
• 20: Susan M. Sanchez , Hong Wan , Thomas W. Lucas, A two-phase screening procedure for simulation experiments, Proceedings of the 37th conference on Winter simulation, December 04-07, 2005, Orlando, Florida
• 21: Linda Trocine , Linda C. Malone, Experimental design and analysis: an overview of newer, advanced screening methods for the initial phase in an experimental design, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 22: Hong Wan , Bruce Ankenman , Barry L. Nelson, Issues on simulation and optimization I: controlled sequential bifurcation: a new factor-screening method for discrete-event simulation, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
• 23: Wan, H., Ankenman, B. E., and Nelson, B. L. 2006. Controlled sequential bifurcation: A new factor-screening method for discrete-event simulation. Oper. Res. 54, 743--755.
• 24: Wan, H., Anknman, B. E., and Nelson, B. L. 2008. Simulation factor screening with controlled sequential bifurcation in the presence of interactions. Working paper, School of Industrial Engineering, Purdue University, West Lafayette, Indiana. http://web.ics.purdue.edu/~hwan/CSBXJOCFinal.pdf {accessed June 1, 2008}.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Experimental design

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

General Terms:
Experimentation, Theory

Keywords:
Simulation theory, controlled sequential bifurcation, experimentation