Two-stage multiple-comparison procedures for steady-state simulations

ABSTRACT

Procedures for multiple comparisons with the best are investigated in the context of steady-state simulation, whereby a number k of different systems (stochastic processes) are compared based upon their (asymptotic) means &mgr;i (i = 1,2,…, k). The variances of these (asymptotically stationary) processes are assumed to be unknown and possibly unequal. We consider the problem of constructing simultaneous confidence intervals for mi-max j≠imj i=1,2,&ldots;,k which is known as multiple comparisons with the best (MCB). Our intervals are constrained to contain 0, and so are called constrained MCB intervals. In particular, two-stage procedures for construction of absolute- and relative-width confidence intervals are presented. Their validity is addressed by showing that the confidence intervals cover the parameters with probability of at least some user-specified threshold value, as the confidence intervals' width parameter shrinks to 0. The general assumption about the processes is that they satisfy a functional central limit theorem. The simulation output analysis procedures are based on the method of standardized time series (the batch means method is a special case). The techniques developed here extend to other multiple-comparison procedures such as unconstrained MCB, multiple comparisons with a control, and all-pairwise comparisons. Although simulation is the context in this paper, the results naturally apply to (asymptotically) stationary time series.

REFERENCES

• 1: ANDERSON, T. 1971. Statistical Analysis of Time Series. John Wiley & Sons, Inc., New York, NY.
• 2: BECHHOFER, R., SANTNER, T., AND GOLDSMAN, D. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. John Wiley & Sons, Inc., New York, NY.
• 3: BILLINGSLEY, P. 1968. Convergence of Probability Measures. John Wiley & Sons, Inc., New York, NY.
• 4: DAMERDJI, H. AND NAKAYAMA, M. 1996. Two-stage multiple-comparison procedures for steady-state simulations: Res. Rep. CIS-96-30. New Jersey Institute of Technology, Newark, NJ.
• 5: DURRETT, R. 1991. Probability: Theory and Examples. Wadsworth and Brooks/Cole, Monterey, CA.
• 6: ETHIER, S. AND KURTZ, T. 1986. Markov Processes: Characterization and Convergence. John Wiley & Sons, Inc., New York, NY.
• 7: P. W. Glynn , D. L. Iglehart, Simulation output analysis using standardized time series, Mathematics of Operations Research, v.15 n.1, p.1-16, Feb. 1990 [doi>10.1287/moor.15.1.1]
• 8: P. W. Glynn , W. Whitt, Sufficient conditions for functional-limit-theorem versions of L=&lgr;W, Queueing Systems: Theory and Applications, v.1 n.3, p.279-287, March 1987 [doi>10.1007/BF01149539]
• 9: GLYNN, P. AND WHITT, W. 1991. Estimating the asymptotic variance with batch means. Oper. Res. Lett. 10, 431-435.
• 10: GOLDSMAN, D., KANG, Z., AND SEILA, n. 1993. Cram r-von Mises variance estimators for simulations: Tech. Rep. College of Computing, Georgia Institute of Technology, Atlanta, GA.
• 11: D. Goldsman , M. Meketon , L. Schruben, Properties of standardized time series weighted area variance estimators, Management Science, v.36 n.5, p.602-612, May 1990 [doi>10.1287/mnsc.36.5.602]
• 12: Lynne Goldsman , Barry L. Nelson, Batch-size effects on simulation optimization using multiple comparisons with the best, Proceedings of the 22nd conference on Winter simulation, p.288-293, December 09-12, 1990, New Orleans, Louisiana, United States
• 13: Donald Gross , Carl M. Harris, Fundamentals of queueing theory (2nd ed.)., John Wiley & Sons, Inc., New York, NY, 1985
• 14: HOCHBERG, Y. AND TAMHANE, A. 1987. Multiple Comparison Methods. John Wiley & Sons, Inc., New York, NY.
• 15: Hsu, J. 1984. Constrained simultaneous confidence intervals for multiple comparisons with the best. Ann. Stat. 12, 1136-1144.
• 16: Hsu, J. 1996. Multiple Comparisons, Theory and Methods. Chapman & Hall, London, UK.
• 17: MATEJCIK, F. AND NELSON, B. 1995. Two-stage multiple comparisons with the best for computer simulation. Oper. Res. 43, 633-640.
• 18: Marvin K. Nakayama, Two-stage stopping procedures based on standardized time series, Management Science, v.40 n.9, p.1189-1206, Sept. 1994 [doi>10.1287/mnsc.40.9.1189]
• 19: NAKAYAMA, M. 1997. Multiple-comparison procedures for steady-state simulations. Ann. Stat. 25, 2433-2450.
• 20: Barry L. Nelson , Frank J. Matejcik, Using common random numbers for indifference-zone selection and multiple comparisons in simulation, Management Science, v.41 n.12, p.1935-1945, Dec. 1995 [doi>10.1287/mnsc.41.12.1935]
• 21: NEWMAN, C. AND WRIGHT, A. 1981. An invariance principle for certain dependent sequences. Ann. Probab. 9, 671-675.
• 22: RINOTT, Y. 1978. On two-stage selection procedures and related probability-inequalities. Commun. Stat. Theor. Methods A7, 799-811.
• 23: SCHMEISER, B. 1982. Batch size effects in the analysis of simulation output. Oper. Res. 30, 556 -568.
• 24: SCHRUBEN, L. 1983. Confidence interval estimation using standardized time series. Oper. Res. 31, 1090-1108.
• 25: SLEPIAN, D. 1962. The one-sided barrier problem for Gaussian noise. Bell Syst. Tech. J. 41, 463-501.
• 26: Gamze Tokol , David Goldsman , Daniel H. Ockerman , James J. Swain, Standardized Time Series L_P-Norm Variance Estimators for Simulations, Management Science, v.44 n.2, p.234-245, February 1998 [doi>10.1287/mnsc.44.2.234]
• 27: W. Whitt, Planning queueing simulations, Management Science, v.35 n.11, p.1341-1366, Nov. 1989 [doi>10.1287/mnsc.35.11.1341]
• 28: WILCOX, R. 1984. A table for Rinott's selection procedure. J. Qual. Technol. 16, 97-100.
• 29: Mingjian Yuan , Barry L. Nelson, Multiple comparisons with the best for steady-state simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.3 n.1, p.66-79, Jan. 1993 [doi>10.1145/151527.151551]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.1 Simulation Theory
          Subjects: Model classification

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

General Terms:
Measurement, Performance, Theory

Keywords:
multiple comparisons, standardized time series, steady-state output analysis, stochastic simulation, two-stage procedures