Comparison with a standard via fully sequential procedures

ABSTRACT

We develop fully sequential procedures for comparison with a standard. The goal is to find systems whose expected performance measures are larger or smaller than a single system referred to as a standard and, if there is any, to find the one with the largest or smallest performance. The general formulation of comparison with a standard gives the standard a special status and tries to protect it when its performance is better than or even equal to performance measures of all the other alternatives. Therefore, the problem cannot be formulated as the selection of the best and a specialized procedure is required. Our procedures allow for unequal variances across systems, the use of common random numbers, and known or unknown expected performance of the standard. Experimental results are provided to compare the efficiency of the procedure with other existing procedures.

REFERENCES

• 1: Bechhofer, R. and Turnbull, B. 1978. Two (k+1)-decision selection procedures for comparing k normal means with a specified standard. J. Amer. Statis. Assoc. 73, 385--392.
• 2: Justin Boesel , Barry L. Nelson , Seong-Hee Kim, Using Ranking and Selection to "Clean Up" after Simulation Optimization, Operations Research, v.51 n.5, p.814-825, September 2003 [doi>10.1287/opre.51.5.814.16751]
• 3: Hsiao-Chang Chen , Liyi Dai , Chun-Hung Chen , Enver Yücesan, New development of optimal computing budget allocation for discrete event simulation, Proceedings of the 29th conference on Winter simulation, p.334-341, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268501]
• 4: Chen, H.-C., Chen, C.-H., and Yücesan, E. 2000. Computing efforts allocation for ordinal optimization and discrete event simulation. IEEE Trans. Auto. Cont. 45, 960--964.
• 5: Stephen E. Chick, Selecting the best system: a decision-theoretic approach, Proceedings of the 29th conference on Winter simulation, p.326-333, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268499]
• 6: Stephen E. Chick , Koichiro Inoue, New Two-Stage and Sequential Procedures for Selecting the Best Simulated System, Operations Research, v.49 n.5, p.732-743, September 2001 [doi>10.1287/opre.49.5.732.10615]
• 7: Stephen E. Chick , Koichiro Inoue, New Procedures to Select the Best Simulated System Using Common Random Numbers, Management Science, v.47 n.8, p.1133-1149, August 2001 [doi>10.1287/mnsc.47.8.1133.10229]
• 8: Fabian, V. 1974. Note on Anderson's sequential procedures with triangular boundary. Annals Statis. 2, 170--176.
• 9: Goldsman, D. and Nelson, B. 1998. Comparing systems via simulation. In Handbook of simulation: Principles, Methodology, Advances, Applications, and Practice, J. Banks, Ed. John Wiley & Sons, New York, 273--306.
• 10: Hartmann, M. 1991. An improvement on Paulson's procedure for selecting the population with the largest mean from k normal populations with a common unknown variance. Sequential Analysis 10, 1--16.
• 11: Hong, L. and Nelson, B. 2005. The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. IIE Trans., to appear.
• 12: Seong-Hee Kim , Barry L. Nelson, A fully sequential procedure for indifference-zone selection in simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.11 n.3, p.251-273, July 2001 [doi>10.1145/502109.502111]
• 13: Averill M. Law , David M. Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1999
• 14: Nelson, B. and Barnerjee, S. 2001. Selecting a good system: Procedures and inference. IIE Trans. 33, 149--166.
• 15: Barry L. Nelson , David Goldsman, Comparisons with a Standard in Simulation Experiments, Management Science, v.47 n.3, p.449-463, March 2001 [doi>10.1287/mnsc.47.3.449.9778]
• 16: Barry L. Nelson , Julie Swann , David Goldsman , Wheyming Song, Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large, Operations Research, v.49 n.6, p.950-963, November 2001 [doi>10.1287/opre.49.6.950.10019]
• 17: Paulson, E. 1952. On the comparison of several experimental categories with a control. Annals Math. Statis. 23, 239--246.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Multivariate statistics

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.6 Simulation Output Analysis

Keywords:
Comparison with a standard, output analysis, ranking and selection, simulation, variance reduction