Getting more from the data in a multinomial selection problem

ABSTRACT

We consider the problem of determining which of k simulated systems is most likely to be the best per former based on some objective performance measure. The standard experiment is to generate v in dependent vector observations (replications) across the k- systems. A classical multinomial selection pro cedure, BEM (Bechhofer, Elmaghraby, and Morse), prescribes a minimum number of replications so that the probability of correctly selecting the true best system meets or exceeds a prespecified probability. Assuming that larger is better, BEM selects as best the system having the largest value of the performance measure in more replications than any other. We propose using these same v replications across k systems to form vk pseudoreplications (no longer in dependent) that contain one observation from each system, and again select as best the system having the largest value of the performance measure in more pseudoreplications than any other. We expect that this new procedure, AVC (all vector comparisons), dominates BEM in the sense that AVC will never require more independent replications than DEM to meet a prespecified probability of correct selection. We present analytical and simulation results to show how AVC fares versus BEM for different underly ing distribution families, different numbers of populations and various values of v. We also present results for the closely related problem of estimating the probability that a specific system is the best.

REFERENCES

• 1: Bechhofer, R. E., S. Elmaghraby, and N. Morse. 1959. A single-sample multiple-decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics 30:102-119.
• 2: Bechhofer, R. E., T. Santner, and D. Goldsman. 1995. Design and analysis of experiments for statistical selection, screening and multiple comparisions. New York: John Wiley & Sons, Inc.
• 3: Kesten, H., and N. Morse. 1959. A property of the multinomial distribution. Annals of Mathematical Statistics 30:120-127.
• 4: P~ndles, R. H., and D. A. Wolfe. 1979. Introduction to the theory of nonparametric statistics. New York: John Wiley.