Efficient simulation procedures: a simulation study on sampling and selecting under fixed computing budget

ABSTRACT

For many real world problems, when the design space is huge and unstructured and time consuming simulation is needed to estimate the performance measure, it is important to decide how many designs should be sampled and how long the simulation should be run for each design alternative given that we only have a fixed amount of computing time. In this paper, we present a simulation study on how the distribution of the performance measure and the distribution of the estimation error/noise will affect the decision. From the analysis, it is observed that when the noise is bounded and if there is a high chance that we can get the smallest noise, then the decision will be to sample as many as possible, but if the noise is unbounded, then it will be important to reduce the level of the noise level by assigning more simulation time to each design alternative.

REFERENCES

• 1: Chen, C. H. 1995. An Effective Approach to Smartly Allocate Computing Budget for Discrete Event Simulation. Proceedings of the 34th IEEE Conference on Decision and Control, 2598--2605.
• 2: Hsiao-Chang Chen , Liyi Dai , Chun-Hung Chen , Enver Yücesan, New development of optimal computing budget allocation for discrete event simulation, Proceedings of the 29th conference on Winter simulation, p.334-341, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268501]
• 3: Chun-Hung Chen , Yu Yuan , Hsiao-Chang Chen , Enver Yücesan , Liyi Dai, Computing budget allocation for simulation experiments with different system structure, Proceedings of the 30th conference on Winter simulation, p.735-742, December 13-16, 1998, Washington, D.C., United States
• 4: Chun-Hung Chen , Jianwu Lin , Enver Yücesan , Stephen E. Chick, Simulation Budget Allocation for Further Enhancing theEfficiency of Ordinal Optimization, Discrete Event Dynamic Systems, v.10 n.3, p.251-270, July 2000 [doi>10.1023/A:1008349927281]
• 5: L. Dai, Convergence properties of ordinal comparison in the simulation of discrete event dynamic systems, Journal of Optimization Theory and Applications, v.91 n.2, p.363-388, Nov. 1996 [doi>10.1007/BF02190101]
• 6: Goldsman, D., B. L. Nelson. 1998. Comparing systems via Simulation. J. BANK ed. The Handbook of Simulation. John Wiley, New York, 273--306.
• 7: Ho, Y. C., R. Sreenivas, P. Vakili. 1992. Ordinal Optimization of Discrete Event Dynamic Systems. Journal of Discrete Event Dynamic Systems, 2, 61--88.
• 8: Lee, L. H., T. W. E. Lau, and Y. C. Ho. 1999. Explanation of Goal softening in Ordinal Optimization. IEEE Transaction on Automatic Control, 44, (1), 94--99.
• 9: Barry L. Nelson , Julie Swann , David Goldsman , Wheyming Song, Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large, Operations Research, v.49 n.6, p.950-963, November 2001 [doi>10.1287/opre.49.6.950.10019]
• 10: Rinott, Y. 1978. On Two-stage Selection Procedures and Related Probability Inequalities. Communications in Statistics A7, 799--811.
• 11: Xie, X. L. 1997. Dynamics and Convergence Rate of Ordinal Comparison of Stochastic Discrete Event Systems. IEEE Transaction on Automatic Control, 42, (4), 586--590.