Steady-state simulation of queueing processes: survey of problems and solutions

ABSTRACT

For years computer-based stochastic simulation has been a commonly used tool in the performance evaluation of various systems. Unfortunately, the results of simulation studies quite often have little credibility, since they are presented without regard to their random nature and the need for proper statistical analysis of simulation output data. This paper discusses the main factors that can affect the accuracy of stochastic simulations designed to give insight into the steady-state behavior of queuing processes. The problems of correctly starting and stopping such simulation experiments to obtain the required statistical accuracy of the results are addressed. In this survey of possible solutions, the emphasis is put on possible applications in the sequential analysis of output data, which adaptively decides about continuing a simulation experiment until the required accuracy of results is reached. A suitable solution for deciding upon the starting point of a steady-state analysis and two techniques for obtaining the final simulation results to a required level of accuracy are presented, together with pseudocode implementations.

REFERENCES

• 1: ABATE, J., AND WHITT, W. 1987a. Transient behavior of regulated brownian motion, I: Starting at the origin. Adv. Appl. Probab. 19, 560-598.
• 2: ABATE, J., AND WHITT, W. 1987b. Transient behavior of regulated brownian motion, II: Non-zero initial conditions. Adv. Appl. Probab. 19, 599-631.
• 3: J. Abate , W. Whitt, Transient behavior of the M/M/1 queue: starting at the origin, Queueing Systems: Theory and Applications, v.2 n.1, p.41-65, June 1987 [doi>10.1007/BF01182933]
• 4: ABATE, J., AND WHITT, W. 1988. Transient behavior of the M/M/1 queue via Laplace transform. Adv. Appl. Probab. 20, 145-178.
• 5: ADAM, N. R. 1983. Achieving a confidence interval for parameters estimated by simulation. Manage. Sci., 29, 856-866.
• 6: AMEMIYA, T. 1973. Generalized least squares with an estimated autocovariance matrix. Econometria 41,723-732.
• 7: Kim Andersen, Some steady-state problems in simulation, Proceedings of the 18th annual symposium on Simulation, p.79-93, March 1985, Tampa, Florida, United States
• 8: Richard W. Andrews , Thomas J. Schriber, Interactive analysis of output from GPSS-based simulations, Proceedings of the 10th conference on Winter simulation, p.267-279, December 04-06, 1978, Miami Beach, FL
• 9: Rowena Añonuevo , Barry L. Nelson, Automated estimation and variance reduction for steady-state simulations, Proceedings of the 18th conference on Winter simulation, p.871-875, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318534]
• 10: Artificial intelligence and simulation: The diversity of applications. 1988. In Proceedings of the SCS Multiconference (San Diego, Calif.). Society for Computer Simulation.
• 11: ASGARKHANI, M., AND PAWLIKOWSKI, K. 1989. Simulation studies of mixed traffic on satellite channels using TDMA-reservation protocol. In Proceedings of the 8th Phoenix International Conference on Computers and Communications (Scottsdale, Ariz.). IEEE, Washington, D.C., 195-200.
• 12: Doo-Kwon Baik , Bernard P. Zeigler, Performance evaluation of hierarchical distributed simulators, Proceedings of the 17th conference on Winter simulation, p.421-427, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253421]
• 13: David William Balmer, Polishing the analysis of the statistical output of comparative simulation experiments, Simulation, v.49 n.3, p.123-126, September 1, 1987 [doi>10.1177/003754978704900306]
• 14: BANKS, J., AND CARSON, J. S. 1984. Discrete Event System Simulation. Prentice-Hall, Englewood Cliffs, N.J.
• 15: BARTLETT, M. S. 1946. On the theoretical specification of the sampling properties of autocorrelated time series. J. Roy. Star. Soc. Ser. B, 8, 27-41.
• 16: Kenneth W. Bauer, Jr. , Sekhar Venkatraman , James R. Wilson, Estimation procedures based on control variates with known covariance matrix, Proceedings of the 19th conference on Winter simulation, p.334-341, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318599]
• 17: Charles W. Beall, A regression technique for determining steady state conditions in time series simulations, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
• 18: Peter C. Bell , Robert M. O'Keefe, Visual interactive simulation - history, recent developments, and major issues, Simulation, v.49 n.3, p.109-116, September 1, 1987 [doi>10.1177/003754978704900304]
• 19: BHARATH-KUMAR, K., AND KERMANI, P. 1984. Performance evaluation tool (PET): An analysis tool for computer communication networks. IEEE J. Select. Areas Commun. SAC-2, 220-225.
• 20: BILLINGSLEY, P. 1968. Convergence of Probability Measures. Wiley, New York.
• 21: BLACKMAN, R. B., AND TUCKEY, J. W. 1958. The measurement of power spectra from the point of view of communications engineering (Part 1 and Part 2). Bell Syst. Tech. J. 37, 185-282, 485-569.
• 22: J. P. C. Blanc, The Transient Behaviour of Networks with Infinite Server Nodes, Proceedings of the Tenth International Symposium on Computer Performance Modelling, Measurement and Evaluation, p.159-174, December 19-21, 1984
• 23: BLANC, J. P. C. 1985b. The relaxation time of two queueing systems in series. Commun. Statist. Stochastic Models 1, 1-16.
• 24: BLOMQVIST, N. 1967. The covariance function of the M/G/1 queueing system. Skandinavisk Aktuarietidskrift, 6, 157-174.
• 25: BLOMQVIST, i. 1970. On the transient behavior of the GI/G/1 waiting-times. Skandinavisk Aktuarietidshrift, 53, 118-129.
• 26: BLUM, A., ET AL. 1985. Experiments with decomposition of extended queueing network models. In Modelling Techniques and Tools for Performance Analysis, D. Potier, Ed. North-Holland, The Netherlands, 623-638.
• 27: George Edward Pelham Box , Gwilym Jenkins, Time Series Analysis, Forecasting and Control, Holden-Day, Incorporated, 1990
• 28: Paul Bratley , Bennett L Fox , Linus E Schrage, A guide to simulation (2nd ed.), Springer-Verlag New York, Inc., New York, NY, 1986
• 29: BRILLINGER, D. R. 1973. Estimation of the mean of a stationary time series by sampling. J. Appl. Prob., 10, 419-431.
• 30: David R. Brillinger, Time series: data analysis and theory, Society for Industrial and Applied Mathematics, Philadelphia, PA, 2001
• 31: Peter J Brockwell , Richard A Davis, Time series: theory and methods, Springer-Verlag New York, Inc., New York, NY, 1986
• 32: William G. Bulgren, Discrete System Simulation, Prentice Hall PTR, Upper Saddle River, NJ, 1982
• 33: James M. Calvin, Covariance of regenerative mean and variance estimators for Markov chains, Proceedings of the 20th conference on Winter simulation, p.473-475, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318235]
• 34: CARSON, J. S., AND LAW, A. M. 1980. Conservation Equations and Variance Reduction in Queueing Simulations. Oper. Res., 28, 535-546.
• 35: Catalog of simulation software, Simulation, v.49 n.4, p.165-181, October 1, 1987
• 36: Brian W. Unger, Distributed simulation, Proceedings of the 20th conference on Winter simulation, p.198-205, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318189]
• 37: John M. Charnes , W. David Kelton, A comparison of confidence region estimators for multivariate simulation output, Proceedings of the 20th conference on Winter simulation, p.458-465, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318233]
• 38: Der-Fa R. Chen , Andrew F. Seila, Multivariate inference in stationary simulation using batch means, Proceedings of the 19th conference on Winter simulation, p.302-304, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318436]
• 39: CHENG, R. C. H. 1976. A note on the effect of initial conditions on a simulation run. Oper. Res. Quart., 24, 467-470.
• 40: Russell C. H. Cheng, Variance reduction methods, Proceedings of the 18th conference on Winter simulation, p.60-68, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318261]
• 41: Gordon M. Clark , Wei-ning Yang, A Bonferroni selection procedure when using commom random numbers with unknown variances, Proceedings of the 18th conference on Winter simulation, p.313-315, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318452]
• 42: COHEN, J. W. 1982. The Single Server Queue. North- Holland, Amsterdam.
• 43: CONWAY, R. W. 1963. Some tactical problems in digital simulation. Manage. Sci., 10, 47-61.
• 44: CONWAY, R. W., JOHNSON, B. M., AND MAXWELL, W. C. 1959. Some problems of digital systems simulation. Manage. Sci. 6, 92-110.
• 45: COTTRELL, M., FORT, J. C., AND MALGOUYRES, G. 1983. Large deviations and rare events in the study of stochastic algorithms. IEEE Trans. Automatic Control, AC-28, 907-918.
• 46: COX, D. R., AND SMITH, S. W. L. 1961. Queues. Wiley, New York.
• 47: Michael A. Crane , Donald L. Iglehart, Simulating Stable Stochastic Systems, I: General Multiserver Queues, Journal of the ACM (JACM), v.21 n.1, p.103-113, Jan. 1974 [doi>10.1145/321796.321805]
• 48: CRANE, M. A., AND IGLEHART, D. L. 1975a. Simulating stable stochastic systems: III. Regenerative processes and discrete-event simulations. Oper. Res., 23, 33-45.
• 49: CRANE, M. A., AND IGLEHART, D. L. 1975b. Simulating stable stochastic systems. IV: Approximation techniques. Manage. Sci., 21, 1215-1224.
• 50: A. A. Crane , J. Lemoine, An Introduction to the Regenerative Method for Simulation Analysis, Springer-Verlag New York, Inc., Secaucus, NJ, 1977
• 51: DALEY, D. J. 1968. The serial correlation coefficients of waiting times in a stationary simple server queue. J. Austral. Math. Soc., 8, 683-699.
• 52: Halim Damerdji, On strong consistency of the variance estimator, Proceedings of the 19th conference on Winter simulation, p.305-308, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318437]
• 53: A. DeCegama, Parallel processing simulation of large computer networks, 1987 Symposium on the simulation of computer networks on Simulation of computer networks, p.51-62, August 04-07, 1987, Colorado Springs, Colorado, United States
• 54: James H. Donnelly , Robert E. Shannon, Minimum mean-squared-error estimators for simulation experiments, Communications of the ACM, v.24 n.4, p.253-259, April 1981 [doi>10.1145/358598.358637]
• 55: Ralph R. Duersch , Lee W. Schruben, An interactive run length control for simulation on PCs, Proceedings of the 18th conference on Winter simulation, p.866-870, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318533]
• 56: DUKET, S. D., AND PRITSKER, A. A. B. 1978. Examination of simulation output using spectral methods. Math. Comput. Simul., 20, 53-60.
• 57: EMSHOFF, J. R., AND SISSON, R. L. 1970. Design and Use of Computer Simulation Models. Macmillan, New York.
• 58: George S. Fishman, Problems in the statistical analysis of simulation experiments: the comparision of means and the length of sample records, Communications of the ACM, v.10 n.2, p.94-99, Feb. 1967 [doi>10.1145/363067.363103]
• 59: FISHMAN, G. S. 1971. Estimating sample size in computing simulation experiments. Manage. Sci., 18, 21-38.
• 60: FISHMAN, G. S. 1972. Bias considerations in simulation experiments. Oper. Res., 20, 685-790.
• 61: FISHMAN, G. S. 1973a. Concepts and Methods in Discrete Event Digital Simulation. John Wiley, New York.
• 62: FISHMAN, G. S. 1973b. Statistical analysis for queueing simulation. Manage. Sci., 20, 363-369.
• 63: FISHMAN, G. S. 1974. Estimation in multiserver queueing simulations. Oper. Res., 22, 72-78.
• 64: George S. Fishman, Achieving specific accuracy in simulation output analysis, Communications of the ACM, v.20 n.5, p.310-315, May 1977 [doi>10.1145/359581.359589]
• 65: George S. Fishman, Principles of Discrete Event Simulation, John Wiley & Sons, Inc., New York, NY, 1978
• 66: FISHMAN, G. S. 1978b. Grouping observations in digital simulation. Manage. Sci., 24, 510-521.
• 67: FISHMAN, G. S., AND KIVIAT, P. J. 1967. The analysis of simulation-generated time series. Manage. ScL, 13, 525-557.
• 68: Robert D. Foley , David Goldsman, Confidence intervals and orthonormally weighted standardized time series, Proceedings of the 20th conference on Winter simulation, p.422-424, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318227]
• 69: Fox, B. 1978. Estimation and simulation. Manage. Sci., 24, 860-861.
• 70: FRIEDMAN, L. W., AND FRIEDMAN, H. H. 1986. Comparing simulated alternatives using a distribution-free statistic with blocking by random number stream. Simulation, 48, 68-70.
• 71: Linda Weiser Friedman, Multivariate simulation output analysis: Past, present, and future, Proceedings of the 16th conference on Winter simulation, p.276-281, December 28-30, 1984, Dallas, TX
• 72: FROST, V. S., LAURE, W. W., AND SHANMUGAN, K. S. 1988. Efficient techniques for simulation of computer communication networks. IEEE J. Selected Areas Commun., SAC-8, 146-157.
• 73: FUJIMOTO, R. M. 1988. Performance measurements of distributed simulation strategies. In Proceedings of Distributed Simulation 1988 (San Diego, Calif.). Society for Computer Simulation, 14-20.
• 74: GAFARIAN, A. V., ANCKER C. J., AND MORISAKU, T. 1978. Evaluation of commonly used rules for detecting "steady state" in computer simulation. Naval Res. Logist. Quart. 78, 511-529.
• 75: GATES, B., CAMMARATA, S., AND ROTHENBERG, J. 1988. A demon facility for object-oriented simulation language. In Proceedings of the 1988 Summer Simulation Conference (Seattle, Wash.). Society for Computer Simulation, San Diego, Calif., 667-673.
• 76: GEISLER, M. A. 1964. The size of simulation samples required to compute certain inventory characteristics with stated precision and confidence. Manage. Sci., 10, 261-271.
• 77: Peter W. Glynn, Coverage error for confidence intervals arising in simulation output analysis, Proceedings of the 14th conference on Winter Simulation, December 06-08, 1982, San Diego, California
• 78: Peter W. Glynn , Donald L. Iglehart, Large-sample theory for standardized time series: an overview, Proceedings of the 17th conference on Winter simulation, p.129-134, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253097]
• 79: P. W. Glynn , W. Whitt, Indirect estimation via L = w, Operations Research, v.37 n.1, p.82-103, Jan.-Feb. 1989 [doi>10.1287/opre.37.1.82]
• 80: David Goldsman, Ranking and selection in simulation, Proceedings of the 15th conference on Winter Simulation, p.387-394, December 12-14, 1983, Arlington, Virginia, United States
• 81: David Goldsman, Tutorial on indifference-zone normal means ranking and selection procedures, Proceedings of the 18th conference on Winter simulation, p.370-375, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318461]
• 82: David Goldsman , Marc S. Meketon, A comparison between standardized time series and overlapping batched means, Proceedings of the 17th conference on Winter simulation, p.208-210, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253154]
• 83: GOLDSMAN, D., AND MEKETON, M. S. 1990. A comparison of several variance estimators. Oper. Res. (to be published).
• 84: David Goldsman , Keebom Kang , Robert G. Sargent, Large and small sample comparisons of various variance estimators, Proceedings of the 18th conference on Winter simulation, p.278-284, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318447]
• 85: GOLDSMAN, D., AND SCHRUBEN, L. 1984. Asymptotic properties of some confidence interval estimators for simulation output. Manage. Sci., 30, 1217-1225.
• 86: Geoffrey Gordon, System Simulation, Prentice Hall PTR, Upper Saddle River, NJ, 1977
• 87: Donald Gross , Carl M. Harris, Fundamentals of queueing theory (2nd ed.)., John Wiley & Sons, Inc., New York, NY, 1985
• 88: GUNTHER, F. L., AND WOLFF, R. W. 1980. The almost regenerative method for stochastic system simulations. Oper. Res., 28, 375-386.
• 89: Jorge Haddock, An expert system framework based on a simulation generator, Simulation, v.48 n.2, p.45-53, February 1, 1987 [doi>10.1177/003754978704800202]
• 90: HAHN, G. J. 1985. More intelligent statistical software and statistical expert systems: Future directions. Am. Star., 39, 1-16.
• 91: Richard W. Hamming, Numerical Methods for Scientists and Engineers, McGraw-Hill, Inc., New York, NY, 1973
• 92: HAND, D. J. 1985. Statistical expert systems: Necessary attributes. J. Appl. Stat., 12, 19-27.
• 93: HANNAN, E. J. 1970. Multiple Time Series. John Wiley, New York.
• 94: HARLING, J. 1985. Simulation techniques in operations research: A review. Oper. Res., 33, 307-319.
• 95: HEIDELBERGER, P. 1979. A variance reduction technique that increases regeneration frequency. In Current Issues in Computer Simulation. Academic Press, New York, 257-269.
• 96: Philip Heidelberger, Statistical analysis of parallel simulations, Proceedings of the 18th conference on Winter simulation, p.290-295, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318448]
• 97: P. Heidelberger , P. A. W. Lewis, Regression-adjusted estimates for regenerative simulations, with graphics, Communications of the ACM, v.24 n.4, p.260-273, April 1981 [doi>10.1145/358598.358641]
• 98: HEIDELBERGER, P., AND LEWIS, P. A. W. 1984. Quantile estimation in dependent sequences. Oper. Res., 32, 185-209.
• 99: Philip Heidelberger , Peter D. Welch, A spectral method for confidence interval generation and run length control in simulations, Communications of the ACM, v.24 n.4, p.233-245, April 1981 [doi>10.1145/358598.358630]
• 100: HEIDELBERGER, P., AND WELCH, P. D. 1981b. Adaptive spectral methods for simulation output analysis. IBM J. Res. Dev., 25, 860-876.
• 101: HEIDELBERGER, P., AND WELCH, P. D. 1983. Simulation run length control in the presence of an initial transient. Oper. Res., 31, 1109-1144.
• 102: Ho, Y. C. 1987. Performance evaluation and perturbation analysis of discrete event dynamic systems. IEEE Trans. Automat. Control, AC-32, 563-572.
• 103: Ho, Y. C., AND CAO, X. 1983. Perturbation analysis and optimization of queueing networks. J. Optim. Theory Appl., 40, 559-582.
• 104: HO, Y. C., CAO, X., AND CASSANDRAS, C. 1983. Infinitesimal and finite perturbation analysis for queueing networks. Automatica, 4, 439-445.
• 105: IGLEHART, D. L. 1975. Simulating stable stochastic systems. V: Comparison of ratio estimators. Naval Res. Logist. Quart., 22, 553-565.
• 106: Donald L. Iglehart, Simulating Stable Stochastic Systems, VI: Quantile Estimation, Journal of the ACM (JACM), v.23 n.2, p.347-360, April 1976 [doi>10.1145/321941.321954]
• 107: IGLEHART, D. L. 1978. The regenerative method for simulation analysis. In Current Trends in Programming Methodology. Vol. IIi, Software Modeling, K. M. Chandy and P. T. Yeh, Eds., Prentice Hall, Englewood Cliffs, N.J., 52-71.
• 108: IZYDORCZYK, J., MIERZWA, J., AND WOLISZ, A. 1984. A control variable method for increasing the efficiency of multiple access protocol simulation. In Performance of Computer-Communication Systems, H. Rudin and W. Bux, Eds., North- Holland, Amsterdam, 95-108.
• 109: JACKMAN, J., AND MEDEIROS, D. J. 1988. A graphical methodology for simulating communication networks. IEEE Trans. Commun., A COM-36, 459-464.
• 110: Raj Jain , Imrich Chlamtac, The P2 algorithm for dynamic calculation of quantiles and histograms without storing observations, Communications of the ACM, v.28 n.10, p.1076-1085, Oct. 1985 [doi>10.1145/4372.4378]
• 111: JENKINS, G. M., AND WATTS, D. G. 1968. Spectral Analysis and Its Applications. Holden-Day, San Francisco.
• 112: Keebom Kang , David Goldsman, The correlation between mean and variance estimators, Proceedings of the 17th conference on Winter simulation, p.211-216, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253155]
• 113: W. David Kelton, Statistical design and analysis, Proceedings of the 18th conference on Winter simulation, p.45-51, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318259]
• 114: W. David Kelton, Transient exponential-Erlang queues and steady-state simulation, Communications of the ACM, v.28 n.7, p.741-749, July 1985 [doi>10.1145/3894.3900]
• 115: KELTON, W. D., AND LAW. A. M. 1985. The transient behaviour of the M/M/S queue, with implications for steady-state simulation. Oper. Res., 33, 378-396.
• 116: KELTON, W. D., AND LAW, A. M. 1984. An analytical evaluation of alternative strategies in steadystate simulation. Oper. Res., 32, 169-184.
• 117: KELTON, W. D., AND LAW, A. M. 1983. A new approach for dealing with the startup problem in discrete event simulation. Naval Res. Logist. Quart., 30, 641-658.
• 118: KERCKHOFFS, E. J. H., AND VANSTEENKISTE, G. C. 1986. The impact of advanced information processing on simulation: An illustrative review. Simulation, 48, 17-26.
• 119: KIEFER, J., AND WOLFOWITZ, J. 1955. On the theory of queues with many servers. Trans. Am. Math. Soc., 78, 1-18.
• 120: KLEIJNEN, J. P. C. 1974. Statistical Techniques in Simulation, Vol. 1. Marcel Dekker, New York.
• 121: KLEIJNEN, J. P. C. 1979. The role of statistical methodology in simulation. In Methodology in Systems Modelling and Simulation, B. P. Zeigler, M. S. Elzas, G. J. Klir, and T. I. Oren, Eds. North-Holland, Amsterdam.
• 122: Jack P C Kleijnen, Statistical tools for simulation practitioners, Marcel Dekker, Inc., New York, NY, 1986
• 123: KLEIJNEN, J. P. C., VAN DER VEN, R., AND SAUNDER, B. 1982. Testing independence of simulation subruns: A note on the power of the yon Neumann test. Eur. J. Oper. Res., 9, 92-93.
• 124: KLEINROCK, L. 1976. Queueing Systems. Vol. 2, Computer Applications. John Wiley, New York.
• 125: Verna Knapp, The Smalltalk simulation environment, Proceedings of the 18th conference on Winter simulation, p.125-128, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318279]
• 126: Wolfgang Kreutzer, System simulation programming styles and languages, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1986
• 127: Murali Krishnamurthi , Usha Chandrasekaran , Sallie Sheppard, Two approaches to the implementation of a distributed simulation system, Proceedings of the 17th conference on Winter simulation, p.435-444, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253423]
• 128: KUMAR, D. 1986. A novel approach to sequential simulation. IEEE Softw., 3, 25-33.
• 129: KUROSE, J. F., AND MOUFTAH, H. T. 1988. Computer-aided modeling, analysis, and design of communication networks. IEEE J. Select. Areas Commun., SAC-6, 130-145.
• 130: LAVENBERG, S. S., AND SAUER, C. I-I. 1977. Sequential stopping rules for the regenerative method of simulation. IBM J. Res. Dev., 2I, 545-558.
• 131: LAVENBERG, S. S., AND WELCH, P. D. 1981. A perspective on the use of control variables to increase the efficiency of Monte Carlo simulations. Manage. Sci., 27, 322-335.
• 132: LAVENBERG, S. S., MOELLER, T. L., AND WELCH, P. D. 1982. Statistical results on control variables with application to queueing network simulation. Oper. Res., 30, 182-202.
• 133: Stephen S. Lavenberg , George S. Fishman , Lee W. Schruben , W. David Kelton , Peter D. Welch, The initial transient in steady state simulation (Panel Discussion), Proceedings of the 13th conference on Winter simulation, p.113-120, December 09-11, 1981, Atlanta, Georgia
• 134: LAW, A. M. 1975. Efficient estimators for simulated queueing systems. Manage. Sci., 22, 30-41.
• 135: LAW, A. M. 1977. Confidence interval in discrete event simulation: A comparison of replication and batch means. Naval Res. Logist. Quart., 24, 667-678.
• 136: LAW, A. M. 1980. Statistical analysis of the output data from terminating simulations. Naval Res. Logist. Quart., 27, 131-143.
• 137: LAW, A. M. 1983. Statistical analysis of simulation output data. Oper. Res., 3I, 983-1029.
• 138: LAW, A. M., AND CARSON, J. C. 1979. A sequential procedure for determining the length of a steady state simulation. Oper. Res., 27, 1011-1025.
• 139: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 140: LAW, A. M., AND KELTON, W. D. 1982b. Confidence intervals for steady state simulations. II: A survey of sequential procedures. Manage. Sci., 28, 550-562.
• 141: LAW, A. M., AND KELTON, W. D. 1984. Confidence intervals for steady state simulations. I: A survey of fixed sample size procedures. Oper. Res. 1221-1239.
• 142: Edward A. MacNair, An introduction to the Research Queueing Package, Proceedings of the 17th conference on Winter simulation, p.257-262, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253163]
• 143: MADANSKY, A. 1976. Optimal initial conditions for a simulation problem. Oper. Res., 24, 172-577.
• 144: Neil B. Marks, Further investigation into spectral analysis for confidence intervals in steady state simulations, Proceedings of the 13th conference on Winter simulation, p.461-464, December 09-11, 1981, Atlanta, Georgia, United States
• 145: MEKETON, M. S. 1980. The variance time-curve: Theory, estimation, and application. Ph.D. dissertation, School of Operations Research and Industrial Engineering, Cornell Univ., Ithaca, N.Y.
• 146: MEKETON, M. S., AND HEIDELBERGER, P. 1982. A renewal theoretic approach to bias reduction on regenerative simulations. Manage. Sci., 28, 173-181.
• 147: Marc S. Meketon , Bruce Schmeiser, Overlapping batch means: something for nothing?, Proceedings of the 16th conference on Winter simulation, p.226-230, December 28-30, 1984, Dallas, TX
• 148: MELLICHAMP, J. M., AND PARK, Y. H. 1989. A statistical expert system for simulation analysis. Simulation, 51,134-139.
• 149: MILLER, R. G. 1974. The Jackknife: A Review. Biometrika, 61, 1-15.
• 150: MILLS, R. 1987. Statistical Analysis of Steady State Simulation Output Data with SIMSCRIPT 11.5. CACI, Los Angeles.
• 151: Do Le Minh, Simulating GI/G/k Queues in heavy traffic, Management Science, v.33 n.9, p.1192-1199, Sept. 1987 [doi>10.1287/mnsc.33.9.1192]
• 152: Jayadev Misra, Distributed discrete-event simulation, ACM Computing Surveys (CSUR), v.18 n.1, p.39-65, March 1986 [doi>10.1145/6462.6485]
• 153: I. Mitrani, Simulation Techniques for Discrete Event Systems, Cambridge University Press, New York, NY, 1982
• 154: MORSE, P. M. 1955. Stochastic processes of waiting lines. Oper. Res., 1,255-261.
• 155: Joseph R. Murray , W. David Kelton, Initializing for bias reduction: some analytical considerations, Proceedings of the 20th conference on Winter simulation, p.546-548, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318252]
• 156: Barry L. Nelson , Bruce W. Schmeiser, Decomposition of some well-known variance reduction techniques, Journal of Statistical Computation and Simulation, v.23 n.3, p.183-209, 1986 [doi>10.1080/00949658608810871]
• 157: NEWELL, G. F. 1971. Applications of Queueing Theory. Chapman & Hall, London.
• 158: David M. Nicol, Parallel discrete-event simulation of FCFS stochastic queueing networks, Proceedings of the ACM/SIGPLAN conference on Parallel programming: experience with applications, languages and systems, p.124-137, July 19-21, 1988, New Haven, Connecticut, United States
• 159: David N. Nicol, High performance parallelized discrete event simulation of stochastic queueing networks, Proceedings of the 20th conference on Winter simulation, p.306-314, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318206]
• 160: NIELSEN, N. R. 1986. Expert systems and simulation. In Computer Networks and Simulation III, S. Schoemaker, Ed., North-Holland, Amsterdam, 41-52.
• 161: ODONI, A. R., AND ROTH, E. 1983. An empirical investigation of the transient behaviour of stationary queueing systems. Oper. Res., 34, 306-314.
• 162: O'KEEFE, R. M. 1986. Simulation and expert systems: A taxonomy and some examples. Simulation, 48, 10-16.
• 163: Tuncer I. Ören, Quality assurance paradigms for artificial intelligence in modeling and simulation, Simulation, v.48 n.4, p.149-151, April 1, 1987 [doi>10.1177/003754978704800406]
• 164: PAREKH, S., AND WALRAND, J. 1989. A quick simulation of excessive backlogs in networks of queues. IEEE Trans. Automatic Control, AC-34, 54-66.
• 165: S. K. Park , K. W. Miller, Random number generators: good ones are hard to find, Communications of the ACM, v.31 n.10, p.1192-1201, Oct. 1988 [doi>10.1145/63039.63042]
• 166: PARZEN, E. 1961. Mathematical consideration in the estimation of spectra. Technometrics, 3, 167-190.
• 167: PAWLIKOWSKI, K., AND ASGARKHANI, M. 1988. Sequential procedures in simulation studies of satellite protocols. In Proceedings of the ITC'12 (Torino, Italy), Int. Advisory Council of ITC vol. 6, 4.3B.3.1-4.3B.3.7. (Also in Teletraffic Science for New Cost-Effective Systems, Systems and Services. Vol. 12, Studies in Telecommunication. North-Holland, Amsterdam, 1989.)
• 168: James A. Payne, Introduction to Simulation: Programming Techniques and Methods Analysis, McGraw-Hill, Inc., New York, NY, 1982
• 169: Michael Pidd, Computer simulation in management science, John Wiley & Sons, Inc., New York, NY, 1984
• 170: POTIER, D. 1984. New user's introduction to QNAP2. Rapport Technique INRIA, No. 40.
• 171: POTIER, D. 1986. The modeling package QNAP2 and applications to computer networks simulation. In Computer Networks and Simulation IiI. North- Holland, Amsterdam, 235-265.
• 172: William H. Press , Brian P. Flannery , Saul A. Teukolsky , William T. Vetterling, Numerical recipes: the art of scientific computing, Cambridge University Press, New York, NY, 1986
• 173: Kimmo E. E. Raatikainen, Simultaneous estimation of several percentiles, Simulation, v.49 n.4, p.159-164, October 1, 1987 [doi>10.1177/003754978704900405]
• 174: RAATIKAINEN, K. E. E. 1988. Validating percentiles of response times in queueing network models. In Proceedings of 1988 Summer Simulation Conference (Seattle, Wash.). Society for Computer Simulation, San Diego, Calif., 136-141.
• 175: Ramana Reddy, Epistemology of knowlege based simulation, Simulation, v.48 n.4, p.162-166, April 1, 1987 [doi>10.1177/003754978704800408]
• 176: Martin I. Reiman , Alan Weiss, Sensitivity analysis via likelihood ratios, Proceedings of the 18th conference on Winter simulation, p.285-289, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318450]
• 177: Paul F. Reynolds, Jr., A spectrum of options for parallel simulation, Proceedings of the 20th conference on Winter simulation, p.325-332, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318209]
• 178: Emily Roth, A relaxation time heuristic for reducing initialization bias in simulation of M/Ek/1 queueing systems, Proceedings of the 17th conference on Winter simulation, p.263-269, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253311]
• 179: Emily Roth , Alan H. Rutan, A relaxation time approach for reducing initialization bias in simulation, Proceedings of the 18th annual symposium on Simulation, p.189-204, March 1985, Tampa, Florida, United States
• 180: R. Y. Rubinstein, Sensitivity analysis and performance extrapolation for computer simulation models, Operations Research, v.37 n.1, p.72-81, Jan.-Feb. 1989 [doi>10.1287/opre.37.1.72]
• 181: RUBINSTEIN, R. Y., AND MARCUS, R. 1985. Efficiency of multivariate control variates in Monte Carlo simulation. Oper. Res., 33, 661-677.
• 182: Sergio Ruiz-Mier , Joseph Talavage, A hybrid paradigm for modeling of complex systems, Simulation, v.48 n.4, p.135-141, April 1, 1987 [doi>10.1177/003754978704800404]
• 183: Robert G. Sargent, The use of graphical models in model validation, Proceedings of the 18th conference on Winter simulation, p.237-241, December 08-10, 1986, Washington, D.C., United States [doi>10.1145/318242.318312]
• 184: SAVER, C. H., MACNAm, E. A., AND KUROSE, J. K. 1984. Queueing network simulations of computer communications. IEEE J. Select. Areas Commun., SAC-l, 203-220.
• 185: SCHMEISER, B. 1982. Batch size effects in the analysis of simulation output. Oper. Res., 30, 556-568.
• 186: Bruce Schmeiser , Wheyming Tina Song, Correlation among estimators of the variance of the sample mean, Proceedings of the 19th conference on Winter simulation, p.309-317, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318438]
• 187: J. William Schmidt , Chin-Fu Ho, An algorithm for testing serial dependence of simulation output data, Proceedings of the 20th conference on Winter simulation, p.532-539, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318248]
• 188: Thomas J. Schriber, Simulation Using GPSS, Krieger Publishing Co., Inc., Melbourne, FL, 1990
• 189: Thomas J. Schriber , Richard W. Andrews, Interactive analysis of simulation output by the method of batch means, Proceedings of the 11th conference on Winter simulation, p.513-526, December 03-05, 1979, San Diego, CA, United States
• 190: Thomas J. Schriber , Richard W. Andrews, A conceptual framework for research in the analysis of simulation output, Communications of the ACM, v.24 n.4, p.218-232, April 1981 [doi>10.1145/358598.358626]
• 191: SCHRUBEN, L. W. 1980. A coverage function for interval estimators of simulation response. Manage. Sci., 26, 18-27.
• 192: Lee W. Shruben, Control of initialization bias in multivariate simulation response, Communications of the ACM, v.24 n.4, p.246-252, April 1981 [doi>10.1145/358598.358634]
• 193: SCHRUBEN, L. W. 1982. Detecting initialisation bias in simulation output. Oper. Res. 569-590.
• 194: SCHRUB~N, L. W. 1983. Confidence interval estimation using standardised time series. Oper. Res., 30, 1090-1108.
• 195: Lee Schruben, Overview of standardized time series, Proceedings of the 17th conference on Winter simulation, p.115-118, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253092]
• 196: Lee Schruben, Using simulation to solve problems: a tutorial on the analysis of simulation output, Proceedings of the 19th conference on Winter simulation, p.40-42, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318380]
• 197: SCHRUBEN, L. W., SINGH, H., AND TIERNEY, L. 1983. Optimal tests for initialisation bias in simulation output. Oper. Res., 31, 1167-1178.
• 198: SEILA, A. F. 1982. A batching approach to quantile estimation in regenerative simulation. Manage. Sci., 28, 573-581.
• 199: SEILA, A. F. 1982. Multivariate estimation in regenerative simulation. Oper. Res. Lett., 1,153-156.
• 200: Andrew F. Seila, Multivariate estimation in simulation, Proceedings of the 15th conference on Winter Simulation, p.693-698, December 12-14, 1983, Arlington, Virginia, United States
• 201: Perwez Shahabuddin , Victor F. Nicola , Philip Heidelberger , Ambuj Goyal , Peter W. Glynn, Variance reduction in mean time to failure simulations, Proceedings of the 20th conference on Winter simulation, p.491-499, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318239]
• 202: Robert E. Shannon, Tests for the verification and validation of computer simulation models, Proceedings of the 13th conference on Winter simulation, p.573-577, December 09-11, 1981, Atlanta, Georgia, United States
• 203: SHANTHIKUMAR, J., AND SARGENT, R. 1983. A unifying view of hybrid simulation/analytic models and modeling. Oper. Res., 31, 1030-1052.
• 204: SHAPIRO, S. S., AND WILK, M. B. 1965. An analysis of variance test for normality (complete) samples. Biometrika, 52, 591-611.
• 205: G. S. Shedler, Regeneration and networks of queues, Springer-Verlag New York, Inc., New York, NY, 1987
• 206: SOLOMON, S. L. 1983. Simulation of Waiting-Line Systems. Prentice-Hall, Englewood Cliffs, N.J.
• 207: SONG, W. T. 1988. Estimators of the Variance of the Simple Mean: Quadratic Forms, Optimal Batch Sizes, and Linear Combinations. Ph.D. Dissertation, School of Industrial Engineering, Purdue University, West Lafayette, Indiana, 1988.
• 208: Wheyming Tina Song , Bruce Schmeiser, Minimal-MSE linear combinations of variance estimators of the sample mean, Proceedings of the 20th conference on Winter simulation, p.414-421, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318226]
• 209: Malcolm C. Stairmand , Wolfgang Kreutzer, POSE: a process-oriented simulation environment embedded in SCHEME, Simulation, v.50 n.4, p.143-153, April 1988 [doi>10.1177/003754978805000404]
• 210: Rajan Suri , Michael A. Zazanis, Perturbation analysis gives strongly consistent sensitivity estimates for the M/G/1 queue, Management Science, v.34 n.1, p.39-64, Jan. 1988 [doi>10.1287/mnsc.34.1.39]
• 211: K. D Tocher, The art of simulation, English Universities Press, 1967
• 212: Kishar Shridharbhai Trivedi, Probability and Statistics with Reliability, Queuing and Computer Science Applications, Prentice Hall PTR, Upper Saddle River, NJ, 1982
• 213: TURNQUIST, M. A., AND SUSSMAN, J. M. 1977. Toward guidelines for designing experiments in queuing simulation. Simulation, 28, 137-144.
• 214: Brian W. Unger, Distributed simulation, Proceedings of the 20th conference on Winter simulation, p.198-205, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318189]
• 215: VASSILACOPOULOS, G. 1989. Testing for initialization bias in simulation output. Simulation, 52, 151-153.
• 216: James M. Velayas , Reuven R. Levary, Validation of simulation models using decision theory, Simulation, v.48 n.3, p.87-92, March 1987 [doi>10.1177/003754978704800302]
• 217: Sekhar Venkatraman , James R. Wilson, Using path control variates in activity network simulation, Proceedings of the 17th conference on Winter simulation, p.217-222, December 11-13, 1985, San Francisco, California, United States [doi>10.1145/21850.253156]
• 218: VENKATRAMAN, S., AND WILSON, J. R. 1986. The efficiency of control variates in multiresponse simulation. Oper. Res. Lett. 5, 37-42.
• 219: YON NEUMANN, J. 1941. Distribution of the ratio of the mean square successive difference to the variance. Ann. Math. Star., 12, 367-395.
• 220: D. B. Wagner , E. D. Lazowska, Parallel simulation of queueing networks: limitations and potentials, ACM SIGMETRICS Performance Evaluation Review, v.17 n.1, p.146-155, May 1989
• 221: WAHBA, G. 1980. Automatic smoothing of the log periodogram, j. Am. Star. Assoc., 57, 122-132.
• 222: WALRAND, J. 1988. Computer Performance and Reliability: Proceedings of the 2nd int. NCPR Workshop (Rome). North-Holland, Amsterdam, 275-286.
• 223: WELCH, P. D. 1983. Statistical Analysis of Simulation Results. In Computer Performance Modelling Handbook, S. S. Lavenberg, Ed. Academic Press, New York, Chap. 6.
• 224: Peter D. Welch, On the relationship between batch means, overlapping means and spectral estimation, Proceedings of the 19th conference on Winter simulation, p.320-323, December 14-16, 1987, Atlanta, Georgia, United States [doi>10.1145/318371.318440]
• 225: WILSON, J. R. 1984. Variance reduction techniques for digital simulation. Am. J. Math. Manage. Sci., 4, 277-312.
• 226: WILSON, J. R. 1983. Variance reduction: The current state. Math. Comput. Simul., 26, 55-59.
• 227: WILSON, J. R., AND PRITSKER, A. A. B. 1978a. A survey of research on the simulation startup problem. Simulation, 31, 55-58.
• 228: WILSON, J. R., AND PRITSKER, A. A. B. 1978b. Evaluation of startup policies in simulation experiments. Simulation, 31, 79-89.
• 229: Wei-Ning Yang , Barry L. Nelson, Multivariate estimation and variance reduction in terminating and steady-state simulation, Proceedings of the 20th conference on Winter simulation, p.466-472, December 12-14, 1988, San Diego, California, United States [doi>10.1145/318123.318234]
• 230: Bernard P. Zeigler, Hierarchical, modular discrete-event modelling in an object-oriented environment, Simulation, v.49 n.5, p.219-230, November 1, 1987 [doi>10.1177/003754978704900506]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Simulation; Stochastic analysis; Queueing theory

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing; Statistical software

General Terms:
Algorithms, Design, Experimentation, Performance, Theory