Estimation of internet file-access/modification rates from indirect data

ABSTRACT

Consider an Internet file for which data on last time of access/modification (A/M) of the file are collected at periodic intervals, but for which direct A/M data are not available. Methodology is developed here that enables estimation of the A/M rates, in spite of having only indirect data of this nature. Both parametric and nonparametric methods are developed. Theoretical and empirical analyses are presented that indicate that the problem is indeed statistically tractable, and that the methods developed are of practical value. Behavior of the parametric estimators is examined when these assumptions are violated, and these estimators are found to be robust against some such violations.

REFERENCES

• 1: Barlow, R., Bartholomew, D., Bremner, J., and Brunk, H. 1972. Statistical Inference Under Order Restrictions. Wiley, New York.
• 2: Junghoo Cho , Hector Garcia-Molina, Effective page refresh policies for Web crawlers, ACM Transactions on Database Systems (TODS), v.28 n.4, p.390-426, December 2003 [doi>10.1145/958942.958945]
• 3: Junghoo Cho , Hector Garcia-Molina, Estimating frequency of change, ACM Transactions on Internet Technology (TOIT), v.3 n.3, p.256-290, August 2003 [doi>10.1145/857166.857170]
• 4: Dalgaard, P. 2002. Introductory Statistics with R. Springer-Verlag.
• 5: Paul S. Hethmon, Illustrated guide to HTTP, Manning Publications Co., Greenwich, CT, 1997
• 6: Lehmann, E. 1999. Elements of Large-Sample Theory. Springer-Verlag, New York.
• 7: Matloff, N. S. 1988. Probability Modeling and Computer Simulation. PWS-Kent Pub.Co., Boston MA.
• 8: Daniel A. Menascé , Virgílio A. F. Almeida, Capacity planning for Web performance: metrics, models, and methods, Prentice-Hall, Inc., Upper Saddle River, NJ, 1998
• 9: Meyer, M. 2001. An alternative unimodal density estimator with a consistent estimate of the mode. Statistica Sinica 11, 4, 1159--1174.
• 10: Emanuel Parzen, Stochastic processes, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1999
• 11: Rey-Bellet, L. 2003. Ergodic properties of markov processes. In Lecture notes of the 2003 Grenoble Summer School on Open Quantum Systems. Springer. To appear; available at http://www.math.umass.edu/lr7q/ps_files/gren1.pdf.
• 12: Serfling, R. 1980. Approximation Theorems of Mathematical Statistics. Wiley.
• 13: Simonoff, J. 1998. Smoothing Methods in Statistics. Springer Verlag.
• 14: Kishor S. Trivedi, Probability and statistics with reliability, queuing and computer science applications, John Wiley and Sons Ltd., Chichester, UK, 2001
• 15: van der Vaart, A. 1998. Asymptotic Statistics. Cambridge University Press.
• 16: Wegman, E. 1975. Maximum likelihood estimation of a probability density. Sankhya (A) 37, 211--224.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Stochastic processes

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Information networks

General Terms:
Measurement, Theory

Keywords:
Access rate, Poisson and renewal processes, World Wide Web, asymptotic distributions, nonparametric density estimation, statistical estimation