Robust methodologies for modeling web click distributions

ABSTRACT

Metrics such as click counts are vital to online businesses but their measurement has been problematic due to inclusion of high variance robot traffic. We posit that by applying statistical methods more rigorous than have been employed to date that we can build a robust model of thedistribution of clicks following which we can set probabilistically sound thresholds to address outliers and robots. Prior research in this domain has used inappropriate statistical methodology to model distributions and current industrial practice eschews this research for conservative ad-hoc click-level thresholds. Prevailing belief is that such distributions are scale-free power law distributions but using more rigorous statistical methods we find the best description of the data is instead provided by a scale-sensitive Zipf-Mandelbrot mixture distribution. Our results are based on ten data sets from various verticals in the Yahoo domain. Since mixture models can overfit the data we take care to use the BIC log-likelihood method which penalizes overly complex models. Using a mixture model in the web activity domain makes sense because there are likely multiple classes of users. In particular, we have noticed that there is a significantly large set of "users" that visit the Yahoo portal exactly once a day. We surmise these may be robots testing internet connectivity by pinging the Yahoo main website.

Backing up our quantitative analysis is graphical analysis in which empirical distributions are plotted against heoretical distributions in log-log space using robust cumulative distribution plots. This methodology has two advantages: plotting in log-log space allows one to visually differentiate the various exponential distributions and secondly, cumulative plots are much more robust to outliers. We plan to use the results of this work for applications for robot removal from web metrics business intelligence systems.

REFERENCES

• 1: Ghaleb Abdulla , Edward A. Fox, Analysis and modeling of world wide web traffic, Virginia Polytechnic Institute and State University, 1998
• 2: A. Agresti. Categorical Data Analysis. Wiley Series in Probability and Statistics, 2002.
• 3: L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web caching and zipf-like distributions: Evidence and implications. In INFOCOM (1), pages 126--134, 1999.
• 4: Richard H. Byrd , Peihuang Lu , Jorge Nocedal , Ciyou Zhu, A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific Computing, v.16 n.5, p.1190-1208, Sept. 1995 [doi>10.1137/0916069]
• 5: G. Casella and R. L. Berger. Statistical Inference. Duxbury Press, 1990.
• 6: U. Frisch and D. Sornette. Extreme deviation and applications. J. Phys. I France 7, 7:1155--1171, 1997.
• 7: Steven Glassman, A caching relay for the World Wide Web, Computer Networks and ISDN Systems, v.27 n.2, p.165-173, Nov. 1994
• 8: D. C. Heilbron. Zero-altered and other regression models for count data with added zeroes. Biometrics, 36:531--547, 1994.
• 9: B. A. Huberman, P. L. T. Pirolli, J. E. Pitkow, and R. M. Lukose. Strong regularities in world wide web surfing. Science, 280:95--97, 1998.
• 10: J. Laherrere and D. Sornette. Stretched exponential distributions in nature and economy: "fat tails" with characteristic scales. The European Physical Journal B, 2:525, 1998.
• 11: Diane Lambert, Zero-inflated Poisson regression, with an application to defects in manufacturing, Technometrics, v.34 n.1, p.1-14, Feb. 1992 [doi>10.2307/1269547]
• 12: D. Lord, S. P. Washington, and J. N. Ivan. Poisson, poisson-gamma and zero-inflated regression models of motor vehicle crashes: balancing statistical fit and theory. Accident Analysis and Prevention, 37:35--46, 2005.
• 13: B. Mandelbrot. An informational theory of the statistical structure of language. In W. Jackson, editor, Communication Theory. Betterworths, 1953.
• 14: S. M. Mwalili, E. Lesaffre, and D. Declerck. The zero-inflated negative binomial regression model with correction for misclassification: An example in caries research. Technical Report TR0462, IAP Statistics Network, 2005.
• 15: R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2004. ISBN 3-900051-00-3.
• 16: J. A. Rice. Mathematical Statistics and Data Analysis. Wadsworth & Brooks/Cole, 1988.
• 17: G. Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6:461--464, 1978.
• 18: H. A. Simon. On a class of skew distribution functions. Biometrika, 42:425--440, 1955.
• 19: E. C. Titchmarsh. The Theory of the Riemann Zeta Function, 2nd ed. Oxford Science Publications, Clarendon Press, Oxford, 1986.
• 20: D. G. Uitenbroek. SISA Pairwise tests. http://home.clara.net/sisa/pairwhlp.htm, 1997.
• 21: D. von Seggern. CRC Standard Curves and Surfaces. CRC Press, 1993.
• 22: J. R. Wilson. Logarithmic series distribution and its use in analyzing discrete data. In Proceedings of the Survey Research Methods Section, American Statistical Association, pages 275--280, 1988.
• 23: G. K. Zipf. Human Behaviour and the Principle of Least-Effort. Addison-Wesley, Cambridge, MA, 1949.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Robust regression

General Terms:
Measurement, Theory

Keywords:
distribution fitting