A frequency-based and a poisson-based definition of the probability of being informative

ABSTRACT

This paper reports on theoretical investigations about the assumptions underlying the inverse document frequency (idf). We show that an intuitive idf-based probability function for the probability of a term being informative assumes disjoint document events. By assuming documents to be independent rather than disjoint, we arrive at a Poisson-based probability of being informative. The framework is useful for understanding and deciding the parameter estimation and combination in probabilistic retrieval models.

REFERENCES

• 1: Akiko Aizawa, An information-theoretic perspective of tf—idf measures, Information Processing and Management: an International Journal, v.39 n.1, p.45-65, January 2003 [doi>10.1016/S0306-4573(02)00021-3]
• 2: Gianni Amati , C. J. van Rijsbergen, Term Frequency Normalization via Pareto Distributions, Proceedings of the 24th BCS-IRSG European Colloquium on IR Research: Advances in Information Retrieval, p.183-192, March 25-27, 2002
• 3: R. K. Belew. Finding out about. Cambridge University Press, 2000.
• 4: A. Bookstein and D. Swanson. Probabilistic models for automatic indexing. Journal of the American Society for Information Science, 25:312--318, 1974.
• 5: I. N. Bronstein. Taschenbuch der Mathematik. Harri Deutsch, Thun, Frankfurt am Main, 1987.
• 6: K. Church and W. Gale. Poisson mixtures. Natural Language Engineering, 1(2):163--190, 1995.
• 7: K. W. Church and W. A. Gale. Inverse document frequency: A measure of deviations from poisson. In Third Workshop on Very Large Corpora, ACL Anthology, 1995.
• 8: T. Lafouge and C. Michel. Links between information construction and information gain: Entropy and bibliometric distribution. Journal of Information Science, 27(1):39--49, 2001.
• 9: Eugene L. Margulis, N-Poisson document modelling, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.177-189, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133195]
• 10: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 11: S. Wong and Y. Yao. An information-theoric measure of term specificity. Journal of the American Society for Information Science, 43(1):54--61, 1992.
• 12: S. K. M. Wong , Y. Y. Yao, On modeling information retrieval with probabilistic inference, ACM Transactions on Information Systems (TOIS), v.13 n.1, p.38-68, Jan. 1995 [doi>10.1145/195705.195713]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Theory

Keywords:
independence assumption, information theory, inverse document frequency (idf), poisson distribution, probabilistic information retrieval