Generalized inverse document frequency

ABSTRACT

Inverse document frequency (IDF) is one of the most useful and widely used concepts in information retrieval. There have been various attempts to provide theoretical justifications for IDF. One of the most appealing derivations follows from the Robertson-Sparck Jones relevance weight. However, this derivation, and others related to it, typically make a number of strong assumptions that are often glossed over. In this paper, we re-examine these assumptions from a Bayesian perspective, discuss possible alternatives, and derive a new, more generalized form of IDF that we call generalized inverse document frequency. In addition to providing theoretical insights into IDF, we also undertake a rigorous empirical evaluation that shows generalized IDF outperforms classical versions of IDF on a number of ad hoc retrieval tasks.

REFERENCES

• 1: Akiko Aizawa, An information-theoretic perspective of tf—idf measures, Information Processing and Management: an International Journal, v.39 n.1, p.45-65, January 2003 [doi>10.1016/S0306-4573(02)00021-3]
• 2: Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling for large collections, Information Retrieval, v.10 n.6, p.491-508, December 2007 [doi>10.1007/s10791-007-9032-x]
• 3: William S. Cooper, Some inconsistencies and misnomers in probabilistic information retrieval, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.57-61, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122866]
• 4: W. B. Croft and D. Harper. Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35:285--295, 1979.
• 5: Arjen P. de Vries , Thomas Roelleke, Relevance information: a loss of entropy but a gain for IDF?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076084]
• 6: Warren R. Greiff, A theory of term weighting based on exploratory data analysis, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.11-19, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290948]
• 7: S. P. Harter. A probabilistic approach to automatic keyword indexing. Journal of the American Society for Information Science, 26:197--206 and 280--289, 1975.
• 8: Ben He , Iadh Ounis, On setting the hyper-parameters of term frequency normalization for information retrieval, ACM Transactions on Information Systems (TOIS), v.25 n.3, p.13-es, July 2007 [doi>10.1145/1247715.1247719]
• 9: K. S. Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11--21, 1972.
• 10: J. Lafferty and C. Zhai. Probabilistic relevance models based on document and query generation. In W. B. Croft and J. Lafferty, editors, Language Modeling and Information Retrieval. 2003.
• 11: Victor Lavrenko , W. Bruce Croft , James Allan, A generative theory of relevance, University of Massachusetts Amherst, 2004
• 12: Lillian Lee, IDF revisited: a simple new derivation within the Robertson-Spärck Jones probabilistic model, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277891]
• 13: Kishore Papineni, Why inverse document frequency?, Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001, p.1-8, June 01-07, 2001, Pittsburgh, Pennsylvania [doi>10.3115/1073336.1073340]
• 14: S. Robertson. The probability ranking principle in IR. Journal of Documentation, 33(4):294--304, 1977.
• 15: S. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In Proc. 3rd Text REtrieval Conference, pages 109--126, 1994.
• 16: S. E. Robertson and K. S. Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27(3):129--146, 1976.
• 17: S. E. Robertson , C. J. van Rijsbergen , M. F. Porter, Probabilistic models of indexing and searching, Proceedings of the 3rd annual ACM conference on Research and development in information retrieval, p.35-56, June 23-27, 1980, Cambridge, England
• 18: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 19: S. E. Robertson , S. Walker, On relevance weights with little relevance information, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.16-24, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 20: Thomas Roelleke, A frequency-based and a poisson-based definition of the probability of being informative, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860478]
• 21: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 22: T. Strohman, D. Metzler, H. Turtle, and W. B. Croft. Indri: A language model-based serach engine for complex queries. In Proceedings of the International Conference on Intelligence Analysis, 2004.
• 23: Zuobing Xu , Ram Akella, A new probabilistic retrieval model based on the dirichlet compound multinomial distribution, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390408]
• 24: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to information retrieval, ACM Transactions on Information Systems (TOIS), v.22 n.2, p.179-214, April 2004 [doi>10.1145/984321.984322]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Theory

Keywords:
estimation, formal models, inverse document frequency