Compression-based document length prior for language models

ABSTRACT

The inclusion of document length factors has been a major topic in the development of retrieval models. We believe that current models can be further improved by more refined estimations of the document's scope. In this poster we present a new document length prior that uses the size of the compressed document. This new prior is introduced in the context of Language Modeling with Dirichlet smoothing. The evaluation performed on several collections shows significant improvements in effectiveness.

REFERENCES

• 1: R. Cilibrasi and P. Vitanyi. Clustering by compression. IEEE Trans. on Information Theory, 51:1523--1545, 2005.
• 2: David E. Losada , Leif Azzopardi, An analysis on document length retrieval trends in language modeling smoothing, Information Retrieval, v.11 n.2, p.109-138, April 2008 [doi>10.1007/s10791-007-9040-x]
• 3: Y. Marton, N. Wu, and L. Hellerstein. On compression--based text classification. In Proc. ECIR-05, 300--314, 2005.
• 4: S. Robertson, S. Walker, S. Jones, M. HancockBeaulieu, and M. Gatford. Okapi at TREC-3. In Proc. TREC-3, 109--127, 1995.
• 5: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 6: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to information retrieval, ACM Transactions on Information Systems (TOIS), v.22 n.2, p.179-214, April 2004 [doi>10.1145/984321.984322]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Performance

Keywords:
compression, document length, document priors, language models