Using parsimonious language models on web data

ABSTRACT

In this paper we explore the use of parsimonious language models for web retrieval. These models are smaller thus more efficient than the standard language models and are therefore well suited for large-scale web retrieval. We have conducted experiments on four TREC topic sets, and found that the parsimonious language model results in improvement of retrieval effectiveness over the standard language model for all data-sets and measures. In all cases the improvement is significant, and more substantial than in earlier experiments on newspaper/newswire data.

REFERENCES

• 1: Djoerd Hiemstra , Stephen Robertson , Hugo Zaragoza, Parsimonious language models for information retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009025]
• 2: J. Kamps. Effective smoothing for a terabyte of text. In The Fourteenth Text REtrieval Conference (TREC 2005). National Institute of Standards and Technology. NIST Special Publication, 2006.
• 3: K. Sparck-Jones, S. Robertson, D. Hiemstra, and H. Zaragoza. Language modelling and relevance. In W. Croft and J. Lafferty, editors, Language Modeling for Information Retrieval, pages 57--71. Kluwer Academic Publishers, 2003.
• 4: TREC. Text REtrieval Conference, 2008. http://trec.nist.gov/.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Measurement, Performance

Keywords:
language models, parsimonious language models, web retrieval