Automatic document prior feature selection for web retrieval

ABSTRACT

Document prior features, such as Pagerank and URL depth, can improve the retrieval effectiveness of Web Information Retrieval (IR) systems. However, not all queries equally benefit from the application of a document prior feature. This paper aims to investigate whether the retrieval performance can be further enhanced by selecting the best document prior feature on a per-query basis. We present a novel method for selecting the best document prior feature on a per-query basis. We evaluate our technique on the TREC .GOV Web test collection and its associated TREC 2003 Web search tasks. Our experiments demonstrate the effectiveness and robustness of our proposed selection method.

REFERENCES

• 1: J. Kamps, G. Mishne, M. de Rijke. Language Models for Searching in Web Corpora. In Proceedings of TREC 2004.
• 2: Wessel Kraaij , Thijs Westerveld , Djoerd Hiemstra, The Importance of Prior Probabilities for Entry Page Search, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564383]
• 3: S. Kullback. Information Theory and Statistics. Jown Wiley & Sons, New York, USA, 1959.
• 4: J. Lin. Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information Theory, 37:145--151, 1991.
• 5: C. Macdonald, V. Plachouras, B. He, C. Lioma, I. Ounis. University of Glasgow at WebCLEF 2005: Experiments in per-field normalisation and language specific stemming. In Proceedings of CLEF 2005.
• 6: R. Manmatha , T. Rath , F. Feng, Modeling score distributions for combining the outputs of search engines, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.267-275, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384005]
• 7: V. Plachouras. Selective Web Information Retrieval. Phd Thesis. Univ of Glasgow, 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Performance

Keywords:
prior feature selection