Revisiting the relationship between document length and relevance

ABSTRACT

The scope hypothesis in Information Retrieval (IR) states that a relationship exists between document length and relevance, such that the likelihood of relevance increases with document length. A number of empirical studies have provided statistical evidence supporting the scope hypothesis. However, these studies make the implicit assumption that modern test collections are complete (i.e. all documents are assessed for relevance). As a consequence the observed evidence is misleading. In this paper we perform a deeper analysis of document length and relevance taking into account that test collections are incomplete. We first demonstrate that previous evidence supporting the scope hypothesis was an artefact of the test collection, where there is a bias towards longer documents in the pooling process. We evaluate whether this length bias affects system comparison when using incomplete test collections. The results indicate that test collections are problematic when considering MAP as a measure of effectiveness but are relatively robust when using bpref. The implications of the study indicate that retrieval models should not be tuned to favour longer documents, and that designers of new test collections should take measures against length bias during the pooling process in order to create more reliable and robust test collections.

REFERENCES

• 1: L. Azzopardi and D. Losada. Fairly retrieving documents of all lengths: A study of document length normalization using the language modeling approach. In Proc. 1st International Conference on the Theory of Information Retrieval, ICTIR'07, pages 65--75, Budapest, October 2007.
• 2: R. Blanco and A. Barreiro. Probabilistic document length priors for language models. In Proc. ECIR-08, the 30th European Conference on Information Retrieval Research, pages 394--405, Glasgow, United Kingdom, March 2008.
• 3: Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling for large collections, Information Retrieval, v.10 n.6, p.491-508, December 2007 [doi>10.1007/s10791-007-9032-x]
• 4: Chris Buckley , Ellen M. Voorhees, Retrieval evaluation with incomplete information, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009000]
• 5: Gordon V. Cormack , Christopher R. Palmer , Charles L. A. Clarke, Efficient construction of large test collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.282-289, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291009]
• 6: D. Harman. TREC:Experiment and Evaluation in Information Retrieval, chapter The TREC AdHoc Experiments, pages 79--97. The MIT press, 2005.
• 7: S. Harter. A probabilistic approach to automatic keyword indexing. Journal of the American Society for Information Science, 26:197--206, 1975.
• 8: D. Hiemstra. A probabilistic justification for using tf x idf term weighting in information retrieval. Int. Journal of Digital Libraries, 3:131--139, 2000.
• 9: Jaap Kamps, Web-centric language models, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099640]
• 10: W. Kraaij and T. Westerveld. Tno/ut at trec-9: how different are web documents. In Proc. TREC-9, the 9th Text Retrieval Conference, Gaithersburg, United States, November 2000.
• 11: Wessel Kraaij , Thijs Westerveld , Djoerd Hiemstra, The Importance of Prior Probabilities for Entry Page Search, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564383]
• 12: David E. Losada , Leif Azzopardi, An analysis on document length retrieval trends in language modeling smoothing, Information Retrieval, v.11 n.2, p.109-138, April 2008 [doi>10.1007/s10791-007-9040-x]
• 13: M. Porter. An algorithm for suffix stripping. Program, 14(3):130--137, 1980.
• 14: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 15: S. Robertson, S. Walker, S. Jones, M. Hancock Beaulieu, and M. Gatford. Okapi at TREC-3. In D.Harman, editor, Proc. of the TREC-3, the 3rd Text Retrieval Conference, pages 109--127. NIST, 1995.
• 16: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 17: K. Sparck Jones and C. J. van Rijsbergen. Report on the need for and provision of and ideal information retrieval test collection. Technical report, British Library Research and Development Report, 1975.
• 18: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]
• 19: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]
• 20: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Performance

Keywords:
document length, evaluation, information retrieval, pooling, relevance