Why do successful search systems fail for some topics

ABSTRACT

This paper describes and evaluates the vector-space and probabilistic IR models used to retrieve news articles from a corpus written in the French language. Based on three CLEF test-collections and 151 queries, we classify the poor retrieval results of difficult topics under 6 categories. The explanations we obtain from this analysis differ from those suggested a priori by our students. We use the Web to manually or automatically find related search terms to the original query. We evaluate these two query expansion strategies in order to improve mean average precision (MAP) and to reduce the number of topics for which no pertinent responses are listed among the top ten references returned.

REFERENCES

• 1: S. E. Robertson , S. Walker , M. Beaulieu, Experimentation as a way of life: Okapi at TREC, Information Processing and Management: an International Journal, v.36 n.1, p.95-108, Jan. 2000 [doi>10.1016/S0306-4573(99)00046-1]
• 2: Hiemstra, D. Using language models for information retrieval. CTIT Ph.D. Thesis, 2000.
• 3: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to information retrieval, ACM Transactions on Information Systems (TOIS), v.22 n.2, p.179-214, April 2004 [doi>10.1145/984321.984322]
• 4: Gianni Amati , Cornelis Joost Van Rijsbergen, Probabilistic models of information retrieval based on measuring the divergence from randomness, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.357-389, October 2002 [doi>10.1145/582415.582416]
• 5: Peters, C., Braschler, M., Gonzalo, J., and Kluck, M. (Eds). Evaluation of cross-language information retrieval. LNCS #2406, Springer, Berlin, 2002.
• 6: Peters, C., Braschler, M., Gonzalo, J., and Kluck, M. (Eds). Advances in cross-language information retrieval. LNCS #2785, Springer, Berlin, 2003.
• 7: Carol Peters , Julio Gonzalo , Martin Braschler , Michael Kluck, Comparative Evaluation of Multilingual Information Access Systems: 4th Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August ... Papers (Lecture Notes in Computer Science), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 8: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 9: Buckley, C., Singhal, A., Mitra, M., and Salton, G. New retrieval approaches using SMART. In Proceedings of TREC-4. Gaithersburg, MA, 1996, 25--48.
• 10: Singhal, A., Choi, J., Hindle, D., Lewis, D. D. and Pereira, F. (1999). AT&T at TREC-7. In Proceedings TREC-7, Gaithersburg, MA, 1999, 239--251.
• 11: Moulinier, I. Thomson Legal and Regulatory at NTCIR-4: Monolingual and pivot-language retrieval experiments. In Proceedings NTCIR-4, Tokyo, 2004, 158--165.
• 12: Robert Krovetz, Viewing morphology as an inference process, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.191-202, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160718]
• 13: Chris Buckley, Why current IR engines fail, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009132]
• 14: Voorhees, E. M. Overview of the TREC 2004 robust retrieval track. In Proceedings TREC-2004. Gaithersburg, MA, 2004, 70--79.
• 15: Ellen M. Voorhees, The TREC 2005 robust track, ACM SIGIR Forum, v.40 n.1, June 2006 [doi>10.1145/1147197.1147205]
• 16: Kwok, K. L., Grunfeld, L., Sun, H. L., and Deng, P. TREC2004 robust track experiments using PIRCS. In Proceedings TREC-2004. Gaithersburg, MA, 2004.
• 17: Kwok, K. L., Grunfeld L., Dinstl, N., and Chan, M. TREC-9 cross-language, web and question-answering track experiments using PIRCS. In Proceedings TREC-9, Gaithersburg, MA, 2001, 417--426

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Measurement, Performance

Keywords:
failure analysis, robust evaluation