The TREC robust retrieval track

ABSTRACT

The robust retrieval track explores methods for improving the consistency of retrieval technology by focusing on poorly performing topics. The retrieval task in the track is a traditional ad hoc retrieval task where the evaluation methodology emphasizes a system's least effective topics. The most promising approach to improving poorly performing topics is exploiting text collections other than the target collection such as the web.The track has also investigated appropriate evaluation measures to support the focus on ineffective topics. Traditional measure are dominated by the better-performing topics, and the first two measures used in the track that do emphasize the poorly performing topics are unstable in practice. A third measure, a variant of the traditional MAP measure that uses a geometric mean rather than an arithmetic mean to average individual topic results, shows promise of giving appropriate emphasis to poorly performing topics while being more stable at equal topic set sizes.

REFERENCES

• 1: Giambattista Amati, Claudio Carpineto, and Giovanni Romano. Fondazione Ugo Bordoni at TREC 2004. In Proceedings of the Thirteenth Text REtrieval Conference. (TREC 2004), 2005.
• 2: Chris Buckley, Why current IR engines fail, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009132]
• 3: Chris Buckley and Donna Harman. Reliable information access final workshop report. ARDA Northeast Regional Research Center Technical Report, 2004.
• 4: K. L. Kwok, L. Grunfeld, H. L. Sun, and P. Deng. TREC2004 robust track experiments using PIRCS. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.
• 5: Shuang Liu, Chaojing Sun, and Clement Yu. UIC at TREC-2004: Robust track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.
• 6: Christine Piatko, James Mayfield, Paul McNamee, and Scott Cost. JHU/APL at TREC 2004: Robust and terabyte tracks. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.
• 7: Vassilis Plachouras, Ben He, and Iadh Ounis. University of Glasgow at TREC2004: Experiments in web, robust and terabyte tracks with Terrier. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.
• 8: Ellen M. Voorhees, Measuring ineffectiveness, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009121]
• 9: Ellen M. Voorhees. Overview of the TREC 2003 robust retrieval track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), pages 69--77, 2004.
• 10: Ellen M. Voorhees , Chris Buckley, The effect of topic set size on retrieval experiment error, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564432]
• 11: Jin Xu, Jun Zhao, and Bo Xu. NLPR at TREC 2004: Robust experiments. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.
• 12: Kiduk Yang, Ning Yu, Adam Wead, Gavin La Rowe, Yu-Hsiu Li, Christopher Friend, and Yoon Lee. WIDIT in TREC-2004 genomics, HARD, robust, and web tracks. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.
• 13: Elad Yom-Tov, Shai Fine, David Carmel, Adam Darlow, and Einat Amitay. Juru at TREC 2004: Experiments with prediction of query difficulty. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models
      H.3.4 Systems and Software

          Nouns: World Wide Web (WWW)

General Terms:
Algorithms, Performance