What's the value of TREC: is there a gap to jump or a chasm to bridge?

ABSTRACT

Work within the TREC Programme has concentrated on generalising, not particularising. Now is the time to think about particularising, that is, to address not further generalisation across information-seeking contexts but context-driven particularisation. This note develops this argument from an analysis of TREC work, applying notions taken from discussions of evaluation for language and information processing in general.

REFERENCES

• 1: Karen Sparck Jones , Julia R. Galliers , J. R. Galliers, Evaluating Natural Language Processing Systems: An Analysis and Review, Springer-Verlag New York, Inc., Secaucus, NJ, 1996
• 2: Sparck Jones, K. 'Factorial summary evaluation', Workshop on Text Summarisation, ACM-SIGIR Conference 2001, 2001. (via http://www-nlpir.nist.gov/projects/duc/pubs/2001papers/cambridge2.pdf)
• 3: Ellen M. Voorhees , Donna K. Harman, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), The MIT Press, 2005

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

General Terms:
Algorithms, Design, Performance