Improving realism of topic tracking evaluation

ABSTRACT

Topic tracking and information filtering are models of interactive tasks, but their evaluations are generally done in a way that does not reflect likely usage. The models either force frequent judgments or disallow any at all, assume the user is always available to make a judgment, and do not allow for user fatigue. In this study we extend the evaluation framework for topic tracking to incorporate those more realistic issues. We demonstrate that tracking can be done in a realistic interactive setting with minimal impact on tracking cost and with substantial reduction in required interaction.

REFERENCES

• 1: James Allan, Relevance feedback with too much data, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.337-343, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215380]
• 2: James Allan, Topic Detection and Tracking: Event-Based Information Organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 3: J. Allan, V. Lavrenko, D. Frey, and V. Khandelwal. UMass at TDT 2000. Notebook publication for participants only, Nov. 2001.
• 4: Christopher Cieri , Stephanie Strassel , David Graff , Nii Martey , Kara Rennert , Mark Liberman, Corpora for topic detection and tracking, Topic detection and tracking: event-based information organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 5: Jonathan G. Fiscus , George R. Doddington, Topic detection and tracking evaluation overview, Topic detection and tracking: event-based information organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 6: W. Hersh and P. Over. The TREC-9 interactive track report. In Proceedings of the Text Retrieval Conference (TREC-9), pages 41--50, 2001.
• 7: G. N. Lance and W. T. Williams. A general theory of classificatory sorting strategies: 1. hierarchical systems. Computer Journal, 9:373--380, 1967.
• 8: Anton V. Leuski , James Allan, Interactive information organization: techniques and evaluation, 2001
• 9: A. Leuski and J. Allan. Evaluating a visual navigation system for a digital library. International Journal on Digital Libraries, 3(2):170--184, 2000.
• 10: David D. Lewis , William A. Gale, A sequential algorithm for training text classifiers, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, July 03-06, 1994, Dublin, Ireland
• 11: Inderjeet Mani , David House , Gary Klein , Lynette Hirschman , Therese Firmin , Beth Sundheim, The TIPSTER SUMMAC Text Summarization Evaluation, Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics, June 08-12, 1999, Bergen, Norway [doi>10.3115/977035.977047]
• 12: K. McKeown and D. Radev. Generating summaries of multiple news articles. In I. Mani and M. Maybury, editors, Advances in Automatic Text Summarization. MIT Press, Cambridge, Massachusetts, 1999.
• 13: S. Robertson and D. A. Hull. The TREC-9 filtering track final report. In Proceedings of the Text Retrieval Conference (TREC-9), pages 25--40, 2001.
• 14: S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In D. Harman and E. Voorhees, editors, Third Text REtrieval Conference (TREC-3). NIST, 1995.
• 15: Gerald Salton, Automatic text processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988
• 16: Proceedings of the TDT 2001 workshop. Notebook publication for participants only, 2001.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

Keywords:
filtering, interactive tracking, topic detection and tracking