Utility-based information distillation over temporally sequenced documents

ABSTRACT

This paper examines a new approach to information distillation over temporally ordered documents, and proposes a novel evaluation scheme for such a framework. It combines the strengths of and extends beyond conventional adaptive filtering, novelty detection and non-redundant passage ranking with respect to long-lasting information needs ("tasks" with multiple queries). Our approach supports fine-grained user feedback via highlighting of arbitrary spans of text, and leverages such information for utility optimization in adaptive settings. For our experiments, we defined hypothetical tasks based on news events in the TDT4 corpus, with multiple queries per task. Answer keys (nuggets) were generated for each query and a semi-automatic procedure was used for acquiring rules that allow automatically matching nuggets against system responses. We also propose an extension of the NDCG metric for assessing the utility of ranked passages as a combination of relevance and novelty. Our results show encouraging utility enhancements using the new approach, compared to the baseline systems without incremental learning or the novelty detection components.

REFERENCES

• 1: James Allan, Incremental relevance feedback for information filtering, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.270-278, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243274]
• 2: James Allan , Courtney Wade , Alvaro Bolivar, Retrieval and novelty detection at the sentence level, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860493]
• 3: C. Buckley, G. Salton, and J. Allan. Automatic Retrieval with Locality Information using SMART. NIST special publication, (500207):59--72, 1993.
• 4: Jamie Callan, Learning while filtering documents, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.224-231, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290998]
• 5: Jaime Carbonell , Jade Goldstein, The use of MMR, diversity-based reranking for reordering documents and producing summaries, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.335-336, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291025]
• 6: E. Efthimiadis. Query Expansion. Annual Review of Information Science and Technology (ARIST), 31:p 121--87, 1996.
• 7: Jonathan G. Fiscus , George R. Doddington, Topic detection and tracking evaluation overview, Topic detection and tracking: event-based information organization, Kluwer Academic Publishers, Norwell, MA, 2002
• 8: R. Florian, H. Hassan, A. Ittycheriah, H. Jing, N. Kambhatla, X. Luo, N. Nicolov, and S. Roukos. A Statistical Model for Multilingual Entity Detection and Tracking. NAACL/HLT, 2004.
• 9: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 10: Jimmy Lin , Dina Demner-Fushman, Automatically evaluating answers to definition questions, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.931-938, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220692]
• 11: Jimmy Lin , Dina Demner-Fushman, Will pyramids built of nuggets topple over?, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.383-390, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220884]
• 12: Xiaoqiang Luo , Abe Ittycheriah , Hongyan Jing , Nanda Kambhatla , Salim Roukos, A mention-synchronous coreference resolution algorithm based on the Bell tree, Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, p.135-es, July 21-26, 2004, Barcelona, Spain [doi>10.3115/1218955.1218973]
• 13: Gregory Marton , Alexey Radul, Nuggeteer: automatic nugget-based evaluation using descriptions and judgements, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.375-382, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220883]
• 14: E. Riloff. Automatically Constructing a Dictionary for Information Extraction Tasks. Proceedings of the Eleventh National Conference on Artificial Intelligence, pages 811--816, 1993.
• 15: S. Robertson and S. Walker. Microsoft Cambridge at TREC-9: Filtering track. The Ninth Text REtrieval Conference (TREC-9), pages 361--368.
• 16: Robert E. Schapire , Yoram Singer , Amit Singhal, Boosting and Rocchio applied to text filtering, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.215-223, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290996]
• 17: T. Strohman, D. Metzler, H. Turtle, and W. Croft. Indri: A Language Model-based Search Engine for Complex Queries. Proceedings of the International Conference on Intelligence Analysis, 2004.
• 18: The Linguistic Data Consortium. http://www.ldc.upenn.edu/.
• 19: E. Voorhees. Overview of the TREC 2003 Question Answering Track. Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), 2003.
• 20: Yiming Yang , Bryan Kisiel, Margin-based local regression for adaptive filtering, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956902]
• 21: Yiming Yang , Shinjae Yoo , Jian Zhang , Bryan Kisiel, Robustness of adaptive filtering methods in a cross-benchmark evaluation, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076054]
• 22: Cheng Xiang Zhai , William W. Cohen , John Lafferty, Beyond independent relevance: methods and evaluation metrics for subtopic retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860440]
• 23: Jian Zhang , Yiming Yang, Robustness of regularized linear classification methods in text categorization, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860471]
• 24: Yi Zhang, Using bayesian priors to combine classifiers for adaptive filtering, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009052]
• 25: Yi Zhang , Jamie Callan , Thomas Minka, Novelty and redundancy detection in adaptive filtering, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564393]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models; Relevance feedback; Selection process

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

General Terms:
Design, Experimentation, Measurement, Performance

Keywords:
adaptive filtering, evaluation methodology, flexible user feedback, novelty detection, passage ranking, unified framework, utility-based distillation