Novelty as a form of contextual re-ranking: efficient KLD models and mixture models

ABSTRACT

Current Information Retrieval systems are often based on topicality. They estimate relevance by comparing the similarity between the user query and each document. These systems do not take into account important contextual information. More specifically, they do not often apply mechanisms to filter out redundant information. We interpret context here as the set of chunks of text from the ranked set of documents that the user has already seen. This is a valuable contextual information to guide the retrieval processes in a way that avoids redundancy. It is desirable that the ranking of results is composed by relevant but also novel material. This means that each document must provide to the user unseen information which is related to his need.

In this work we study different novelty detection approaches that make good use of this contextual information. We show that these techniques can be applied effectively and efficiently at the sentence level.

REFERENCES

• 1: James Allan , Courtney Wade , Alvaro Bolivar, Retrieval and novelty detection at the sentence level, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860493]
• 2: Yaniv Bernstein , Justin Zobel, Redundant documents and search effectiveness, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099733]
• 3: Jaime Carbonell , Jade Goldstein, The use of MMR, diversity-based reranking for reordering documents and producing summaries, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.335-336, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291025]
• 4: W. Bruce Croft , John Lafferty, Language Modeling for Information Retrieval, Kluwer Academic Publishers, Norwell, MA, 2003
• 5: R. T. Fernández. The effect of smoothing in Language Models for novelty detection. In Proceedings of the BCS IRSG Symposium: Future Directions in Information Access 2007, pages 11--16, 2007.
• 6: Ronald T. Fernández , David E. Losada, Novelty detection using local context analysis, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277878]
• 7: Donna Harman, Overview of the second text retrieval conference (TREC-2), Proceedings of the second conference on Text retrieval conference, p.271-289, May 1995, Washington, D.C., United States
• 8: Xiaoyan Li , W. Bruce Croft, Sentence level information patterns for novelty detection, University of Massachusetts Amherst, 2006
• 9: Xiaoyan Li , W. Bruce Croft, Novelty detection based on sentence level patterns, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099734]
• 10: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 11: I. Soboroff. Overview of the TREC 2004 Novelty Track. In Proceedings of the 13^th Text REtrieval Conference (TREC 2004), 2004.
• 12: I. Soboroff and D. Harman. Overview of the TREC 2003 Novelty Track. In Proceedings of the 12^th Text REtrieval Conference (TREC 2003), 2003.
• 13: Jinxi Xu , W. Bruce Croft, Improving the effectiveness of information retrieval with local context analysis, ACM Transactions on Information Systems (TOIS), v.18 n.1, p.79-112, Jan. 2000 [doi>10.1145/333135.333138]
• 14: Jinxi Xu , W. Bruce Croft, Query expansion using local and global document analysis, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.4-11, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243202]
• 15: Cheng Xiang Zhai , William W. Cohen , John Lafferty, Beyond independent relevance: methods and evaluation metrics for subtopic retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860440]
• 16: Yi Zhang , Jamie Callan , Thomas Minka, Novelty and redundancy detection in adaptive filtering, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564393]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval