Novelty and redundancy detection in adaptive filtering

ABSTRACT

This paper addresses the problem of extending an adaptive information filtering system to make decisions about the novelty and redundancy of relevant documents. It argues that relevance and redundance should each be modelled explicitly and separately. A set of five redundancy measures are proposed and evaluated in experiments with and without redundancy thresholds. The experimental results demonstrate that the cosine similarity metric and a redundancy measure based on a mixture of language models are both effective for identifying redundant documents.

REFERENCES

• 1: J. Allan, J. Carbonell, G. Doddington, J. Yamron, and Y. Yang. Topic detection and tracking pilot study. In Topic Detection and Tracking Workshop Report 2001.
• 2: James Allan , Victor Lavrenko , Hubert Jin, First story detection in TDT is hard, Proceedings of the ninth international conference on Information and knowledge management, p.374-381, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354843]
• 3: James Allan , Ron Papka , Victor Lavrenko, On-line new event detection and tracking, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.37-45, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290954]
• 4: J. Carbonell, Y. Yang, R. Brown, C. Jin, and J. Zhang. CMU TDT report 13-- 14 Nov 2001. In Topic Detection and Tracking Workshop Report 2001.
• 5: M. Franz, A. Ittycheriah, J. S. McCarley, andT. Ward. First story detection: Combining similarity and novelty based approaches. In Topic Detection and Tracking Workshop Report 2001.
• 6: W. P. Jones and G. W. Furnas. Pictures of relevance. Journal of the American Society for Information Science 1987.
• 7: W. Kraaij, R. Pohlmann, and D. Hiemstra. Twenty- one at TREC- 8: using language technology for information retrieval. In Proceedings of the Eighth Text REtrieval Conference (TREC- 8) , 1999.
• 8: Lillian Lee, Measures of distributional similarity, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.25-32, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034693]
• 9: Andrew McCallum , Ronald Rosenfeld , Tom M. Mitchell , Andrew Y. Ng, Improving Text Classification by Shrinkage in a Hierarchy of Classes, Proceedings of the Fifteenth International Conference on Machine Learning, p.359-367, July 24-27, 1998
• 10: David R. H. Miller , Tim Leek , Richard M. Schwartz, A hidden Markov model information retrieval system, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.214-221, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312680]
• 11: S. Robertson. Threshold setting in adaptive filtering. Journal of Documentation 2000.
• 12: S. Robertson and D. Hull. The TREC- 9 Filtering track report. In The Ninth Text REtrieval Conference (TREC- 9) , 2001.
• 13: M. Spitters and W. Kraaij. TNO at TDT2001: Language model-based topic detection. In Topic Detection and Tracking Workshop Report. 2001.
• 14: Nicola Stokes , Joe Carthy, Combining semantic and syntactic document classifiers to improve first story detection, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.424-425, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384068]
• 15: J. Yamron, S. Knecht, and P. van Mulbregt. Dragon's tracking and detection systems for the TDT2000 evaluation. In Proceedings of the Broadcast News Transcription and Understanding Workshop 1998.
• 16: Chengxiang Zhai , John Lafferty, Model-based feedback in the language modeling approach to information retrieval, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502654]
• 17: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]
• 18: Yi Zhang , Jamie Callan, Maximum likelihood estimation for filtering thresholds, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.294-302, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384012]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Algorithms, Design, Experimentation