A study of methods for negative relevance feedback

ABSTRACT

Negative relevance feedback is a special case of relevance feedback where we do not have any positive example; this often happens when the topic is difficult and the search results are poor. Although in principle any standard relevance feedback technique can be applied to negative relevance feedback, it may not perform well due to the lack of positive examples. In this paper, we conduct a systematic study of methods for negative relevance feedback. We compare a set of representative negative feedback methods, covering vector-space models and language models, as well as several special heuristics for negative feedback. Evaluating negative feedback methods requires a test set with sufficient difficult topics, but there are not many naturally difficult topics in the existing test collections. We use two sampling strategies to adapt a test collection with easy topics to evaluate negative feedback. Experiment results on several TREC collections show that language model based negative feedback methods are generally more effective than those based on vector-space models, and using multiple negative models is an effective heuristic for negative feedback. Our results also show that it is feasible to adapt test collections with easy topics for evaluating negative feedback methods through sampling.

REFERENCES

• 1: R. Attar , A. S. Fraenkel, Local Feedback in Full-Text Retrieval Systems, Journal of the ACM (JACM), v.24 n.3, p.397-417, July 1977 [doi>10.1145/322017.322021]
• 2: Chris Buckley, Why current IR engines fail, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009132]
• 3: David Carmel , Elad Yom-Tov , Adam Darlow , Dan Pelleg, What makes a query difficult?, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148238]
• 4: Mark D. Dunlop, The effect of accessing nonmatching documents on relevance feedback, ACM Transactions on Information Systems (TOIS), v.15 n.2, p.137-153, April 1997 [doi>10.1145/248625.248650]
• 5: D. A. Evans and R. G. Lefferts. Design and evaluation of the CLARIT TREC-2 system. In D. Harman, editor, TREC-2, pages 137--150, 1994.
• 6: D. Harman and C. Buckley. SIGIR 2004 Workshop: RIA and where can IR go from here. SIGIR Forum, 38(2):45--49, 2004.
• 7: Marti A. Hearst , Jan O. Pedersen, Reexamining the cluster hypothesis: scatter/gather on retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.76-84, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243216]
• 8: Makoto Iwayama, Relevance feedback with a small number of relevance judgements: incremental relevance feedback vs. document clustering, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.10-16, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345538]
• 9: John Lafferty , Chengxiang Zhai, Document language models, query models, and risk minimization for information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.111-119, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383970]
• 10: S. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129--146, 1976.
• 11: S. E. Robertson, S. Walker, S. Jones, M. M.Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In D. K. Harman, editor, The Third Text REtrieval Conference (TREC-3), pages 109--126, 1995.
• 12: J. J. Rocchio. Relevance feedback in information retrieval. In The SMART Retrieval System Experiments in Automatic Document Processing, pages 313--323, 1971.
• 13: G. Salton and C. Buckley. Improving retrieval performance by relevance feedback. JASIS, 44(4):288--297, 1990.
• 14: Xuehua Shen , Bin Tan , ChengXiang Zhai, Context-sensitive information retrieval using implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076045]
• 15: A. Singhal. Modern information retrieval: A brief overview. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 24(4):35--43, 2001.
• 16: Amit Singhal , Mandar Mitra , Chris Buckley, Learning routing queries in a query zone, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.25-32, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 17: E. M. Voorhees. Draft: Overview of the trec 2005 robust retrieval track. In Notebook of TREC2005, 2005.
• 18: E. M. Voorhees. Overview of the trec 2004 robust retrieval track. In TREC2004, 2005.
• 19: Xuanhui Wang , Hui Fang , ChengXiang Zhai, Improve retrieval accuracy for difficult queries using negative feedback, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321593]
• 20: Jinxi Xu , W. Bruce Croft, Query expansion using local and global document analysis, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.4-11, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243202]
• 21: Elad Yom-Tov , Shai Fine , David Carmel , Adam Darlow, Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076121]
• 22: Chengxiang Zhai , John Lafferty, Model-based feedback in the language modeling approach to information retrieval, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502654]
• 23: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms

Keywords:
difficult topics, language models, negative feedback, vector space models