Tuning before feedback: combining ranking discovery and blind feedback for robust retrieval

ABSTRACT

Both ranking functions and user queries are very important factors affecting a search engine's performance. Prior research has looked at how to improve ad-hoc retrieval performance for existing queries while tuning the ranking function, or modify and expand user queries using a fixed ranking scheme using blind feedback. However, almost no research has looked at how to combine ranking function tuning and blind feedback together to improve ad-hoc retrieval performance. In this paper, we look at the performance improvement for ad-hoc retrieval from a more integrated point of view by combining the merits of both techniques. In particular, we argue that the ranking function should be tuned first, using user-provided queries, before applying the blind feedback technique. The intuition is that highly-tuned ranking offers more high quality documents at the top of the hit list, thus offers a stronger baseline for blind feedback. We verify this integrated model in a large scale heterogeneous collection and the experimental results show that combining ranking function tuning and blind feedback can improve search performance by almost 30% over the baseline Okapi system.

REFERENCES

• 1: Wolfgang Banzhaf , Frank D. Francone , Robert E. Keller , Peter Nordin, Genetic programming: an introduction: on the automatic evolution of computer programs and its applications, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 2: Brian T. Bartell , Garrison W. Cottrell , Richard K. Belew, Automatic combination of multiple ranked retrieval systems, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.173-181, July 03-06, 1994, Dublin, Ireland
• 3: Claudio Carpineto , Renato de Mori , Giovanni Romano , Brigitte Bigi, An information-theoretic approach to automatic query expansion, ACM Transactions on Information Systems (TOIS), v.19 n.1, p.1-27, Jan. 2001 [doi>10.1145/366836.366860]
• 4: Claudio Carpineto , Giovanni Romano , Vittorio Giannini, Improving retrieval feedback with multiple term-ranking function combination, ACM Transactions on Information Systems (TOIS), v.20 n.3, p.259-290, July 2002 [doi>10.1145/568727.568728]
• 5: Young Mee Chung , Jae Yun Lee, Optimization of some factors affecting the performance of query expansion, Information Processing and Management: an International Journal, v.40 n.6, p.891-917, November 2004 [doi>10.1016/j.ipm.2003.11.003]
• 6: Fan, W., Gordon, M. D. and Pathak, P., Effective profiling of consumer information retrieval needs: a unified framework and empirical comparison, Decision Support Systems, in press, 2004.
• 7: Weiguo Fan , Michael D. Gordon , Praveen Pathak , Wensi Xi , Edward A. Fox, Ranking Function Optimization for Effective Web Search by Genetic Programming: An Empirical Study, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 4, p.40105, January 05-08, 2004
• 8: Weiguo Fan , Edward A. Fox , Praveen Pathak , Harris Wu, The effects of fitness functions on genetic programming-based ranking discovery for Web search: Research Articles, Journal of the American Society for Information Science and Technology, v.55 n.7, p.628-636, May 2004 [doi>10.1002/asi.v55:7]
• 9: Weiguo Fan , Michael D. Gordon , Praveen Pathak, A generic ranking function discovery framework by genetic programming for information retrieval, Information Processing and Management: an International Journal, v.40 n.4, p.587-602, May 2004 [doi>10.1016/j.ipm.2003.08.001]
• 10: Weiguo Fan , Michael D. Gordon , Praveen Pathak, Discovery of Context-Specific Ranking Functions for Effective Information Retrieval Using Genetic Programming, IEEE Transactions on Knowledge and Data Engineering, v.16 n.4, p.523-527, April 2004 [doi>10.1109/TKDE.2004.1269663]
• 11: Norbert Fuhr , Chris Buckley, A probabilistic learning approach for document indexing, ACM Transactions on Information Systems (TOIS), v.9 n.3, p.223-248, July 1991 [doi>10.1145/125187.125189]
• 12: Norbert Fuhr , Ulrich Pfeifer, Probabilistic information retrieval as a combination of abstraction, inductive learning, and probabilistic assumptions, ACM Transactions on Information Systems (TOIS), v.12 n.1, p.92-115, Jan. 1994 [doi>10.1145/174608.174612]
• 13: Fredric C. Gey, Inferring probability of relevance using the method of logistic regression, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.222-231, July 03-06, 1994, Dublin, Ireland
• 14: M. Gordon, Probabilistic and genetic algorithms in document retrieval, Communications of the ACM, v.31 n.10, p.1208-1218, Oct. 1988 [doi>10.1145/63039.63044]
• 15: Donna Harman, Relevance feedback revisited, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.1-10, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133167]
• 16: John H. Holland, Adaptation in natural and artificial systems, MIT Press, Cambridge, MA, 1992
• 17: Ide, E., New experiments in relevance feedback, in Gerard Salton, Editor, The SMART Retrieval System, Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1971.
• 18: M. Shamim Khan , Sebastian Khor, Enhanced web document retrieval using automatic query expansion, Journal of the American Society for Information Science and Technology, v.55 n.1, p.29-40, 1 January 2004 [doi>10.1002/asi.10341]
• 19: John R. Koza, Genetic programming: on the programming of computers by means of natural selection, MIT Press, Cambridge, MA, 1992
• 20: Foundations of genetic programming, Springer-Verlag New York, Inc., New York, NY, 2002
• 21: Joon Ho Lee, Analyses of multiple evidence combination, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.267-276, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 22: Carol Lundquist , David A. Grossman , Ophir Frieder, Improving relevance feedback in the vector space model, Proceedings of the sixth international conference on Information and knowledge management, p.16-23, November 10-14, 1997, Las Vegas, Nevada, United States [doi>10.1145/266714.266844]
• 23: Pathak, P., Gordon, M. D., and W. Fan, Effective information retrieval using genetic algorithms based matching function adaptation, in Proceedings of the 33rd Hawaii International Conference on System Science (HICSS), Hawaii, USA, 2000.
• 24: S. E. Robertson, On term selection for query expansion, Journal of Documentation, v.46 n.4, p.359-364, Dec. 1990 [doi>10.1108/eb026866]
• 25: Robertson, S. E., Walker, S., Jones S., Hancock-Beaulieu M. M., and Gatford, M., Okapi at TREC-4, in D. K. Harman, editor, Proceedings of the Fourth Text Retrieval Conference, 73--97. NIST Special Publication 500--236, 1996.
• 26: Rocchio, Jr., J.J., Relevance feedback in information retrieval, in Gerard Salton, Editor, The SMART Retrieval System, Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1971.
• 27: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 28: Salton, G. and Buckley, C., Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4), 288--297, 1990.
• 29: Christopher C. Vogt , Garrison W. Cottrell, Fusion Via a Linear Combination of Scores, Information Retrieval, v.1 n.3, p.151-173, October 1999 [doi>10.1023/A:1009980820262]
• 30: Justin Zobel , Alistair Moffat, Exploring the similarity space, ACM SIGIR Forum, v.32 n.1, p.18-34, Spring 1998 [doi>10.1145/281250.281256]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Retrieval models; Relevance feedback; Selection process

General Terms:
Algorithms, Experimentation, Measurement, Performance

Keywords:
blind feedback, genetic programming, information retrieval, intelligent information retrieval, query expansion, ranking function, search engine