Relevance score normalization for metasearch

ABSTRACT

Given the ranked lists of documents returned by multiple search engines in response to a given query, the problem of metasearch is to combine these lists in a way which optimizes the performance of the combination. This problem can be naturally decomposed into three subproblems: (1) normalizing the relevance scores given by the input systems, (2) estimating relevance scores for unretrieved documents, and (3) combining the newly-acquired scores for each document into one, improved score.Research on the problem of metasearch has historically concentrated on algorithms for combining (normalized) scores. In this paper, we show that the techniques used for normalizing relevance scores and estimating the relevance scores of unretrieved documents can have a significant effect on the overall performance of metasearch. We propose two new normalization/estimation techniques and demonstrate empirically that the performance of well known metasearch algorithms can be significantly improved through their use.

REFERENCES

• 1: TREC 2, Gaithersburg, MD, USA, Mar. 1994. U.S. Government Printing Office, Washington D.C.
• 2: TREC 5, Gaithersburg, MD, USA, 1997. U.S. Government Printing Office, Washington D.C.
• 3: ACM SZGZR 2001, New Orleans, Louisiana, USA, 2001. ACM Press, New York.
• 4: Javed A. Aslam , Mark Montague, Models for metasearch, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.276-284, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384007]
• 5: Brian Theodore Bartell, Optimizing ranking functions: a connectionist approach to adaptive information retrieval, University of California at San Diego, La Jolla, CA, 1994
• 6: N. Belkin, P. Kantor, C. Cool, and R. Quatrain. Combining evidence for information retrieval. In TREC 2 {I}, pages 35-43.
• 7: W. B. Croft. Combining approaches to information retrieval. In W. B. Croft, editor, Advances in Information Retrieval: Recent Research jrvm the Center for Intelligent Information Retrieval, chapter 1. Kluwer, 2000.
• 8: E. A. Fox, M. P. Koushik, J. Shaw, , R. Modlin, and D. Rao. Combining evidence from multiple searches. In TREC 1, pages 319-328, Gaithersburg, MD, USA, Mar. 1993. U.S. Government Printing Office, Washington D.C.
• 9: E. A. Fox and J. A. Shaw. Combination of multiple searches. In TREC 2 {l}, pages 243-249.
• 10: Kevin L. Fox , Ophir Frieder , Margaret M. Knepper , Eric J. Snowberg, SENTINEL: a multiple engine information retrieval and visualization system, Journal of the American Society for Information Science, v.50 n.7, p.616-625, May 15, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:7<616::AID-ASI6>3.0.CO;2-E]
• 11: David A. Hull , Jan O. Pedersen , Hinrich Schütze, Method combination for document filtering, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.279-287, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243275]
• 12: Joon Ho Lee, Analyses of multiple evidence combination, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.267-276, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 13: R. Manmatha , T. Rath , F. Feng, Modeling score distributions for combining the outputs of search engines, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.267-275, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384005]
• 14: Mark Montague , Javed A. Aslam, Metasearch consistency, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.386-387, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384030]
• 15: K. B. Ng. An Investigation of the Conditions for Effective Data Fusion in Information Retrieval. PhD thesis, School of Communication, Information, and Library Studies, Rutgers University, 1998.
• 16: K. B. Ng and P. B. Kantor. An investigation of the preconditions for effective data fusion in ir: A pilot study. In Proceedings of the 61th Annual Meeting of the American Society for Information Science, 1998.
• 17: K. B. Ng, D. Loewenstern, C. Basu, H. Hirsh, and P. B. Kantor. Data fusion of machine-learning methods for the TREC5 routing task (and other work). In TREC 5 {2}, pages 477-487.
• 18: Content-Based Multimedia Information Access (RIAO), Paris, France, Apr. 2000.
• 19: Erik Warren Selberg , Oren Etzioni, Towards comprehensive web search, 1999
• 20: J. A. Shaw and E. A. Fox. Combination of multiple searches. In TREC 3, pages 105-108, Gaithersburg, MD, USA, Apr. 1995. U.S. Government Printing Office, Washington D.C.
• 21: P. Thompson, A combination of expert opinion approach to probabilistic information retrieval, part 1: The conceptual model, Information Processing and Management: an International Journal, v.26 n.3, p.371-382, 1990 [doi>10.1016/0306-4573(90)90097-L]
• 22: Christopher Charles Vogt , Garrison W. Cottrell, Adaptive combination of evidence for information retrieval, 1999
• 23: C. C. Vogt. How much more is better? Characterizing the effects of adding more IR systems to a combination. In RIAO {18}, pages 457-475.
• 24: Christopher C. Vogt , Garrison W. Cottrell, Fusion Via a Linear Combination of Scores, Information Retrieval, v.1 n.3, p.151-173, October 1999 [doi>10.1023/A:1009980820262]
• 25: C. C. Vogt, G. W. Cottrell, R. K.Belew, and B. T. Bartell. Using relevance to train a linear mixture of experts. In TREC 5 {2}, pages 503-515.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Query processing
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

General Terms:
Algorithms, Design, Performance