Expert agreement and content based reranking in a meta search environment using Mearf

ABSTRACT

Recent increase in the number of search engines on the Web and the availability of meta search engines that can query multiple search engines makes it important to find effective methods for combining results coming from different sources. In this paper we introduce novel methods for reranking in a meta search environment based on expert agreement and contents of the snippets. We also introduce an objective way of evaluating different methods for ranking search results that is based upon implicit user judgements. We incorporated our methods and two variations of commonly used merging methods in our meta search engine, Mearf, and carried out an experimental study using logs accumulated over a period of twelve months. Our experiments show that the choice of the method used for merging the output produced by different search engines plays a significant role in the overall quality of the search results. In almost all cases examined, results produced by some of the new methods introduced were consistently better than the ones produced by traditional methods commonly used in various meta search engines. These observations suggest that the proposed methods can offer a relatively inexpensive way of improving the meta search experience over existing methods.

REFERENCES

• 1: Brian T. Bartell , Garrison W. Cottrell , Richard K. Belew, Automatic combination of multiple ranked retrieval systems, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.173-181, July 03-06, 1994, Dublin, Ireland
• 2: C4.com. http://www.c4.com/.
• 3: James P. Callan , Zhihong Lu , W. Bruce Croft, Searching distributed collections with inference networks, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-28, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215328]
• 4: Daniel Dreilinger , Adele E. Howe, Experiences with selecting search engines using metasearch, ACM Transactions on Information Systems (TOIS), v.15 n.3, p.195-222, July 1997 [doi>10.1145/256163.256164]
• 5: James C. French , Allison L. Powell, Metrics for Evaluating Database Selection Techniques, Proceedings of the 10th International Workshop on Database & Expert Systems Applications, p.726, September 01-03, 1999
• 6: James C. French , Allison L. Powell , Jamie Callan , Charles L. Viles , Travis Emmitt , Kevin J. Prey , Yun Mou, Comparing the performance of database selection algorithms, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.238-245, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312684]
• 7: Eric Jay Glover , William P. Birmingham , Michael D. Gordon, Using extra-topical user preferences to improve web-based metasearch, 2001
• 8: Luis Gravano , Héctor García-Molina , Anthony Tomasic, The effectiveness of GIOSS for the text database discovery problem, ACM SIGMOD Record, v.23 n.2, p.126-137, June 1994
• 9: Adele E. Howe and Daniel Dreilinger. SAVVYSEARCH: A metasearch engine that learns which search engines to query. AI Magazine, 18(2):19--25, 1997.
• 10: Inquirus. http://www.inquirus.com/.
• 11: Panagiotis Ipeirotis, Luis Gravano, and Mehran Sahami. Automatic classification of text databases through query probing. Technical Report CUCS-004-00, Computer Science Department, Columbia University, March 2000.
• 12: Ixquick. http://www.ixquick.com/.
• 13: David D. Lewis, Evaluating and optimizing autonomous text classification systems, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.246-254, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215366]
• 14: Longzhuang Li and Li Shang. Statistical performance evaluation of search engines. In WWW10 conference posters, May 2--5, 2001, Hong Kong.
• 15: Mamma. http://www.mamma.com/.
• 16: M. Catherine McCabe , Abdur Chowdhury , David A. Grossman , Ophir Frieder, A unified environment for fusion of information retrieval approaches, Proceedings of the eighth international conference on Information and knowledge management, p.330-334, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320024]
• 17: Metacrawler. http://www.metacrawler.com/.
• 18: Profusion. http://www.profusion.com/.
• 19: Erik Warren Selberg , Oren Etzioni, Towards comprehensive web search, 1999
• 20: E. Selberg and O. Etzioni. Multi-service search and comparison using the MetaCrawler. In Proceedings of the 4th International World-Wide Web Conference, Darmstadt, Germany, December 1995.
• 21: E. Selberg and O. Etzioni. The MetaCrawler architecture for resource aggregation on the Web. IEEE Expert, (January--February):11--14, 1997.
• 22: Joseph A. Shaw and Edward A. Fox. Combination of multiple searches. In Third Text REtrieval Conference, 1994.
• 23: Mario Gomez Susan Gauch, Guijun Wang. Profusion: Intelligent fusion from multiple, distributed search engines. Journal of Universal Computer Science, 2(9):637--649, 1996.
• 24: Christopher C. Vogt , Garrison W. Cottrell, Fusion Via a Linear Combination of Scores, Information Retrieval, v.1 n.3, p.151-173, October 1999 [doi>10.1023/A:1009980820262]
• 25: Zonghuan Wu , Weiyi Meng , Clement Yu , Zhuogang Li, Towards a highly-scalable and effective metasearch engine, Proceedings of the 10th international conference on World Wide Web, p.386-395, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372093]
• 26: Clement Yu , Weiyi Meng , King-Lup Liu , Wensheng Wu , Naphtali Rishe, Efficient and effective metasearch for a large number of text databases, Proceedings of the eighth international conference on Information and knowledge management, p.217-224, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320005]

INDEX TERMS

General Terms:
Algorithms, Experimentation, Performance

Keywords:
collection fusion, expert agreement, merging, meta search, reranking