Adaptation of offline vertical selection predictions in the presence of user feedback

ABSTRACT

Web search results often integrate content from specialized corpora known as verticals. Given a query, one important aspect of aggregated search is the selection of relevant verticals from a set of candidate verticals. One drawback to previous approaches to vertical selection is that methods have not explicitly modeled user feedback. However, production search systems often record a variety of feedback information. In this paper, we present algorithms for vertical selection which adapt to user feedback. We evaluate algorithms using a novel simulator which models performance of a vertical selector situated in realistic query traffic.

REFERENCES

• 1: S. Acharya, P. Krishnamurthy, K. Deshpande, T. Yan, and C.-C. Chang. A simulation framework for evaluating designs for sponsored search markets. In WWW 2007 Workshop on Sponsored Search Auctions, 2007.
• 2: J. Aitchison and S.M. Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261--272, August 1980.
• 3: Jaime Arguello , Fernando Diaz , Jamie Callan , Jean-Francois Crespo, Sources of evidence for vertical selection, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA [doi>10.1145/1571941.1571997]
• 4: Leif Azzopardi , Maarten de Rijke , Krisztian Balog, Building simulated queries for known-item topics: an analysis using six european languages, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277820]
• 5: Steven M. Beitzel , Eric C. Jensen , Ophir Frieder , David D. Lewis , Abdur Chowdhury , Aleksander Kolcz, Improving Automatic Query Classification via Semi-Supervised Learning, Proceedings of the Fifth IEEE International Conference on Data Mining, p.42-49, November 27-30, 2005 [doi>10.1109/ICDM.2005.80]
• 6: J. Callan. Distributed information retrieval. In W.B. Croft, editor, Advances in Information Retrieval. 2000.
• 7: Massimiliano Ciaramita , Vanessa Murdock , Vassilis Plachouras, Online learning from click data for sponsored search, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367529]
• 8: M.D. Cooper. A simulation model of an information retrieval system. Information Storage and Retrieval, 9(1):13--32, 1973.
• 9: Steve Cronen-Townsend , Yun Zhou , W. Bruce Croft, Predicting query performance, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564429]
• 10: Fernando Diaz, Integration of news content into web results, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain [doi>10.1145/1498759.1498825]
• 11: J.-M. Griffiths. The computer simulation of information retrieval systems. PhD thesis, University College London, 1977.
• 12: Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
• 13: Rosie Jones , Kristina Lisa Klinkner, Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA [doi>10.1145/1458082.1458176]
• 14: Peter J. Lenk , Barry D. Floyd, Dynamically updating relevance judgements in probabilistic information systems via users' feedback, Management Science, v.34 n.12, p.1450-1459, Dec. 1988 [doi>10.1287/mnsc.34.12.1450]
• 15: Xiao Li , Ye-Yi Wang , Alex Acero, Learning query intent from regularized click graphs, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390393]
• 16: D. Metzler, S.T. Dumais, and C. Meek. Similarity measures for short segments of text. In ECIR 2007, pages 16--27, 2007.
• 17: Javed Mostafa , Snehasis Mukhopadhyay , Mathew Palakal, Simulation Studies of Different Dimensions of Users' Interests and their Impact on User Modeling and Information Filtering, Information Retrieval, v.6 n.2, p.199-223, April 2003 [doi>10.1023/A:1023932221048]
• 18: V. Murdock and M. Lalmas, editors. Proceedings of the SIGIR Workshop on Aggregated Search, 2008.
• 19: Filip Radlinski , Robert Kleinberg , Thorsten Joachims, Learning diverse rankings with multi-armed bandits, Proceedings of the 25th international conference on Machine learning, p.784-791, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390255]
• 20: Filip Radlinski , Madhu Kurup , Thorsten Joachims, How does clickthrough data reflect retrieval quality?, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA [doi>10.1145/1458082.1458092]
• 21: Matthew Richardson , Ewa Dominowska , Robert Ragno, Predicting clicks: estimating the click-through rate for new ads, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242643]
• 22: Mehran Sahami , Timothy D. Heilman, A web-based kernel function for measuring the similarity of short text snippets, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135834]
• 23: Dou Shen , Jian-Tao Sun , Qiang Yang , Zheng Chen, Building bridges for web query classification, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148196]
• 24: R. Sutton and A. Barto. Reinforcement Learning. 1998.
• 25: Jean Tague , Michael Nelson , Harry Wu, Problems in the simulation of bibliographic retrieval systems, Proceedings of the 3rd annual ACM conference on Research and development in information retrieval, p.236-255, June 23-27, 1980, Cambridge, England
• 26: Ryen W. White , Ian Ruthven , Joemon M. Jose , C. J. Van Rijsbergen, Evaluating implicit feedback models using searcher simulations, ACM Transactions on Information Systems (TOIS), v.23 n.3, p.325-361, July 2005 [doi>10.1145/1080343.1080347]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms

Keywords:
aggregated search, distributed information retrieval, resource selection, simulation, user feedback, vertical selection