To personalize or not to personalize: modeling queries with variation in user intent

ABSTRACT

In most previous work on personalized search algorithms, the results for all queries are personalized in the same manner. However, as we show in this paper, there is a lot of variation across queries in the benefits that can be achieved through personalization. For some queries, everyone who issues the query is looking for the same thing. For other queries, different people want very different results even though they express their need in the same way. We examine variability in user intent using both explicit relevance judgments and large-scale log analysis of user behavior patterns. While variation in user behavior is correlated with variation in explicit relevance judgments the same query, there are many other factors, such as result entropy, result quality, and task that can also affect the variation in behavior. We characterize queries using a variety of features of the query, the results returned for the query, and people's interaction history with the query. Using these features we build predictive models to identify queries that can benefit from personalization.

REFERENCES

• 1: David Carmel , Elad Yom-Tov , Adam Darlow , Dan Pelleg, What makes a query difficult?, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148238]
• 2: Chickering, D. M. (2002). The WinMine Toolkit. Technical Report MSR-TR-2002-103, Microsoft, Redmond, WA.
• 3: Paul Alexandru Chirita , Wolfgang Nejdl , Raluca Paiu , Christian Kohlschütter, Using ODP metadata to personalize search, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076067]
• 4: Steve Cronen-Townsend , Yun Zhou , W. Bruce Croft, Predicting query performance, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564429]
• 5: Zhicheng Dou , Ruihua Song , Ji-Rong Wen, A large-scale evaluation and analysis of personalized search strategies, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242651]
• 6: Raya Fidel , Michael Crandall, Users' perception of the performance of a filtering system, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.198-205, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 7: Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): 378--382.
• 8: Zhiwei Guan , Edward Cutrell, An eye tracking study of the effect of target rank on web search, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240691]
• 9: Taher H. Haveliwala, Topic-sensitive PageRank, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511513]
• 10: Kalervo Järvelin , Jaana Kekäläinen, IR evaluation methods for retrieving highly relevant documents, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.41-48, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345545]
• 11: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076063]
• 12: Karypis, G. (2002). Cluto: A clustering toolkit. http://www.cs.umn.edu/~cluto
• 13: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM), v.46 n.5, p.604-632, Sept. 1999 [doi>10.1145/324133.324140]
• 14: Uichin Lee , Zhenyu Liu , Junghoo Cho, Automatic identification of user goals in Web search, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060804]
• 15: Jure Leskovec , Susan Dumais , Eric Horvitz, Web projections: learning from contextual subgraphs of the web, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242636]
• 16: Page, L., Brin, S., Motwani, R. and Winograd, T. (1998). The pagerank citation ranking: Bringing order to the Web. Technical report, Stanford Dig. Lib. Tech. Proj.
• 17: Daniel E. Rose , Danny Levinson, Understanding user goals in web search, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988675]
• 18: Tefko Saracevic, Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance, Journal of the American Society for Information Science and Technology, v.58 n.13, p.1915-1933, November 2007 [doi>10.1002/asi.v58:13]
• 19: Selberg, E. and Etzinoi, O. (2000). On the instability of Web search engines. In Proceedings of RIAO 2000.
• 20: Xuehua Shen , Bin Tan , ChengXiang Zhai, Implicit user modeling for personalized search, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099747]
• 21: Ruihua Song , Zhenxiao Luo , Ji-Rong Wen , Yong Yu , Hsiao-Wuen Hon, Identifying ambiguous queries in web search, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242749]
• 22: Amanda Spink , Bernard J. Jansen, Web Search: Public Searching of the Web (Information Science and Knowledge Management), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 23: Jaime Teevan , Susan T. Dumais , Eric Horvitz, Personalizing search via automated analysis of interests and activities, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076111]
• 24: Teevan, J., Dumais, S. T., and Horvitz, E. (2008). Potential for Personalization. Under submission.
• 25: Ryen W. White , Steven M. Drucker, Investigating behavioral variability in web search, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242576]
• 26: Yun Zhou , W. Bruce Croft, Query performance prediction in web search environments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277835]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

General Terms:
Algorithms, Experimentation, Human Factors, Measurement, Performance, Reliability

Keywords:
clarity, personalized search, potential for personalization