Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search

Full text

Pdf (267 KB)

Source

ACM Transactions on Information Systems (TOIS) archive
Volume 25 , Issue 2 (April 2007) table of contents

Article No. 7

Year of Publication: 2007

ISSN:1046-8188

Authors

Thorsten Joachims	Cornell University, Ithaca, NY
Laura Granka	Google Inc., Mountain View, CA
Bing Pan	College of Charleston, Charleston, SC
Helene Hembrooke	Cornell University, Ithaca, NY
Filip Radlinski	Cornell University, Ithaca, NY
Geri Gay	Cornell University, Ithaca, NY

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 25, Downloads (12 Months): 346, Citation Count: 29

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1229179.1229181 What is a DOI?

ABSTRACT

This article examines the reliability of implicit feedback generated from clickthrough data and query reformulations in World Wide Web (WWW) search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average. We find that such relative preferences are accurate not only between results from an individual query, but across multiple sets of results within chains of query reformulations.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Eugene Agichtein , Eric Brill , Susan Dumais, Improving web search ranking by incorporating user behavior information, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148177]
	2	Eugene Agichtein , Eric Brill , Susan Dumais , Robert Ragno, Learning user interaction models for predicting web search result preferences, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148175]
	3	Rodrigo B. Almeida , Virgilio A. F. Almeida, A community-aware search engine, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988728]
	4	Brian T. Bartell , Garrison W. Cottrell , Richard K. Belew, Automatic combination of multiple ranked retrieval systems, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.173-181, July 03-06, 1994, Dublin, Ireland
	5	Belew, R. 2000. Finding Out About. Cambridge, University Press, Cambridge, U.K.
	6	Boyan, J., Freitag, D., and Joachims, T. 1996. A machine learning architecture for optimizing Web search engines. In Proceedings of the AAAI Workshop on Internet Based Information Systems. 1--8.
	7	Andrei Broder, A taxonomy of web search, ACM SIGIR Forum, v.36 n.2, Fall 2002 [doi>10.1145/792550.792552]
	8	Brumby, D. and Howes, A. 2003. Interdependence and past-experience in menu choice assessment. In Poster presented at the 25th Annual Meeting of the Cognitive Science Society.
	9	Brumby, D. and Howes, A. 2004. Good enough but I'll just check: Web-page search as attentional refocusing. In Proceedings of the International Conference on Cognitive Modeling.
	10	Chris Burges , Tal Shaked , Erin Renshaw , Ari Lazier , Matt Deeds , Nicole Hamilton , Greg Hullender, Learning to rank using gradient descent, Proceedings of the 22nd international conference on Machine learning, p.89-96, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102363]
	11	Mark Claypool , Phong Le , Makoto Wased , David Brown, Implicit interest indicators, Proceedings of the 6th international conference on Intelligent user interfaces, p.33-40, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.359836]
	12	Cohen, W., Shapire, R., and Singer, Y. 1999. Learning to order things. J. Artific. Intell. Res. 10, 243--270.
	13	Crammer, K. and Singer, Y. 2001. Pranking with ranking. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS).
	14	Steve Fox , Kuldeep Karnawat , Mark Mydland , Susan Dumais , Thomas White, Evaluating implicit measures to improve web search, ACM Transactions on Information Systems (TOIS), v.23 n.2, p.147-168, April 2005 [doi>10.1145/1059981.1059982]
	15	Yoav Freund , Raj D. Iyer , Robert E. Schapire , Yoram Singer, An Efficient Boosting Algorithm for Combining Preferences, Proceedings of the Fifteenth International Conference on Machine Learning, p.170-178, July 24-27, 1998
	16	Norbert Fuhr, Optimum polynomial retrieval functions based on the probability ranking principle, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.183-204, July 1989 [doi>10.1145/65943.65944]
	17	George W. Furnas, Experience with an adaptive indexing scheme, Proceedings of the SIGCHI conference on Human factors in computing systems, p.131-135, April 1985, San Francisco, California, United States
	18	Joseph H. Goldberg , Mark J. Stimson , Marion Lewenstein , Neil Scott , Anna M. Wichansky, Eye tracking in web search tasks: design implications, Proceedings of the 2002 symposium on Eye tracking research & applications, March 25-27, 2002, New Orleans, Louisiana [doi>10.1145/507072.507082]
	19	Laura A. Granka , Thorsten Joachims , Geri Gay, Eye-tracking analysis of user behavior in WWW search, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009079]
	20	Tim Halverson , Anthony J. Hornof, Link colors guide a search, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986066]
	21	Herbrich, R., Graepel, T., and Obermayer, K. 2000. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers. MIT Press, Cambridge, MA, 115--132.
	22	Holland, S., Ester, M., and Kieling, W. 2003. Preference mining: A novel approach on mining user preferences for personalized applications. In Proceedings of the Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD). 204--216.
	23	Hutchinson, W. and Alba, J. 1991. Ignoring irrelevant information: Situational determinats of consumer learning. J. Consum. Res. 18, 325--345.
	24	Bernard J. Jansen , Amanda Spink , Tefko Saracevic, Real life, real users, and real needs: a study and analysis of user queries on the web, Information Processing and Management: an International Journal, v.36 n.2, p.207-227, Jan.1.2000 [doi>10.1016/S0306-4573(99)00056-4]
	25	Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
	26	Joachims, T., Freitag, D., and Mitchell, T. 1997. WebWatcher: A tour guide for the World Wide Web. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Vol. 1. Morgan Kaufmann, San Francisco, CA, 770--777.
	27	Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076063]
	28	Rosie Jones , Daniel C. Fain, Query word deletion prediction, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860538]
	29	Just, M. and Carpenter, P. 1980. A theory of reading: From eye fixations to comprehension. Psycholog. Rev. 87, 329--354.
	30	Diane Kelly , Nicholas J. Belkin, Display time as implicit feedback: understanding task effects, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009057]
	31	Diane Kelly , Jaime Teevan, Implicit feedback for inferring user preference: a bibliography, ACM SIGIR Forum, v.37 n.2, p.18-28, Fall 2003 [doi>10.1145/959258.959260]
	32	Charles Kemp , Kotagiri Ramamohanarao, Long-Term Learning for Web Search Engines, Proceedings of the 6th European Conference on Principles of Data Mining and Knowledge Discovery, p.263-274, August 19-23, 2002
	33	Kerstin Klöckner , Nadine Wirschum , Anthony Jameson, Depth- and breadth-first processing of search result lists, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986115]
	34	Chris Lankford, Gazetracker: software designed to facilitate eye movement analysis, Proceedings of the 2000 symposium on Eye tracking research & applications, p.51-55, November 06-08, 2000, Palm Beach Gardens, Florida, United States [doi>10.1145/355017.355025]
	35	Lieberman, H. 1995. Letizia: An agent that assists Web browsing. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI '95). Morgan Kaufmann, San Francisco, CA, 924--929.
	36	Mantell, S. and Kardes, F. 1999. The role of direction of comparison, attribute-based processing, and attitude-based processing in consumer preference. J. Consum. Res. 25, 335--352.
	37	Masahiro Morita , Yoichi Shinoda, Information filtering based on user behavior analysis and best match text retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.272-281, July 03-06, 1994, Dublin, Ireland
	38	Oard, D. and Kim, J. 1998. Implicit feedback for recommender systems. In Proceedings of the AAAI Workshop on Recommender Systems. 81--83.
	39	Bing Pan , Helene A. Hembrooke , Geri K. Gay , Laura A. Granka , Matthew K. Feusner , Jill K. Newman, The determinants of web page viewing behavior: an eye-tracking study, Proceedings of the 2004 symposium on Eye tracking research & applications, p.147-154, March 22-24, 2004, San Antonio, Texas [doi>10.1145/968363.968391]
	40	Filip Radlinski , Thorsten Joachims, Query chains: learning to rank from implicit feedback, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081899]
	41	Rayner, K. 1998. Eye movements in reading and information processing. Psycholog. Bull. 124, 372--252.
	42	Salogarvi, J., Kojo, I., Jaana, S., and Kaski, S. 2003. Can relevance be inferred from eye movements in information retrieval? In Proceedings of the Workshop on Self-Organizing Maps. 261--266.
	43	Samuelson, P. 1948. Consumption theory in terms of revealed preferences. Econometrica 15, 243--253.
	44	Xuehua Shen , Bin Tan , ChengXiang Zhai, Context-sensitive information retrieval using implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076045]
	45	Craig Silverstein , Hannes Marais , Monika Henzinger , Michael Moricz, Analysis of a very large web search engine query log, ACM SIGIR Forum, v.33 n.1, p.6-12, Fall 1999 [doi>10.1145/331403.331405]
	46	Teevan, J., Dumais, S., and Horvitz, E. 2005. Beyond the commons: Investigating the value of personalizing Web search. In Proceedings of the Workshop on New Technologies for Personalized Information Access (PIA). 84--92.
	47	Varian, H. 1992. Microeconomic Analysis. Noron, New York, NY.
	48	Ryen White , Ian Ruthven , Joemon M. Jose, The Use of Implicit Evidence for Relevance Feedback in Web Retrieval, Proceedings of the 24th BCS-IRSG European Colloquium on IR Research: Advances in Information Retrieval, p.93-109, March 25-27, 2002
	49	Ryen W. White , Ian Ruthven , Joemon M. Jose, A study of factors affecting the utility of implicit relevance feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076044]

CITED BY 29

	Mingfang Wu , Andrew Turpin , Justin Zobel, An investigation on a community's web search variability, Proceedings of the thirty-first Australasian conference on Computer science, January 01-01, 2008, Wollongong, Australia
	Kuansan Wang , Toby Walker , Zijian Zheng, PSkip: estimating relevance ranking quality from web search clickthrough data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Chao Liu , Fan Guo , Christos Faloutsos, BBM: bayesian browsing model from petabyte-scale data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Jimmy Lin , Mark D. Smucker, How do users find things with PubMed?: towards automatic utility evaluation with user simulations, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Yandong Liu , Jiang Bian , Eugene Agichtein, Predicting information seeker satisfaction in community question answering, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Filip Radlinski , Thorsten Joachims, Active exploration for learning rankings from clickthrough data, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA
	Georges E. Dupret , Benjamin Piwowarski, A user browsing model to predict search engine click data from past observations., Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	GunWoo Park , JinGi Chae , Dae Hee Lee , SangHoon Lee, User intention based personalized search: HPS(hierarchical phrase search), WSEAS Transactions on Circuits and Systems, v.7 n.4, p.266-276, April 2008
	Fan Guo , Lei Li , Christos Faloutsos, Tailoring click models to user goals, Proceedings of the 2009 workshop on Web Search Click Data, p.88-92, February 09-09, 2009, Barcelona, Spain
	Arpita Ghosh , Mohammad Mahdian, Externalities in online advertising, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Norbert Fuhr, A probability ranking principle for interactive information retrieval, Information Retrieval, v.11 n.3, p.251-265, June 2008
	Filip Radlinski , Madhu Kurup , Thorsten Joachims, How does clickthrough data reflect retrieval quality?, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	Thorsten Joachims , Filip Radlinski, Search Engines that Learn from Implicit Feedback, Computer, v.40 n.8, p.34-40, August 2007
	GunWoo Park , JinGi Chae , Dae Hee Lee , SangHoon Lee, Personalized search based on user intention through the hierarchical phrase vector model, Proceedings of the WSEAS International Conference on Applied Computing Conference, p.205-210, May 27-30, 2008, Istanbul, Turkey
	Zhicheng Dou , Ruihua Song , Xiaojie Yuan , Ji-Rong Wen, Are click-through data adequate for learning web search rankings?, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	R. Agrawal , A. Halverson , K. Kenthapadi , N. Mishra , P. Tsaparas, Generating labels from clicks, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain
	Fan Guo , Chao Liu , Yi Min Wang, Efficient multiple-click models in web search, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain
	Michael Bendersky , W. Bruce Croft, Analysis of long queries in a large scale search log, Proceedings of the 2009 workshop on Web Search Click Data, p.8-14, February 09-09, 2009, Barcelona, Spain
	Fan Guo , Chao Liu , Anitha Kannan , Tom Minka , Michael Taylor , Yi-Min Wang , Christos Faloutsos, Click chain model in web search, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Yisong Yue , Thorsten Joachims, Interactively optimizing information retrieval systems as a dueling bandits problem, Proceedings of the 26th Annual International Conference on Machine Learning, p.1201-1208, June 14-18, 2009, Montreal, Quebec, Canada
	Xing Jiang , Ah-Hwee Tan, Learning and inferencing in user ontology for personalized Semantic Web search, Information Sciences: an International Journal, v.179 n.16, p.2794-2808, July, 2009
	Yossi Azar , Iftah Gamzu , Xiaoxin Yin, Multiple intents re-ranking, Proceedings of the 41st annual ACM symposium on Theory of computing, May 31-June 02, 2009, Bethesda, MD, USA
	Eugene Agichtein , Yandong Liu , Jiang Bian, Modeling information-seeker satisfaction in community question answering, ACM Transactions on Knowledge Discovery from Data (TKDD), v.3 n.2, p.1-27, April 2009
	Zeynel Abidin Misirli , Ferit Karakoyun , Abdullah Kuzu, Effective use of search strategies, Proceedings of the international conference on Computational and information science 2009, p.382-385, April 30-May 02, 2009, Houston, USA
	Georg Buscher , Ludger van Elst , Andreas Dengel, Segment-level display time as implicit feedback: a comparison to eye tracking, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Ariel Fuxman , Anitha Kannan , Andrew B. Goldberg , Rakesh Agrawal , Panayiotis Tsaparas , John Shafer, Improving classification accuracy using automatically extracted training data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Mingfang Wu , James A. Thom , Andrew Turpin , Ross Wilkinson, Cost and benefit analysis of mediated enterprise search, Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries, June 15-19, 2009, Austin, TX, USA
	Jimmy Lin , W. John Wilbur, Modeling actions of PubMed users with n-gram language models, Information Retrieval, v.12 n.4, p.487-503, August 2009
	Josh Attenberg , Sandeep Pandey , Torsten Suel, Modeling and predicting user behavior in sponsored search, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France