Accurately interpreting clickthrough data as implicit feedback

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Accurately interpreting clickthrough data as implicit feedback

Full text

Pdf (199 KB)

Source

Annual ACM Conference on Research and Development in Information Retrieval archive
Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval table of contents

Salvador, Brazil

SESSION: Evaluation table of contents

Pages: 154 - 161

Year of Publication: 2005

ISBN:1-59593-034-5

Authors

Thorsten Joachims	Cornell University, Ithaca, NY
Laura Granka	Stanford University, Palo Alto, CA
Bing Pan	Cornell University, Ithaca, NY
Helene Hembrooke	Cornell University, Ithaca, NY
Geri Gay	Cornell University, Ithaca, NY

Sponsor

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 45, Downloads (12 Months): 347, Citation Count: 115

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1076034.1076063 What is a DOI?

ABSTRACT

This paper examines the reliability of implicit feedback generated from clickthrough data in WWW search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Rodrigo B. Almeida , Virgilio A. F. Almeida, A community-aware search engine, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988728]
	2	Brian T. Bartell , Garrison W. Cottrell , Richard K. Belew, Automatic combination of multiple ranked retrieval systems, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.173-181, July 03-06, 1994, Dublin, Ireland
	3	R. Belew. Finding Out About. Cambridge, 2000.
	4	J. Boyan, D. Freitag, and T. Joachims. A machine learning architecture for optimizing web search engines. In AAAI Workshop on Internet Based Information Systems, pages 1--8, August 1996.
	5	Andrei Broder, A taxonomy of web search, ACM SIGIR Forum, v.36 n.2, Fall 2002 [doi>10.1145/792550.792552]
	6	Mark Claypool , Phong Le , Makoto Wased , David Brown, Implicit interest indicators, Proceedings of the 6th international conference on Intelligent user interfaces, p.33-40, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.359836]
	7	W. Cohen, R. Shapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243--270, 1999.
	8	S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve the search experiences. In Talk presented at SIGIR03 Workshop on Implicit Measures of User Interests and Preferences, 2003.
	9	Norbert Fuhr, Optimum polynomial retrieval functions based on the probability ranking principle, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.183-204, July 1989 [doi>10.1145/65943.65944]
	10	Joseph H. Goldberg , Mark J. Stimson , Marion Lewenstein , Neil Scott , Anna M. Wichansky, Eye tracking in web search tasks: design implications, Proceedings of the 2002 symposium on Eye tracking research & applications, March 25-27, 2002, New Orleans, Louisiana [doi>10.1145/507072.507082]
	11	Laura A. Granka , Thorsten Joachims , Geri Gay, Eye-tracking analysis of user behavior in WWW search, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009079]
	12	Tim Halverson , Anthony J. Hornof, Link colors guide a search, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986066]
	13	S. Holland, M. Ester, and W. Kieβling. Preference mining: A novel approach on mining user preferences for personalized applications. In Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD), pages 204 -- 216, 2003.
	14	Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
	15	M. Just and P. Carpenter. A theory of reading: From eye fixations to comprehension. Psychological Review, 87:329--354, 1980.
	16	Diane Kelly , Nicholas J. Belkin, Display time as implicit feedback: understanding task effects, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009057]
	17	Charles Kemp , Kotagiri Ramamohanarao, Long-Term Learning for Web Search Engines, Proceedings of the 6th European Conference on Principles of Data Mining and Knowledge Discovery, p.263-274, August 19-23, 2002
	18	Kerstin Klöckner , Nadine Wirschum , Anthony Jameson, Depth- and breadth-first processing of search result lists, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986115]
	19	Chris Lankford, Gazetracker: software designed to facilitate eye movement analysis, Proceedings of the 2000 symposium on Eye tracking research & applications, p.51-55, November 06-08, 2000, Palm Beach Gardens, Florida, United States [doi>10.1145/355017.355025]
	20	Masahiro Morita , Yoichi Shinoda, Information filtering based on user behavior analysis and best match text retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.272-281, July 03-06, 1994, Dublin, Ireland
	21	Bing Pan , Helene A. Hembrooke , Geri K. Gay , Laura A. Granka , Matthew K. Feusner , Jill K. Newman, The determinants of web page viewing behavior: an eye-tracking study, Proceedings of the 2004 symposium on Eye tracking research & applications, p.147-154, March 22-24, 2004, San Antonio, Texas [doi>10.1145/968363.968391]
	22	Filip Radlinski , Thorsten Joachims, Query chains: learning to rank from implicit feedback, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081899]
	23	K. Rayner. Eye movements in reading and information processing. Psychological Bulletin, 124:372--252, 1998.
	24	E. Russo and F. LeClerc. An eye-fixation analysis of choice processes for consumer nondurables. Journal of Consumer Research, 21(2):274--290, 1994.
	25	J. Salogarvi, I. Kojo, S. Jaana, and S. Kaski. Can relevance be inferred from eye movements in information retrieval? In Proceedings of the Workshop on Self-Organizing Maps, pages 261--266, 2003.
	26	R. White, J. Jose, and I. Ruthven. Comparing explicit and implicit feedback techniques for web retrieval: Trec-10 interactive track report. In Text Retrieval Conference (TREC), 2001.

CITED BY 119

	Edward Cutrell , Daniel Robbins , Susan Dumais , Raman Sarin, Fast, flexible filtering with phlat, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	Filip Radlinski , Thorsten Joachims, Query chains: learning to rank from implicit feedback, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA
	Jaime Teevan, How people recall search result lists, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	Eugene Agichtein , Zijian Zheng, Identifying "best bet" web search results by mining past user behavior, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA
	Qiankun Zhao , Steven C. H. Hoi , Tie-Yan Liu , Sourav S. Bhowmick , Michael R. Lyu , Wei-Ying Ma, Time-dependent semantic similarity measure of queries using historical click-through data, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Irina Matveeva , Chris Burges , Timo Burkard , Andy Laucius , Leon Wong, High accuracy retrieval with multiple nested ranker, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Michael J. Welch , Junghoo Cho, Automatically identifying localizable queries, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Georg Buscher , Andreas Dengel , Ludger van Elst, Query expansion using gaze-based feedback on the subdocument level, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Jaime Teevan , Susan T. Dumais , Daniel J. Liebling, To personalize or not to personalize: modeling queries with variation in user intent, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Zhaohui Zheng , Keke Chen , Gordon Sun , Hongyuan Zha, A regression framework for learning ranking functions using relative relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Nick Craswell , Martin Szummer, Random walks on the click graph, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Narayanan Sadagopan , Jie Li, Characterizing typical and atypical user sessions in clickstreams, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Eric C. Jensen , Steven M. Beitzel , Abdur Chowdhury , Ophir Frieder, Repeatable evaluation of search services in dynamic environments, ACM Transactions on Information Systems (TOIS), v.26 n.1, p.1-es, November 2007
	Eugene Agichtein , Eric Brill , Susan Dumais , Robert Ragno, Learning user interaction models for predicting web search result preferences, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Oisín Boydell , Barry Smyth, Capturing community search expertise for personalized web search using snippet-indexes, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	André Luiz da Costa Carvalho , Paul - Alexandru Chirita , Edleno Silva de Moura , Pável Calado , Wolfgang Nejdl, Site level noise removal for search engines, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Matthew Richardson , Amit Prakash , Eric Brill, Beyond PageRank: machine learning for static ranking, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	Andrew Turpin , Falk Scholer, User performance versus precision measures for simple search tasks, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Eugene Agichtein , Eric Brill , Susan Dumais, Improving web search ranking by incorporating user behavior information, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Pavel A. Dmitriev , Nadav Eiron , Marcus Fontoura , Eugene Shekita, Using annotations in enterprise search, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland
	En Cheng , Feng Jing , Lei Zhang , Hai Jin, Scalable relevance feedback using click-through data for web image retrieval, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA
	Paul Thomas , David Hawking, Evaluation by comparing result sets in context, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Hongyuan Zha , Zhaohui Zheng , Haoying Fu , Gordon Sun, Incorporating query difference for learning retrieval functions in world wide web search, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Paul-Alexandru Chirita , Claudiu S. Firan , Wolfgang Nejdl, Summarizing local context to personalize global web search, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Ryen W. White , Diane Kelly, A study on the effects of personalization and task information on implicit feedback performance, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Zhiwei Guan , Edward Cutrell, An eye tracking study of the effect of target rank on web search, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA
	Yuanhua Lv , Le Sun , Junlin Zhang , Jian-Yun Nie , Wan Chen , Wei Zhang, An iterative implicit feedback approach to personalized search, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL, p.585-592, July 17-18, 2006, Sydney, Australia
	Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007
	Shihao Ji , Ke Zhou , Ciya Liao , Zhaohui Zheng , Gui-Rong Xue , Olivier Chapelle , Gordon Sun , Hongyuan Zha, Global ranking by exploiting user clicks, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Chao Liu , Fan Guo , Christos Faloutsos, BBM: bayesian browsing model from petabyte-scale data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Seikyung Jung , Jonathan L. Herlocker , Janet Webster, Click data as implicit relevance feedback in web search, Information Processing and Management: an International Journal, v.43 n.3, p.791-807, May, 2007
	Mikhail Bilenko , Ryen W. White, Mining the search trails of surfing crowds: identifying relevant websites from user activity, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Alistair Moffat , William Webber , Justin Zobel, Strategic system comparisons via targeted relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Edward Cutrell , Zhiwei Guan, What are you looking for?: an eye-tracking study of information usage in web search, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA
	Mark T. Keane , Maeve O'Brien , Barry Smyth, Are people biased in their use of search engines?, Communications of the ACM, v.51 n.2, p.49-52, February 2008
	Jianfeng Gao , Wei Yuan , Xiao Li , Kefeng Deng , Jian-Yun Nie, Smoothing clickthrough data for web search ranking, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Xin Fu, Evaluating sources of implicit feedback in web searches, Proceedings of the 2007 ACM conference on Recommender systems, October 19-20, 2007, Minneapolis, MN, USA
	Stefan Pohl , Filip Radlinski , Thorsten Joachims, Recommending related papers based on digital library access records, Proceedings of the 2007 conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada
	Charles L.A. Clarke , Maheedhar Kolla , Gordon V. Cormack , Olga Vechtomova , Azin Ashkan , Stefan Büttcher , Ian MacKinnon, Novelty and diversity in information retrieval evaluation, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Ke Zhou , Gui-Rong Xue , Hongyuan Zha , Yong Yu, Learning to rank with ties, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Judit Bar-Ilan , Kevin Keenoy , Eti Yaari , Mark Levene, User rankings of search engine results, Journal of the American Society for Information Science and Technology, v.58 n.9, p.1254-1266, July 2007
	Yang Sun , Huajing Li , Isaac G. Councill , Wang-Chien Lee , C. Lee Giles, Measuring user preference changes in digital libraries, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	Donald Metzler , W. Bruce Croft, Linear feature-based models for information retrieval, Information Retrieval, v.10 n.3, p.257-274, June 2007
	Maarten van der Heijden , Max Hinne , Wessel Kraaij , Suzan Verberne , Theo van der Weide, Using query logs and click data to create improved document descriptions, Proceedings of the 2009 workshop on Web Search Click Data, p.64-67, February 09-09, 2009, Barcelona, Spain
	Ryen W. White , Mikhail Bilenko , Silviu Cucerzan, Leveraging popular destinations to enhance Web search interaction, ACM Transactions on the Web (TWEB), v.2 n.3, p.1-30, July 2008
	Georges E. Dupret , Benjamin Piwowarski, A user browsing model to predict search engine click data from past observations., Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Songhua Xu , Hao Jiang , Francis C.M. Lau, User-oriented document summarization through vision-based eye-tracking, Proceedings of the 13th international conference on Intelligent user interfaces, February 08-11, 2009, Sanibel Island, Florida, USA
	Fan Guo , Lei Li , Christos Faloutsos, Tailoring click models to user goals, Proceedings of the 2009 workshop on Web Search Click Data, p.88-92, February 09-09, 2009, Barcelona, Spain
	Sara Cohen , Carmel Domshlak , Naama Zwerdling, On ranking techniques for desktop search, ACM Transactions on Information Systems (TOIS), v.26 n.2, p.1-24, March 2008
	Charles L. A. Clarke , Eugene Agichtein , Susan Dumais , Ryen W. White, The influence of caption features on clickthrough patterns in web search, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Maurice Coyle , Barry Smyth, Supporting intelligent Web search, ACM Transactions on Internet Technology (TOIT), v.7 n.4, p.20-es, October 2007
	Xin Fu , Diane Kelly , Chirag Shah, Using collaborative queries to improve retrieval for difficult topics, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Tanuja Bompada , Chi-Chao Chang , John Chen , Ravi Kumar , Rajesh Shenoy, On the robustness of relevance measures with incomplete judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Wilfred Ng , Lin Deng , Dik Lun Lee, Mining User preference using Spy voting for search engine personalization, ACM Transactions on Internet Technology (TOIT), v.7 n.4, p.19-es, October 2007
	Eugene Agichtein , Carlos Castillo , Debora Donato , Aristides Gionis , Gilad Mishne, Finding high-quality content in social media, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA
	Massimo Melucci , Ryen W. White, Utilizing a geometry of context for enhanced implicit feedback, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Jiang Bian , Yandong Liu , Eugene Agichtein , Hongyuan Zha, Finding the right facts in the crowd: factoid question answering over social media, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Massimiliano Ciaramita , Vanessa Murdock , Vassilis Plachouras, Online learning from click data for sponsored search, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	N. Sadat Shami , Kate Ehrlich , David R. Millen, Pick me!: link selection in expertise search results, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy
	Kerry Rodden , Xin Fu , Anne Aula , Ian Spiro, Eye-mouse coordination patterns on web search results pages, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy
	Nick Craswell , Onno Zoeter , Michael Taylor , Bill Ramsey, An experimental comparison of click position-bias models, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA
	Eyke Hüllermeier , Johannes Fürnkranz , Weiwei Cheng , Klaus Brinker, Label ranking by learning pairwise preferences, Artificial Intelligence, v.172 n.16-17, p.1897-1916, November, 2008
	Azzah Al-Maskari , Mark Sanderson , Paul Clough , Eija Airio, The good and the bad system: does the test collection predict users' effectiveness?, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Rosta Farzan , Peter Brusilovsky, Where did the researchers go?: supporting social navigation at a large academic, Proceedings of the nineteenth ACM conference on Hypertext and hypermedia, June 19-21, 2008, Pittsburgh, PA, USA
	Shawn R. Jeffery , Michael J. Franklin , Alon Y. Halevy, Pay-as-you-go user feedback for dataspace systems, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada
	Wai-Tat Fu , Peter Pirolli, SNIF-ACT: a cognitive model of user navigation on the world wide web, Human-Computer Interaction, v.22 n.4, p.355-412, November 2007
	Jaime Teevan, How people recall, recognize, and reuse search results, ACM Transactions on Information Systems (TOIS), v.26 n.4, p.1-27, September 2008
	Jiang Bian , Yandong Liu , Eugene Agichtein , Hongyuan Zha, A few bad votes too many?: towards robust ranking in social media, Proceedings of the 4th international workshop on Adversarial information retrieval on the web, April 22-22, 2008, Beijing, China
	Claudio Biancalana , Alessandro Micarelli , Claudio Squarcella, Nereau: a social approach to query expansion, Proceeding of the 10th ACM workshop on Web information and data management, October 30-30, 2008, Napa Valley, California, USA
	Zhicheng Dou , Ruihua Song , Xiaojie Yuan , Ji-Rong Wen, Are click-through data adequate for learning web search rankings?, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	Qiaozhu Mei , Dengyong Zhou , Kenneth Church, Query suggestion using hitting time, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	Keke Chen , Rongqing Lu , C. K. Wong , Gordon Sun , Larry Heck , Belle Tseng, Trada: tree based ranking function adaptation, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	Yang Sun , Huajing Li , Isaac G. Councill , Jian Huang , Wang-Chien Lee , C. Lee Giles, Personalized ranking for digital libraries based on log analysis, Proceeding of the 10th ACM workshop on Web information and data management, October 30-30, 2008, Napa Valley, California, USA
	Songhua Xu , Hao Jiang , Francis C.M. Lau, Personalized online document, image and video recommendation via commodity eye-tracking, Proceedings of the 2008 ACM conference on Recommender systems, October 23-25, 2008, Lausanne, Switzerland
	Greg Buehrer , Jack W. Stokes , Kumar Chellapilla, A large-scale study of automated web search traffic, Proceedings of the 4th international workshop on Adversarial information retrieval on the web, April 22-22, 2008, Beijing, China
	Ziv Bar-Yossef , Maxim Gurevich, Mining search engine query logs via suggestion sampling, Proceedings of the VLDB Endowment, v.1 n.1, August 2008
	Atish Das Sarma , Sreenivas Gollapudi , Samuel Ieong, Bypass rates: reducing query abandonment using negative inferences, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2008, Las Vegas, Nevada, USA
	Kai Puolamäki , Antti Ajanki , Samuel Kaski, Learning to learn implicit queries from gaze patterns, Proceedings of the 25th international conference on Machine learning, p.760-767, July 05-09, 2008, Helsinki, Finland
	Chariste Papaeconomou , Annemarie F. Zijlema , Peter Ingwersen, Searchers' relevance judgments and criteria in evaluating web pages in a learning style perspective, Proceedings of the second international symposium on Information interaction in context, October 14-17, 2008, London, United Kingdom
	Daniel Gayo-Avello, A survey on session detection methods in query logs and a proposal for future evaluation, Information Sciences: an International Journal, v.179 n.12, p.1822-1843, May, 2009
	Ziv Bar-Yossef , Maxim Gurevich, Estimating the impressionrank of web pages, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Sofia Stamou , Alexandros Ntoulas, Search personalization through query and page topical analysis, User Modeling and User-Adapted Interaction, v.19 n.1-2, p.5-33, February 2009
	Alistair Moffat , Justin Zobel, Rank-biased precision for measurement of retrieval effectiveness, ACM Transactions on Information Systems (TOIS), v.27 n.1, p.1-27, December 2008
	Jaap Kamps , Marijn Koolen , Andrew Trotman, Comparative analysis of clicks and judgments for IR evaluation, Proceedings of the 2009 workshop on Web Search Click Data, p.80-87, February 09-09, 2009, Barcelona, Spain
	R. Agrawal , A. Halverson , K. Kenthapadi , N. Mishra , P. Tsaparas, Generating labels from clicks, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain
	Fan Guo , Chao Liu , Yi Min Wang, Efficient multiple-click models in web search, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain
	Masaya Murata , Hiroyuki Toda , Yumiko Matsuura , Ryoji Kataoka, Query-page intention matching using clicked titles and snippets to boost search rankings, Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries, June 15-19, 2009, Austin, TX, USA
	Milad Shokouhi , Justin Zobel, Robust result merging using sample-based score estimates, ACM Transactions on Information Systems (TOIS), v.27 n.3, p.1-29, May 2009
	Olivier Chapelle , Ya Zhang, A dynamic bayesian network click model for web search ranking, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Fan Guo , Chao Liu , Anitha Kannan , Tom Minka , Michael Taylor , Yi-Min Wang , Christos Faloutsos, Click chain model in web search, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Xin Yang , Peifeng Xiang , Yuanchun Shi, Finding User's Interest Blocks using Significant Implicit Evidence for Web Browsing on Small Screen Devices, World Wide Web, v.12 n.2, p.213-234, June 2009
	Jun Yan , Ning Liu , Elaine Qing Chang , Lei Ji , Zheng Chen, Search result re-ranking based on gap between search queries and social tags, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Aleksandra Korolova , Krishnaram Kenthapadi , Nina Mishra , Alexandros Ntoulas, Releasing search queries and clicks privately, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Jay Chen , Lakshminarayanan Subramanian , Kentaro Toyama, Web search and browsing behavior under poor connectivity, Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA
	Masaya Murata , Hiroyuki Toda , Yumiko Matsuura , Ryoji Kataoka, Access concentration detection in click logs to improve mobile Web-IR, Information Sciences: an International Journal, v.179 n.12, p.1859-1869, May, 2009
	Jane Li , Scott Huffman , Akihito Tokuda, Good abandonment in mobile and PC internet search, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Filip Radlinski , Thorsten Joachims, Minimally invasive randomization for collecting unbiased preferences from clickthrough logs, proceedings of the 21st national conference on Artificial intelligence, p.1406-1412, July 16-20, 2006, Boston, Massachusetts
	D. Sculley , Robert G. Malkin , Sugato Basu , Roberto J. Bayardo, Predicting bounce rates in sponsored search advertisements, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	S. Sendhilkumar , T. V. Geetha, Architecture for effective personalised web search, International Journal of Computer Applications in Technology, v.35 n.2/3/4, p.219-233, June 2009
	Milad Shokouhi , Leif Azzopardi , Paul Thomas, Effective query expansion for federated search, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Yi Chang , Anlei Dong , Ciya Liao , Zhaohui Zheng, Enhancing topical ranking with preferences from click-through data, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Erick Delage, Regret-based online ranking for a growing digital library, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	William Webber , Laurence A. F. Park, Score adjustment for correction of pooling bias, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Meliha Yetisgen-Yildiz , Wanda Pratt, A new evaluation methodology for literature-based discovery systems, Journal of Biomedical Informatics, v.42 n.4, p.633-643, August, 2009
	Andrew Turpin , Falk Scholer , Kalvero Jarvelin , Mingfang Wu , J. Shane Culpepper, Including summaries in system evaluation, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Songhua Xu , Yi Zhu , Hao Jiang , Francis C. M. Lau, A user-oriented webpage ranking algorithm based on user attention time, Proceedings of the 23rd national conference on Artificial intelligence, p.1255-1260, July 13-17, 2008, Chicago, Illinois
	Hila Becker , Christopher Meek , David Maxwell Chickering, Modeling contextual factors of click rates, Proceedings of the 22nd national conference on Artificial intelligence, p.1310-1315, July 22-26, 2007, Vancouver, British Columbia, Canada
	Ying Zhang , Bernard J. Jansen , Amanda Spink, Identification of factors predicting clickthrough in Web searching using neural network analysis, Journal of the American Society for Information Science and Technology, v.60 n.3, p.557-570, March 2009
	Benyah Shaparenko , Özgür Çetin , Rukmini Iyer, Data-driven text features for sponsored search click prediction, Proceedings of the Third International Workshop on Data Mining and Audience Intelligence for Advertising, p.46-54, June 28-28, 2009, Paris, France
	Diane Kelly, Methods for Evaluating Interactive Information Retrieval Systems with Users, Foundations and Trends in Information Retrieval, v.3 n.1—2, p.1-224, January 2009
	Guangyu Zhu , Gilad Mishne, Mining rich session context to improve web search, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Paul Thomas , Milad Shokouhi, SUSHI: scoring scaled samples for server selection, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Yunzhang Zhu , Gang Wang , Junli Yang , Dakan Wang , Jun Yan , Jian Hu , Zheng Chen, Optimizing search engine revenue in sponsored search, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Sonal Gupta , Mikhail Bilenko , Matthew Richardson, Catching the drift: learning broad matches from clickthrough data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Josh Attenberg , Sandeep Pandey , Torsten Suel, Modeling and predicting user behavior in sponsored search, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Helen Ashman , Michael Antunovic , Christoph Donner , Rebecca Frith , Eric Rebelos , Jan-Felix Schmakeit , Gavin Smith , Mark Truran, Are Clickthroughs Useful for Image Labelling?, Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, p.191-197, September 15-18, 2009
	Azin Ashkan , Charles L. A. Clarke , Eugene Agichtein , Qi Guo, Estimating Ad Clickthrough Rate through Query Intent Analysis, Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, p.222-229, September 15-18, 2009
	Philip O'Brien , Xiao Luo , Tony Abou-Assaleh , Weizheng Gao , Shujie Li, Personalization of Content Ranking in the Context of Local Search, Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, p.532-539, September 15-18, 2009
	En Cheng , Feng Jing , Lei Zhang, A unified relevance feedback framework for web image retrieval, IEEE Transactions on Image Processing, v.18 n.6, p.1350-1357, June 2009