Segment-level display time as implicit feedback: a comparison to eye tracking

ABSTRACT

We examine two basic sources for implicit relevance feedback on the segment level for search personalization: eye tracking and display time. A controlled study has been conducted where 32 participants had to view documents in front of an eye tracker, query a search engine, and give explicit relevance ratings for the results. We examined the performance of the basic implicit feedback methods with respect to improved ranking and compared their performance to a pseudo relevance feedback baseline on the segment level and the original ranking of a Web search engine.

Our results show that feedback based on display time on the segment level is much coarser than feedback from eye tracking. But surprisingly, for re-ranking and query expansion it did work as well as eye-tracking-based feedback. All behavior-based methods performed significantly better than our non-behavior-based baseline and especially improved poor initial rankings of the Web search engine.

The study shows that segment-level display time yields comparable results as eye-tracking-based feedback. Thus, it should be considered in future personalization systems as an inexpensive but precise method for implicit feedback.

REFERENCES

• 1: Eugene Agichtein , Eric Brill , Susan Dumais, Improving web search ranking by incorporating user behavior information, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148177]
• 2: Jae-wook Ahn , Peter Brusilovsky , Daqing He , Jonathan Grady , Qi Li, Personalized web exploration with task models, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367499]
• 3: Ioannis Arapakis , Joemon M. Jose , Philip D. Gray, Affective feedback: an investigation into the role of emotions in the information seeking process, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390403]
• 4: Nicholas J. Belkin, Some(what) grand challenges for information retrieval, ACM SIGIR Forum, v.42 n.1, June 2008 [doi>10.1145/1394251.1394261]
• 5: Georg Buscher , Andreas Dengel , Ludger van Elst, Eye movements as implicit relevance feedback, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358796]
• 6: Georg Buscher , Andreas Dengel , Ludger van Elst, Query expansion using gaze-based feedback on the subdocument level, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390401]
• 7: Mark Claypool , Phong Le , Makoto Wased , David Brown, Implicit interest indicators, Proceedings of the 6th international conference on Intelligent user interfaces, p.33-40, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.359836]
• 8: Steve Fox , Kuldeep Karnawat , Mark Mydland , Susan Dumais , Thomas White, Evaluating implicit measures to improve web search, ACM Transactions on Information Systems (TOIS), v.23 n.2, p.147-168, April 2005 [doi>10.1145/1059981.1059982]
• 9: Gene Golovchinsky , Morgan N. Price , Bill N. Schilit, From reading to retrieval: freeform ink annotations as queries, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.19-25, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312637]
• 10: Karl Gyllstrom , Craig Soules, Seeing is retrieving: building information context from what the user sees, Proceedings of the 13th international conference on Intelligent user interfaces, January 13-16, 2008, Gran Canaria, Spain [doi>10.1145/1378773.1378798]
• 11: Kalervo Järvelin , Jaana Kekäläinen, IR evaluation methods for retrieving highly relevant documents, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.41-48, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345545]
• 12: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007 [doi>10.1145/1229179.1229181]
• 13: Diane Kelly , Nicholas J. Belkin, Reading time, scrolling and interaction: exploring implicit sources of user preferences for relevance feedback, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.408-409, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384045]
• 14: Diane Kelly , Nicholas J. Belkin, Display time as implicit feedback: understanding task effects, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009057]
• 15: Diane Kelly , Jaime Teevan, Implicit feedback for inferring user preference: a bibliography, ACM SIGIR Forum, v.37 n.2, p.18-28, Fall 2003 [doi>10.1145/959258.959260]
• 16: M. Melucci and R.W. White. Discovering hidden contextual factors for implicit feedback. In Proc. of the CIR'07 Workshop on Context-Based Information Retrieval in conjunction with CONTEXT-07, 2007.
• 17: Masahiro Morita , Yoichi Shinoda, Information filtering based on user behavior analysis and best match text retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.272-281, July 03-06, 1994, Dublin, Ireland
• 18: D.W. Oard and J. Kim. Modeling information content using observable behavior. In Proc. of the 64 Annual Meeting of the American Society for Information Science and Technology, pp. 38--45, 2001.
• 19: R. Rafter and B. Smyth. Passive profiling from server logs in an online recruitment environment. In ITWP 2001: Proc. of the IJCAI Workshop on Intelligent Techniques for Web Personalization, 2001.
• 20: J. Rocchio. The SMART Retrieval System: Experiments in Automatic Document Processing, chapter Relevance feedback in information retrieval, pp. 313--323. Prentice Hall, 1971.
• 21: J. Salojarvi, K. Puolamäki, and S. Kaski. Implicit relevance feedback from eye movements. In ICANN'05, pp. 513--518, 2005.
• 22: Ryen W. White , Diane Kelly, A study on the effects of personalization and task information on implicit feedback performance, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183659]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Experimentation, Measurement

Keywords:
display time, eye tracking, implicit feedback, personalization