A study on the effects of personalization and task information on implicit feedback performance

ABSTRACT

While Implicit Relevance Feedback (IRF) algorithms exploit users' interactions with information to customize support offered to users of search systems, it is unclear how individual and task differences impact the effectiveness of such algorithms. In this paper we describe a study on the effect on retrieval performance of using additional information about the user and their search tasks when developing IRF algorithms. We tested four algorithms that use document display time to estimate relevance, and tailored the threshold times (i.e., the time distinguishing relevance from non-relevance) to the task, the user, a combination of both, or neither. Interaction logs gathered during a longitudinal naturalistic study of online information-seeking behavior are used as stimuli for the algorithms. The findings show that tailoring display time thresholds based on task information improves IRF algorithm performance, but doing so based on user information worsens performance. This has implications for the development of effective IRF algorithms.

REFERENCES

• 1: Eugene Agichtein , Eric Brill , Susan Dumais, Improving web search ranking by incorporating user behavior information, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148177]
• 2: Belkin, N. J., Oddy, R., and Brooks, H. (1982). ASK for information retrieval: Part I. Journal of Documentation, 38 (2): 61--71.
• 3: Chris Buckley , Gerard Salton , James Allan, The effect of adding relevance information in a relevance feedback environment, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.292-300, July 03-06, 1994, Dublin, Ireland
• 4: Mark Claypool , Phong Le , Makoto Wased , David Brown, Implicit interest indicators, Proceedings of the 6th international conference on Intelligent user interfaces, p.33-40, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.359836]
• 5: Cleverdon, C. W., Mills, J., and Keen, M. (1966). Factors determining the performance of indexing systems. ASLIB Cranfield project, Cranfield.
• 6: Anton N. Dragunov , Thomas G. Dietterich , Kevin Johnsrude , Matthew McLaughlin , Lida Li , Jonathan L. Herlocker, TaskTracer: a desktop environment to support multi-tasking knowledge workers, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040855]
• 7: Efthimis N. Efthimiadis, Interactive query expansion: a user-based evaluation in a relevance feedback environment, Journal of the American Society for Information Science, v.51 n.11, p.989-1003, Sept. 2000 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1002>3.0.CO;2-B]
• 8: Steve Fox , Kuldeep Karnawat , Mark Mydland , Susan Dumais , Thomas White, Evaluating implicit measures to improve web search, ACM Transactions on Information Systems (TOIS), v.23 n.2, p.147-168, April 2005 [doi>10.1145/1059981.1059982]
• 9: Gravetter, F.J. and Wallnau, L. B. (2004). Essentials of statistics for the behavioral sciences. Wadsworth Publishing: New York.
• 10: D. Harman, Towards interactive query expansion, Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval, p.321-331, May 1988, Grenoble, France [doi>10.1145/62437.62469]
• 11: Bernard J. Jansen , Amanda Spink , Tefko Saracevic, Real life, real users, and real needs: a study and analysis of user queries on the web, Information Processing and Management: an International Journal, v.36 n.2, p.207-227, Jan.1.2000 [doi>10.1016/S0306-4573(99)00056-4]
• 12: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076063]
• 13: Joyce Diane Kelly , Nicholas J. Belkin, Understanding implicit feedback and document preference: a naturalistic user study, 2004
• 14: Diane Kelly , Nicholas J. Belkin, Display time as implicit feedback: understanding task effects, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009057]
• 15: Diane Kelly , Vijay Deepak Dollu , Xin Fu, The loquacious user: a document-independent source of terms for query expansion, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076112]
• 16: Diane Kelly , Jaime Teevan, Implicit feedback for inferring user preference: a bibliography, ACM SIGIR Forum, v.37 n.2, p.18-28, Fall 2003 [doi>10.1145/959258.959260]
• 17: Oddy, R. N. (1977). Information retrieval through man-machine dialogue. Journal of Documentation, 33(1): 1--14.
• 18: Peter Pirolli , Stuart K. Card , Mija M. Van Der Wege, Visual information foraging in a focus + context visualization, Proceedings of the SIGCHI conference on Human factors in computing systems, p.506-513, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365337]
• 19: S. E. Robertson, On term selection for query expansion, Journal of Documentation, v.46 n.4, p.359-364, Dec. 1990 [doi>10.1108/eb026866]
• 20: Daniel E. Rose , Danny Levinson, Understanding user goals in web search, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988675]
• 21: Salton, G. and Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4): 288--197.
• 22: Taylor, R. S. (1968). Question negotiation and information seeking in libraries. College and Research Libraries, 29: 178--194.
• 23: Jaime Teevan , Susan T. Dumais , Eric Horvitz, Personalizing search via automated analysis of interests and activities, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076111]
• 24: Ryen W. White, Using searcher simulations to redesign a polyrepresentative implicit feedback interface, Information Processing and Management: an International Journal, v.42 n.5, p.1185-1202, September 2006 [doi>10.1016/j.ipm.2006.02.005]
• 25: White, R. W. and Marchionini, G. (2006). Examining the effectiveness of real-time query expansion. Information Processing and Management, in press.
• 26: Ryen W. White , Ian Ruthven , Joemon M. Jose, A study of factors affecting the utility of implicit relevance feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076044]
• 27: Ryen W. White , Ian Ruthven , Joemon M. Jose , C. J. Van Rijsbergen, Evaluating implicit feedback models using searcher simulations, ACM Transactions on Information Systems (TOIS), v.23 n.3, p.325-361, July 2005 [doi>10.1145/1080343.1080347]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Experimentation, Human Factors

Keywords:
evaluation, implicit feedback