How do users find things with PubMed?: towards automatic utility evaluation with user simulations

ABSTRACT

In the context of document retrieval in the biomedical domain, this paper explores the complex relationship between the quality of initial query results and the overall utility of an interactive retrieval system. We demonstrate that a content-similarity browsing tool can compensate for poor retrieval results, and that the relationship between retrieval performance and overall utility is non-linear. Arguments are advanced with user simulations, which characterize the relevance of documents that a user might encounter with different browsing strategies. With broader implications to IR, this work provides a case study of how user simulations can be exploited as a formative tool for automatic utility evaluation. Simulation-based studies provide researchers with an additional evaluation tool to complement interactive and Cranfield-style experiments.

REFERENCES

• 1: IJsbrand Jan Aalbersberg, Incremental relevance feedback, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.11-22, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133169]
• 2: Eugene Agichtein , Eric Brill , Susan Dumais , Robert Ragno, Learning user interaction models for predicting web search result preferences, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148175]
• 3: James Allan , Ben Carterette , Joshua Lewis, When will information retrieval be "good enough"?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076109]
• 4: A. Aula, P. Majaranta, and K.-J. Räihä. Eye-tracking reveals the personal styles for search result evaluation. In INTERACT 2005.
• 5: Ed H. Chi , Peter Pirolli , Kim Chen , James Pitkow, Using information scent to model user information needs and actions and the Web, Proceedings of the SIGCHI conference on Human factors in computing systems, p.490-497, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365325]
• 6: C. Cleverdon, J. Mills, and E. Keen. Factors determining the performance of indexing systems. Technical report, Cranfield Research Project, 1968.
• 7: W. Hersh, A. Cohen, J. Yang, R. Bhupatiraju, P. Roberts, and M. Hearst. TREC 2005 Genomics Track overview. In TREC 2005.
• 8: W. Hersh and P. Over. Interactivity at the Text Retrieval Conference. IP&M, 37(3):365--367, 2001.
• 9: W. Hersh, S. Price, D. Kraemer, B. Chan, L. Sacherek, and D. Olson. A large-scale comparison of Boolean vs. natural language searching for the TREC-7 interactive track. In TREC 7.
• 10: William Hersh , Andrew Turpin , Susan Price , Benjamin Chan , Dale Kramer , Lynetta Sacherek , Daniel Olson, Do batch and user evaluations give the same results?, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.17-24, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345539]
• 11: Michael Huggett , Joel Lanir, Static reformulation: a user study of static hypertext for query-based reformulation, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255238]
• 12: Melody Y. Ivory , Marti A Hearst, The state of the art in automating usability evaluation of user interfaces, ACM Computing Surveys (CSUR), v.33 n.4, p.470-516, December 2001 [doi>10.1145/503112.503114]
• 13: Bernard J. Jansen , Amanda Spink, How are we searching the world wide web?: a comparison of nine search engine transaction logs, Information Processing and Management: an International Journal, v.42 n.1, p.248-263, January 2006 [doi>10.1016/j.ipm.2004.10.007]
• 14: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007 [doi>10.1145/1229179.1229181]
• 15: Kerstin Klöckner , Nadine Wirschum , Anthony Jameson, Depth- and breadth-first processing of search result lists, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986115]
• 16: Jürgen Koenemann , Nicholas J. Belkin, A case for interaction: a study of interactive information retrieval behavior and effectiveness, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.205-212, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238487]
• 17: Lemur. Lemur Toolkit for Language Modeling and IR, 2003. http://www.lemurproject.org/.
• 18: A. Leuski and J. Allan. Strategy-based interactive cluster visualization for information retrieval. International Journal on Digital Libraries, 3(2):170--184, 2000.
• 19: Jimmy Lin , Michael DiCuccio , Vahan Grigoryan , W. John Wilbur, Navigating information spaces: A case study of related article search in PubMed, Information Processing and Management: an International Journal, v.44 n.5, p.1771-1783, September, 2008 [doi>10.1016/j.ipm.2008.04.002]
• 20: J. Lin and W. Wilbur. PubMed related articles: A probabilistic topic-based model for content similarity. BMC Bioinformatics, 8:423, 2007.
• 21: Mark Magennis , Cornelis J. van Rijsbergen, The potential and actual effectiveness of interactive query expansion, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.324-332, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 22: Javed Mostafa , Snehasis Mukhopadhyay , Mathew Palakal, Simulation Studies of Different Dimensions of Users' Interests and their Impact on User Modeling and Information Filtering, Information Retrieval, v.6 n.2, p.199-223, April 2003 [doi>10.1023/A:1023932221048]
• 23: Mark D. Smucker , James Allan, Find-similar: similarity browsing as a search tool, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148250]
• 24: Andrew H. Turpin , William Hersh, Why batch and user evaluations do not give the same results, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.225-231, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383992]
• 25: Andrew Turpin , Falk Scholer, User performance versus precision measures for simple search tasks, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148176]
• 26: Howard Turtle, Natural language vs. Boolean query evaluation: a comparison of retrieval performance, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.212-220, July 03-06, 1994, Dublin, Ireland
• 27: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 28: Ryen W. White , Ian Ruthven , Joemon M. Jose , C. J. Van Rijsbergen, Evaluating implicit feedback models using searcher simulations, ACM Transactions on Information Systems (TOIS), v.23 n.3, p.325-361, July 2005 [doi>10.1145/1080343.1080347]
• 29: W. John Wilbur , Leona Coffee, The effectiveness of document neighboring in search enhancement, Information Processing and Management: an International Journal, v.30 n.2, p.253-266, March-April 1994 [doi>10.1016/0306-4573(94)90068-X]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Measurement

Keywords:
find-similar, related article search