SERF: integrating human recommendations with search

ABSTRACT

Today's university library has many digitally accessible resources, both indexes to content and considerable original content. Using off-the-shelf search technology provides a single point of access into library resources, but we have found that such full-text indexing technology is not entirely satisfactory for library searching.

In response to this, we report initial usage results from a prototype of an entirely new type of search engine - The System for Electronic Recommendation Filtering (SERF) - that we have designed and deployed for the Oregon State University (OSU) Libraries. SERF encourages users to enter longer and more informative queries, and collects ratings from users as to whether search results meet their information need or not. These ratings are used to make recommendations to later users with similar needs. Over time, SERF learns from the users what documents are valuable for what information needs.

In this paper, we focus on understanding whether such recommendations can increase other users' search efficiency and effectiveness in library website searching.

Based on examination of three months of usage as an alternative search interface available to all users of the Oregon State University Libraries website (http://osulibrary.oregonstate.edu/), we found strong evidence that the recommendations with human evaluation could increase the efficiency as well as effectiveness of the library website search process. Those users who received recommendations needed to examine fewer results, and recommended documents were rated much higher than documents returned by a traditional search engine.

REFERENCES

• 1: Peter Anick, Using terminological feedback for web search refinement: a log-based study, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860453]
• 2: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 3: N. J. Belkin , D. Kelly , G. Kim , J.-Y. Kim , H.-J. Lee , G. Muresan , M.-C. Tang , X.-J. Yuan , C. Cool, Query length in interactive information retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860474]
• 4: Boros, E., Kantor, P.B. and Neu, D.J. Pheromonic Representation of User Quests by Digital Structures. Proceedings of the 62nd American Society for Information Science (ASIS), (1999).
• 5: Burke, R., Hammond, K., Kulyukin, V., Lytinen, S., Tomuro, N. and Schoenberg, S. Natural Language Processing in the FAQ Finder System: Results and Prospects, in Working Notes from AAAI Spring Symposium on NLP on the WWW, (1997), 17--26
• 6: Cosley, D. Lawrence, S. and Pennock, D.M. REFEREE: An open framework for practical testing of recommender systems using ResearchIndex. Proceedings of the 28th VLDB Conference (2002)
• 7: DirectHit, http://www.directhit.com/
• 8: Ken Goldberg , Theresa Roeder , Dhruv Gupta , Chris Perkins, Eigentaste: A Constant Time Collaborative Filtering Algorithm, Information Retrieval, v.4 n.2, p.133-151, July 2001 [doi>10.1023/A:1011419012209]
• 9: Luis Gravano , Vasileios Hatzivassiloglou , Richard Lichtenstein, Categorizing web queries according to geographical locality, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956925]
• 10: Seikyung Jung , Juntae Kim , Jonathan L. Herlocker, Applying Collaborative Filtering for Efficient Document Search, Proceedings of the Web Intelligence, IEEE/WIC/ACM International Conference on (WI'04), p.640-643, September 20-24, 2004 [doi>10.1109/WI.2004.33]
• 11: Will Hill , Larry Stead , Mark Rosenstein , George Furnas, Recommending and evaluating choices in a virtual community of use, Proceedings of the SIGCHI conference on Human factors in computing systems, p.194-201, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223929]
• 12: Kantor, P.B., Boros, E., Melamed, B. and Menkov, V. The information Quest: A Dynamic Model of User's Information Needs. Proceedings of the 62nd Annual Meeting of the American Society for Information Science (ASIS), (1999).
• 13: Paul B. Kantor , Endre Boros , Benjamin Melamed , Vladimir Meñkov , Bracha Shapira , David J. Neu, Capturing human intelligence in the net, Communications of the ACM, v.43 n.8, p.112-115, Aug. 2000 [doi>10.1145/345124.345162]
• 14: MetaCrawler, http://www.metacrawler.com.
• 15: Sean M. McNee , Istvan Albert , Dan Cosley , Prateep Gopalkrishnan , Shyong K. Lam , Al Mamunur Rashid , Joseph A. Konstan , John Riedl, On the recommending of citations for research papers, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587096]
• 16: Vladimir Menkov , Dave Neu , Qin Shi, AntWorld: A Collaborative Web Search Tool, Proceedings of the Third International Workshop on Distributed Communities on the Web, p.13-22, June 19-21, 2000
• 17: Page L., Brin S., Montwani R. and Winograd T. The PageRank Citation Ranking: Bringing Order to the Web. Technical report, Standford University Database Group, (1998)
• 18: Porter, M.F. An Algorithm for Suffix Stripping. Program, 14, 3 (July, 1980), 130--137
• 19: Upendra Shardanand , Pattie Maes, Social information filtering: algorithms for automating “word of mouth”, Proceedings of the SIGCHI conference on Human factors in computing systems, p.210-217, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223931]
• 20: Amanda Spink , H. Cenk Ozmultu, Characteristics of question format web queries: an exploratory study, Information Processing and Management: an International Journal, v.38 n.4, p.453-471, July 2002 [doi>10.1016/S0306-4573(01)00042-5]
• 21: Ryen W. White , Joemon M. Jose , Ian Ruthven, A task-oriented study on the influencing effects of query-biased summarisation in web searching, Information Processing and Management: an International Journal, v.39 n.5, p.707-733, September 2003 [doi>10.1016/S0306-4573(02)00033-X]
• 22: Gui-Rong Xue , Hua-Jun Zeng , Zheng Chen , Wei-Ying Ma , Hong-Jiang Zhang , Chao-Jun Lu, Implicit link analysis for small web search, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860448]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
collaborative filtering, digital libraries, information retrieval, user studies, web search