Don't look stupid: avoiding pitfalls when recommending research papers

ABSTRACT

If recommenders are to help people be more productive, they need to support a wide variety of real-world information seeking tasks, such as those found when seeking research papers in a digital library. There are many potential pitfalls, including not knowing what tasks to support, generating recommendations for the wrong task, or even failing to generate any meaningful recommendations whatsoever. We posit that different recommender algorithms are better suited to certain information seeking tasks. In this work, we perform a detailed user study with over 130 users to understand these differences between recommender algorithms through an online survey of paper recommendations from the ACM Digital Library. We found that pitfalls are hard to avoid. Two of our algorithms generated 'atypical' recommendations recommendations that were unrelated to their input baskets. Users reacted accordingly, providing strong negative results for these algorithms. Results from our 'typical' algorithms show some qualitative differences, but since users were exposed to two algorithms, the results may be biased. We present a wide variety of results, teasing out differences between algorithms. Finally, we succinctly summarize our most striking results as "Don't Look Stupid" in front of users.

REFERENCES

• 1: J.S. Breese, D. Heckerman and C. Kadie, "Empirical Analysis of Predictive Algorithms for Collaborative Filtering", in Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 43--52, 1998.
• 2: John Browning , David J. Miller, A Maximum Entropy Approach for Collaborative Filtering, Journal of VLSI Signal Processing Systems, v.37 n.2-3, p.199-209, June-July 2004 [doi>10.1023/B:VLSI.0000027485.11890.15]
• 3: Robin Burke, Hybrid Recommender Systems: Survey and Experiments, User Modeling and User-Adapted Interaction, v.12 n.4, p.331-370, November 2002 [doi>10.1023/A:1021240730564]
• 4: John Canny, Collaborative filtering with privacy via factor analysis, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564419]
• 5: D.O. Case, Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior, San Diego: Academic Press, 2002, pp. 350.
• 6: P. Domingos and M. Pazzani, "Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier", in Proc. of the 13th International Conference on Machine Learning (ICML 96), pp. 105--112, 1996.
• 7: Ken Goldberg , Theresa Roeder , Dhruv Gupta , Chris Perkins, Eigentaste: A Constant Time Collaborative Filtering Algorithm, Information Retrieval, v.4 n.2, p.133-151, July 2001 [doi>10.1023/A:1011419012209]
• 8: Nathaniel Good , J. Ben Schafer , Joseph A. Konstan , Al Borchers , Badrul Sarwar , Jon Herlocker , John Riedl, Combining collaborative filtering with personal agents for better recommendations, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.439-446, July 18-22, 1999, Orlando, Florida, United States
• 9: Jonathan L. Herlocker , Joseph A. Konstan , Loren G. Terveen , John T. Riedl, Evaluating collaborative filtering recommender systems, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.5-53, January 2004 [doi>10.1145/963770.963772]
• 10: Jonathan L. Herlocker , Joseph A. Konstan , Al Borchers , John Riedl, An algorithmic framework for performing collaborative filtering, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.230-237, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312682]
• 11: Thomas Hofmann, Latent semantic models for collaborative filtering, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.89-115, January 2004 [doi>10.1145/963770.963774]
• 12: Zan Huang , Hsinchun Chen , Daniel Zeng, Applying associative retrieval techniques to alleviate the sparsity problem in collaborative filtering, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.116-142, January 2004 [doi>10.1145/963770.963775]
• 13: I. Im and A. Hars, "Finding Information just for You: Knowledge Reuse using Collaborative Filtering Systems", in Proc. of the Twenty-Second International Conference on Information Systems, pp. 349--360, 2001.
• 14: G. Karypis, "SUGGEST Top-N Recommendation Engine", http://www-users.cs.umn.edu/~karypis/suggest/, 2000.
• 15: Joseph A. Konstan , Bradley N. Miller , David Maltz , Jonathan L. Herlocker , Lee R. Gordon , John Riedl, GroupLens: applying collaborative filtering to Usenet news, Communications of the ACM, v.40 n.3, p.77-87, March 1997 [doi>10.1145/245108.245126]
• 16: C.C. Kuhlthau, Seeking Meaning: A Process Approach to Library and Information Services, Westport, CT: Libraries Unlimited, 2004, pp. 247.
• 17: A. K. McCallum, "Bow: A Toolkit for Statistical Language Modeling, Text Retrieval, Classification and Clustering", http://www-2.cs.cmu.edu/mccallum/bow/, 1996.
• 18: S.M. McNee. Meeting User Information Needs in Recommender Systems. Ph.D. Dissertation, University of Minnesota. 2006.
• 19: S.M. McNee, J. Riedl and J.A. Konstan, "Being Accurate is Not enough: How Accuracy Metrics have Hurt
• 20: Sean M. McNee , John Riedl , Joseph A. Konstan, Making recommendations better: an analytic model for human-recommender interaction, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125660]
• 21: Sean M. McNee , Istvan Albert , Dan Cosley , Prateep Gopalkrishnan , Shyong K. Lam , Al Mamunur Rashid , Joseph A. Konstan , John Riedl, On the recommending of citations for research papers, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587096]
• 22: Prem Melville , Raymod J. Mooney , Ramadass Nagarajan, Content-boosted collaborative filtering for improved recommendations, Eighteenth national conference on Artificial intelligence, p.187-192, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 23: National Institutes of Health (NIH), "Entrez PubMed", http://www.ncbi.nlm.nih.gov/entrez/, 2006.
• 24: Peter Pirolli, Computational models of information scent-following in a very large browsable text collection, Proceedings of the SIGCHI conference on Human factors in computing systems, p.3-10, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258558]
• 25: M. F. Porter, An algorithm for suffix stripping, Readings in information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 26: Paul Resnick , Neophytos Iacovou , Mitesh Suchak , Peter Bergstrom , John Riedl, GroupLens: an open architecture for collaborative filtering of netnews, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.175-186, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192905]
• 27: Badrul Sarwar , George Karypis , Joseph Konstan , John Reidl, Item-based collaborative filtering recommendation algorithms, Proceedings of the 10th international conference on World Wide Web, p.285-295, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372071]
• 28: Upendra Shardanand , Pattie Maes, Social information filtering: algorithms for automating “word of mouth”, Proceedings of the SIGCHI conference on Human factors in computing systems, p.210-217, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223931]
• 29: Craig Silverstein , Hannes Marais , Monika Henzinger , Michael Moricz, Analysis of a very large web search engine query log, ACM SIGIR Forum, v.33 n.1, p.6-12, Fall 1999 [doi>10.1145/331403.331405]
• 30: R.S. Taylor, "Question-Negotiation and Information Seeking in Libraries", College and Research Libraries, vol. 29pp. 178--194, May, 1968.
• 31: Roberto Torres , Sean M. McNee , Mara Abel , Joseph A. Konstan , John Riedl, Enhancing digital libraries with TechLens+, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996402]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models; Relevance feedback

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
collaborative filtering, content-based filtering, digital libraries, human-recommender interaction, information seeking, personalization, recommender systems