On the recommending of citations for research papers

ABSTRACT

Collaborative filtering has proven to be valuable for recommending items in many different domains. In this paper, we explore the use of collaborative filtering to recommend research papers, using the citation web between papers to create the ratings matrix. Specifically, we tested the ability of collaborative filtering to recommend citations that would be suitable additional references for a target research paper. We investigated six algorithms for selecting citations, evaluating them through offline experiments against a database of over 186,000 research papers contained in ResearchIndex. We also performed an online experiment with over 120 users to gauge user opinion of the effectiveness of the algorithms and of the utility of such recommendations for common research tasks. We found large differences in the accuracy of the algorithms in the offline experiment, especially when balanced for coverage. In the online experiment, users felt they received quality recommendations, and were enthusiastic about the idea of receiving recommendations in this domain.

REFERENCES

• 1: Breese, J., Heckerman, D., and Kadie, C. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In Proc. UAI 98, Madison, 1998, 43--52.
• 2: Kurt D. Bollacker , Steve Lawrence , C. Lee Giles, Discovering Relevant Scientific Literature on the Web, IEEE Intelligent Systems, v.15 n.2, p.42-47, March 2000 [doi>10.1109/5254.850826]
• 3: Egghe, L., and Rousseau, R. Introduction to Informetrics. Elsevier, Amsterdam, 1990.
• 4: Nir Friedman , Dan Geiger , Moises Goldszmidt, Bayesian Network Classifiers, Machine Learning, v.29 n.2-3, p.131-163, Nov./Dec. 1997 [doi>10.1023/A:1007465528199]
• 5: Garfield, E. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities. Wiley, New York, 1979.
• 6: Ken Goldberg , Theresa Roeder , Dhruv Gupta , Chris Perkins, Eigentaste: A Constant Time Collaborative Filtering Algorithm, Information Retrieval, v.4 n.2, p.133-151, July 2001 [doi>10.1023/A:1011419012209]
• 7: Jonathan L. Herlocker , Joseph A. Konstan , Al Borchers , John Riedl, An algorithmic framework for performing collaborative filtering, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.230-237, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312682]
• 8: Will Hill , Larry Stead , Mark Rosenstein , George Furnas, Recommending and evaluating choices in a virtual community of use, Proceedings of the SIGCHI conference on Human factors in computing systems, p.194-201, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223929]
• 9: Karypis, G. SUGGEST Top-N Recommendation Engine. Available for download from http://www.cs.umn.edu/ karpyis/suggest/.
• 10: Henry Kautz , Bart Selman , Mehul Shah, Referral Web: combining social networks and collaborative filtering, Communications of the ACM, v.40 n.3, p.63-65, March 1997 [doi>10.1145/245108.245123]
• 11: Steve Lawrence , Kurt Bollacker , C. Lee Giles, Indexing and retrieval of scientific literature, Proceedings of the eighth international conference on Information and knowledge management, p.139-146, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.319970]
• 12: Steve Lawrence , C. Lee Giles , Kurt Bollacker, Digital Libraries and Autonomous Citation Indexing, Computer, v.32 n.6, p.67-71, June 1999 [doi>10.1109/2.769447]
• 13: Newman, M. E. J. Scientific collaboration networks: I. Network construction and fundamental results. Phys. Rev. E 64, 016131, 2001.
• 14: Al Mamunur Rashid , Istvan Albert , Dan Cosley , Shyong K. Lam , Sean M. McNee , Joseph A. Konstan , John Riedl, Getting to know you: learning new user preferences in recommender systems, Proceedings of the 7th international conference on Intelligent user interfaces, January 13-16, 2002, San Francisco, California, USA [doi>10.1145/502716.502737]
• 15: Paul Resnick , Neophytos Iacovou , Mitesh Suchak , Peter Bergstrom , John Riedl, GroupLens: an open architecture for collaborative filtering of netnews, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.175-186, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192905]
• 16: Badrul Sarwar , George Karypis , Joseph Konstan , John Reidl, Item-based collaborative filtering recommendation algorithms, Proceedings of the 10th international conference on World Wide Web, p.285-295, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372071]
• 17: Scott, J. Social Network Analysis: A Handbook, 2nd Edition. Sage Publications, London, 2000.
• 18: Upendra Shardanand , Pattie Maes, Social information filtering: algorithms for automating “word of mouth”, Proceedings of the SIGCHI conference on Human factors in computing systems, p.210-217, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223931]
• 19: Loren Terveen , Will Hill , Brian Amento , David McDonald , Josh Creter, PHOAKS: a system for sharing recommendations, Communications of the ACM, v.40 n.3, p.59-62, March 1997 [doi>10.1145/245108.245122]
• 20: Allison Woodruff , Rich Gossweiler , James Pitkow , Ed H. Chi , Stuart K. Card, Enhancing a digital book with a reading recommender, Proceedings of the SIGCHI conference on Human factors in computing systems, p.153-160, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332419]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.1 Systems and Information Theory
          Subjects: Value of information
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Information networks
      H.3.7 Digital Libraries
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Collaborative computing

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
ResearchIndex, citation graphs, collaborative filtering, digital libraries, recommender systems, social networks