Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.

REFERENCES

• 1: Charu C. Aggarwal , Joel L. Wolf , Kun-Lung Wu , Philip S. Yu, Horting hatches an egg: a new graph-theoretic approach to collaborative filtering, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.201-212, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312230]
• 2: Brian Amento , Loren Terveen , Will Hill , Deborah Hix , Robert Schulman, Experiments in social data mining: The TopicShop system, ACM Transactions on Computer-Human Interaction (TOCHI), v.10 n.1, p.54-85, March 2003 [doi>10.1145/606658.606661]
• 3: Brian Amento , Will Hill , Loren Terveen , Deborah Hix , Peter Ju, An empirical evaluation of user interfaces for topic management of Web sites, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.552-559, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303156]
• 4: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 5: Bailey, B. P., Gurak, L. J., and Konstan, J. A. 2001. An examination of trust production in computer-mediated exchange. In Proceedings of the 7th Conference on Human Factors and the Web (July).
• 6: Marko Balabanović , Yoav Shoham, Fab: content-based, collaborative recommendation, Communications of the ACM, v.40 n.3, p.66-72, March 1997 [doi>10.1145/245108.245124]
• 7: Chumki Basu , Haym Hirsh , William Cohen, Recommendation as classification: using social and content-based information in recommendation, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.714-720, July 1998, Madison, Wisconsin, United States
• 8: Daniel Billsus , Michael J. Pazzani, Learning Collaborative Information Filters, Proceedings of the Fifteenth International Conference on Machine Learning, p.46-54, July 24-27, 1998
• 9: Breese, J. S., Heckerman, D., and Kadie, C. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI-98). G. F. Cooper, and S. Moral, Eds. Morgan-Kaufmann, San Francisco, Calif., 43--52.
• 10: John Canny, Collaborative filtering with privacy via factor analysis, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564419]
• 11: Mark Claypool , David Brown , Phong Le , Makoto Waseda, Inferring User Interest, IEEE Internet Computing, v.5 n.6, p.32-39, November 2001 [doi>10.1109/4236.968829]
• 12: Cleverdon, C. and Kean, M. 1968. Factors Determining the Performance of Indexing Systems. Aslib Cranfield Research Project, Cranfield, England.
• 13: Cosley, D., Lam, S. K., Albert, I., Konstan, J. A., and Riedl, J. 2003. Is seeing believing? How recommender interfaces affect users' opinions. CHI Lett. 5.
• 14: Dahlen, B. J., Konstan, J. A., Herlocker, J. L., Good, N., Borchers, A., and Riedl, J. 1998. Jump-starting movielens: User benefits of starting a collaborative filtering system with "dead data". TR 98-017. University of Minnesota.
• 15: Pedro Domingos , Matt Richardson, Mining the network value of customers, Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, p.57-66, August 26-29, 2001, San Francisco, California [doi>10.1145/502512.502525]
• 16: David Goldberg , David Nichols , Brian M. Oki , Douglas Terry, Using collaborative filtering to weave an information tapestry, Communications of the ACM, v.35 n.12, p.61-70, Dec. 1992 [doi>10.1145/138859.138867]
• 17: Ken Goldberg , Theresa Roeder , Dhruv Gupta , Chris Perkins, Eigentaste: A Constant Time Collaborative Filtering Algorithm, Information Retrieval, v.4 n.2, p.133-151, July 2001 [doi>10.1023/A:1011419012209]
• 18: Nathaniel Good , J. Ben Schafer , Joseph A. Konstan , Al Borchers , Badrul Sarwar , Jon Herlocker , John Riedl, Combining collaborative filtering with personal agents for better recommendations, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.439-446, July 18-22, 1999, Orlando, Florida, United States
• 19: Hanley, J. A. and Mcneil, B. J. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29--36.
• 20: Harman, D. 1995. The TREC conferences. Hypertext---Information Retrieval---Multimedia: Synergieeffekte Elektronisher Informationssysteme. In Proceedings of HIM '95.
• 21: Stephen P. Harter, Variations in relevance assessments and the measurement of retrieval effectiveness, Journal of the American Society for Information Science, v.47 n.1, p.37-49, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.3.CO;2-I]
• 22: David Heckerman , David Maxwell Chickering , Christopher Meek , Robert Rounthwaite , Carl Kadie, Dependency networks for inference, collaborative filtering, and data visualization, The Journal of Machine Learning Research, 1, p.49-75, 9/1/2001 [doi>10.1162/153244301753344614]
• 23: Martin G. Helander , Thomas K. Landauer , P. Prabhu , Prasad V. Prabhu, Handbook of Human-Computer Interaction, Elsevier Science Inc., New York, NY, 1997
• 24: Jonathan L. Herlocker , Joseph A. Konstan , Al Borchers , John Riedl, An algorithmic framework for performing collaborative filtering, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.230-237, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312682]
• 25: Jonathan L. Herlocker , Joseph A. Konstan , John Riedl, Explaining collaborative filtering recommendations, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.241-250, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358995]
• 26: Jon Herlocker , Joseph A. Konstan , John Riedl, An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms, Information Retrieval, v.5 n.4, p.287-310, October 2002 [doi>10.1023/A:1020443909834]
• 27: Will Hill , Larry Stead , Mark Rosenstein , George Furnas, Recommending and evaluating choices in a virtual community of use, Proceedings of the SIGCHI conference on Human factors in computing systems, p.194-201, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223929]
• 28: Joseph A. Konstan , Bradley N. Miller , David Maltz , Jonathan L. Herlocker , Lee R. Gordon , John Riedl, GroupLens: applying collaborative filtering to Usenet news, Communications of the ACM, v.40 n.3, p.77-87, March 1997 [doi>10.1145/245108.245126]
• 29: Le, C. T., Lindren, B. R. 1995. Construction and comparison of two receiver operating characteristics curves derived from the same samples. Biom. J. 37, 869--877.
• 30: Linton, F., Charron, A., and Joy, D. 1998. OWL: A recommender system for organziation-wide learning. In Proceedings of the 1998 Workshop on Recommender Systems 65--69.
• 31: David W. McDonald, Evaluating expertise recommendations, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500319]
• 32: Sean M. McNee , Istvan Albert , Dan Cosley , Prateep Gopalkrishnan , Shyong K. Lam , Al Mamunur Rashid , Joseph A. Konstan , John Riedl, On the recommending of citations for research papers, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587096]
• 33: Bradley N. Miller , Istvan Albert , Shyong K. Lam , Joseph A. Konstan , John Riedl, MovieLens unplugged: experiences with an occasionally connected recommender system, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604094]
• 34: Miller, B. N., Riedl, J., and Konstan, J. A. 1997. Experiences with GroupLens: Making Usenet useful again. In Proceedings of the 1997 Usenix Technical Conference.
• 35: Bamshad Mobasher , Honghua Dai , Tao Luo , Miki Nakagawa, Effective personalization based on association rule discovery from web usage data, Proceedings of the 3rd international workshop on Web information and data management, November 09-01, 2001, Atlanta, Georgia, USA [doi>10.1145/502932.502935]
• 36: Masahiro Morita , Yoichi Shinoda, Information filtering based on user behavior analysis and best match text retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.272-281, July 03-06, 1994, Dublin, Ireland
• 37: Mui, L., Ang, C., and Mohtashemi, M. 2001. A Probabilistic Model for Collaborative Sanctioning. Technical Memorandum 617. MIT LCS.
• 38: William M. Newman, Better or just different? On the benefits of designing interactive systems in terms of critical parameters, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.239-245, August 18-20, 1997, Amsterdam, The Netherlands [doi>10.1145/263552.263615]
• 39: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 40: David M. Pennock , Eric Horvitz , Steve Lawrence , C. Lee Giles, Collaborative Filtering by Personality Diagnosis: A Hybrid Memory and Model-Based Approach, Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, p.473-480, June 30-July 03, 2000
• 41: Al Mamunur Rashid , Istvan Albert , Dan Cosley , Shyong K. Lam , Sean M. McNee , Joseph A. Konstan , John Riedl, Getting to know you: learning new user preferences in recommender systems, Proceedings of the 7th international conference on Intelligent user interfaces, January 13-16, 2002, San Francisco, California, USA [doi>10.1145/502716.502737]
• 42: P. Krishna Reddy , Masaru Kitsuregawa , P. Sreekanth , S. Srinivasa Rao, A Graph Based Approach to Extract a Neighborhood Customer Community for Collaborative Filtering, Proceedings of the Second International Workshop on Databases in Networked Information Systems, p.188-200, December 16-18, 2002
• 43: Paul Resnick , Neophytos Iacovou , Mitesh Suchak , Peter Bergstrom , John Riedl, GroupLens: an open architecture for collaborative filtering of netnews, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.175-186, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192905]
• 44: Paul Resnick , Hal R. Varian, Recommender systems, Communications of the ACM, v.40 n.3, p.56-58, March 1997 [doi>10.1145/245108.245121]
• 45: Rogers, S. C. 2001. Marketing Strategies, Tactics, and Techniques : A handbook for practitioners. Quorum Books, Westport, Conn.
• 46: Badrul Sarwar , George Karypis , Joseph Konstan , John Riedl, Analysis of recommendation algorithms for e-commerce, Proceedings of the 2nd ACM conference on Electronic commerce, p.158-167, October 17-20, 2000, Minneapolis, Minnesota, United States [doi>10.1145/352871.352887]
• 47: Sarwar, B. M., Karypis, G., Konstan, J. A., and Riedl, J. 2000b. Application of dimensionality reduction in recommender system--A case study. In Proceedings of the ACM WebKDD 2000 Web Mining for E-Commerce Workshop.
• 48: Badrul Sarwar , George Karypis , Joseph Konstan , John Reidl, Item-based collaborative filtering recommendation algorithms, Proceedings of the 10th international conference on World Wide Web, p.285-295, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372071]
• 49: Badrul M. Sarwar , Joseph A. Konstan , Al Borchers , Jon Herlocker , Brad Miller , John Riedl, Using filtering agents to improve prediction quality in the GroupLens research collaborative filtering system, Proceedings of the 1998 ACM conference on Computer supported cooperative work, p.345-354, November 14-18, 1998, Seattle, Washington, United States [doi>10.1145/289444.289509]
• 50: J. Ben Schafer , Joseph A. Konstan , John Riedl, Meta-recommendation systems: user-controlled integration of diverse recommendations, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA [doi>10.1145/584792.584803]
• 51: Andrew I. Schein , Alexandrin Popescul , Lyle H. Ungar , David M. Pennock, Methods and metrics for cold-start recommendations, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564421]
• 52: Andrew I. Schein , Alexandrin Popescul , Lyle H. Ungar , David M. Pennock, Methods and metrics for cold-start recommendations, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564421]
• 53: Upendra Shardanand , Pattie Maes, Social information filtering: algorithms for automating “word of mouth”, Proceedings of the SIGCHI conference on Human factors in computing systems, p.210-217, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223931]
• 54: Rashmi Sinha , Kirsten Swearingen, The role of transparency in recommender systems, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506619]
• 55: Swearingen, K. and Sinha, R. 2001. Beyond algorithms: An HCI perspective on recommender systems. In Proceedings of the SIGIR 2001 Workshop on Recommender Systems.
• 56: Swets, J. A. 1963. Information retrieval systems. Science 141, 245--250.
• 57: Swets, J. A. 1969. Effectiveness of information retrieval methods. Amer. Doc. 20, 72--89.
• 58: Andrew H. Turpin , William Hersh, Why batch and user evaluations do not give the same results, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.225-231, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383992]
• 59: Voorhees, E. M. and Harman, D. K. 1999. Overview of the seventh Text REtrieval Conference (TREC-7). In NIST Special Publication 500-242 (July), E. M. Voorhees, and D. K. Harman, Eds. NIST, 1--24.
• 60: Alan Wexelblat , Pattie Maes, Footprints: history-rich tools for information foraging, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.270-277, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303060]
• 61: Whittaker, S., Terveen, L. G., and Nardi, B. 2000. Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI. Human-Computer Interact. 15, 2-3 (Sept.), 75--106.
• 62: Y. Y. Yao, Measuring retrieval effectiveness based on user preference of documents, Journal of the American Society for Information Science, v.46 n.2, p.133-145, March 1995 [doi>10.1002/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement, Performance

Keywords:
Collaborative filtering, evaluation, metrics, recommender systems