An iterative rating method: application to web-based conference management

ABSTRACT

Given a large set of items and a set of users, we consider the problem of collecting user preferences - or ratings - on items. The paper describes a simple method which provides an approximate solution to the problem without requiring each user to rate each item. The method relies on an iterative process. Each step, or ballot, requires each user to rate a sample of the items. A collaborative filtering algorithm is then performed to predict the missing ratings as well as their level of confidence (which is initially 0). Perfoming a new ballot allows to improve the accuracy of predictions. The administrator of the system is responsible for stopping the iteration when a satisfactory level is reached.We apply this method to the assignment of reviewers to papers prior to the review phase of conference management, and describe its implementation in the MYREVIEW web-based system.

REFERENCES

• 1: Marko Balabanović , Yoav Shoham, Fab: content-based, collaborative recommendation, Communications of the ACM, v.40 n.3, p.66-72, March 1997 [doi>10.1145/245108.245124]
• 2: The BYU Paper Review System. Web Site, 1999. http://dagwood.cs.byu.edu/PaperReview.
• 3: The Microsoft Conference Management Toolkit. Web Site, 2002. http://cmt.research.microsoft.com/cmt/.
• 4: The ConfMan Software. Web Site, 1999. http://www.ifi.uio.no/confman/demo/.
• 5: The CyberChair Software. Web Site, 1997. http://cyberchair.org/.
• 6: CyberChair: a Web-based Groupware Application to Facilitate the paper Reviewing Process. Available at www.cyberchair.org, 2001.
• 7: Jonathan L. Herlocker , Joseph A. Konstan , Al Borchers , John Riedl, An algorithmic framework for performing collaborative filtering, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.230-237, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312682]
• 8: Will Hill , Larry Stead , Mark Rosenstein , George Furnas, Recommending and evaluating choices in a virtual community of use, Proceedings of the SIGCHI conference on Human factors in computing systems, p.194-201, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223929]
• 9: J. E. Hopcroft and R. M. Karp. An n2.5 Algorithm for Maximum Weighting in Bipartite Graphs. SIAM Journal on computing, 2:225--231, 1973.
• 10: J. S.Breese, D. Heckerman, and C. Kadie. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In Proc. Intl. Conf. on Uncertainty in Artificial Intelligence, pages 43--52, 1998.
• 11: Joseph A. Konstan , Bradley N. Miller , David Maltz , Jonathan L. Herlocker , Lee R. Gordon , John Riedl, GroupLens: applying collaborative filtering to Usenet news, Communications of the ACM, v.40 n.3, p.77-87, March 1997 [doi>10.1145/245108.245126]
• 12: H. W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, 2:83--97, 1955.
• 13: Paul Resnick , Hal R. Varian, Recommender systems, Communications of the ACM, v.40 n.3, p.56-58, March 1997 [doi>10.1145/245108.245121]
• 14: Badrul Sarwar , George Karypis , Joseph Konstan , John Riedl, Analysis of recommendation algorithms for e-commerce, Proceedings of the 2nd ACM conference on Electronic commerce, p.158-167, October 17-20, 2000, Minneapolis, Minnesota, United States [doi>10.1145/352871.352887]
• 15: Upendra Shardanand , Pattie Maes, Social information filtering: algorithms for automating “word of mouth”, Proceedings of the SIGCHI conference on Human factors in computing systems, p.210-217, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223931]
• 16: L. H. Ungar and D. P. Foster. Clustering Methods for Collaborative Filtering. In Proc. Intl. Workshop on Recommender Systems, pages 1--10, 1998.