An algorithm for automated rating of reviewers

ABSTRACT

The current system for scholarly information dissemination may be amen able to significant improvement. In particular, going from the current system of journal publication to one of self-distributed documents offers significant cost and timeliness advantages. A major concern with such alternatives is how to provide the value currently afforded by the peer review system.Here we propose a mechanism that could plausibly supply such value. In the peer review system, papers are judged meritorious if good reviewers give them good reviews. In its place, we propose a collaborative filtering algorithm which automatically rates reviewers, and incorporates the quality of the reviewer into the metric of merit for the paper. Such a system seems to provide all the benefits of the current peer review system, while at the same time being much more flexible.We have implemented a number of parameterized variations of this algorithm, and tested them on data available from a quite different application. Our initial experiments suggest that the algorithm is in fact ranking reviewers reasonably.

REFERENCES

• 1: ACM-SIGIR 1999 Workshopon Recommender Systems: Algorithms and Evaluation. http://www.csee.umbc.edu/ian/sigir99- rec/summary.html.
• 2: The Berkeley Electronic Press. http://www.bepress.com/.
• 3: CiteSeer.http://citeseer.nj.nec.com.
• 4: Collaborative filtering. http://www.sims.ber eley.edu/resources/collab/.
• 5: The UC Berkeley Digital Library Project. http://elib.cs.berkeley.edu.
• 6: C.Avery, P. Resnick, and R. Zeckhauser. The Market for Evaluations. American Economic Review 89(3):564-584,1999.
• 7: Marko Balabanović , Yoav Shoham, Fab: content-based, collaborative recommendation, Communications of the ACM, v.40 n.3, p.66-72, March 1997 [doi>10.1145/245108.245124]
• 8: J. Canny. Personal communication with the authors.
• 9: David Goldberg , David Nichols , Brian M. Oki , Douglas Terry, Using collaborative filtering to weave an information tapestry, Communications of the ACM, v.35 n.12, p.61-70, Dec. 1992 [doi>10.1145/138859.138867]
• 10: Jonathan L. Herlocker , Joseph A. Konstan , Al Borchers , John Riedl, An algorithmic framework for performing collaborative filtering, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.230-237, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312682]
• 11: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Proceedings of the ninth annual ACM-SIAM symposium on Discrete algorithms, p.668-677, January 25-27, 1998, San Francisco, California, United States
• 12: G. Z. A. Moukas and P. Maes. Collaborative Reputation Mechanisms in Electronic Mar etplaces. In Proceedings of the 32nd Hawaii International Conference on System Sciences 1999.
• 13: Thomas A. Phelps , Robert Wilensky, Multivalent documents, Communications of the ACM, v.43 n.6, p.82-90, June 2000 [doi>10.1145/336460.336480]
• 14: James Rucker , Marcos J. Polanco, Siteseer: personalized navigation for the Web, Communications of the ACM, v.40 n.3, p.73-76, March 1997 [doi>10.1145/245108.245125]
• 15: J. Ben Schafer , Joseph Konstan , John Riedi, Recommender systems in e-commerce, Proceedings of the 1st ACM conference on Electronic commerce, p.158-166, November 03-05, 1999, Denver, Colorado, United States [doi>10.1145/336992.337035]
• 16: R. Smith. Opening up BMJ peer review. BMJ, 318:23-27, 1999.
• 17: C. Tenopir and D. W. King. Trends in scientific scholarly journal publishing in the United States. Journal of Scholarly Publishing 28:135-170, 1997.
• 18: Loren Terveen , Will Hill , Brian Amento , David McDonald , Josh Creter, PHOAKS: a system for sharing recommendations, Communications of the ACM, v.40 n.3, p.59-62, March 1997 [doi>10.1145/245108.245122]
• 19: H. R. Varian. The Future of Electronic Journals. Technology and Scholarly Communication 1999.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING

General Terms:
Algorithms, Design, Documentation, Experimentation, Human Factors, Management, Measurement, Performance, Theory