Automating the assignment of submitted manuscripts to reviewers

ABSTRACT

The 117 manuscripts submitted for the Hypertext '91 conference were assigned to members of the review committee, using a variety of automated methods based on information retrieval principles and Latent Semantic Indexing. Fifteen reviewers provided exhaustive ratings for the submitted abstracts, indicating how well each abstract matched their interests. The automated methods do a fairly good job of assigning relevant papers for review, but they are still somewhat poorer than assignments made manually by human experts and substantially poorer than an assignment perfectly matching the reviewers' own ranking of the papers. A new automated assignment method called “n of 2n” achieves better performance than human experts by sending reviewers more papers than they actually have to review and then allowing them to choose part of their review load themselves.

REFERENCES

• 1: Robert M. Akscyn, Hypertext compendium, ACM Press, New York, NY, 1992
• 2: Applegate, D., and Cook, W. (1992). Solving large-scale matching problems. Paper to appear.
• 3: Paul R. Cohen , Rick Kjeldsen, Information retrieval by constrained spreading activation in semantic networks, Information Processing and Management: an International Journal, v.23 n.4, p.255-268, July 1987 [doi>10.1016/0306-4573(87)90017-3]
• 4: Jane K. Cullum , Ralph A. Willoughby, Lanczos Algorithms for Large Symmetric Eigenvalue Computations, Vol. 1, Society for Industrial and Applied Mathematics, Philadelphia, PA, 2002
• 5: Deerwester, S., Dumais, S.T., Landauer, T.K., Furnas, G.W., and Harshman, R.A. (1990). Indexing by latent semantic analysis. Journal of the Society for Information Science 41, 6, 391-407.
• 6: Dumais, S.T. (1991). improving the retrieval of information from external sources. Behavior Research Methods, Instruments and Computers, 23, 2, 229-236.
• 7: Edmonds, J. (1965). Maximum matching and a polyhedron with 0,1 - vertices. Journal of Research of the National Bureau of Standards 69B, 125-130.
• 8: G. W. Furnas, Generalized fisheye views, Proceedings of the SIGCHI conference on Human factors in computing systems, p.16-23, April 13-17, 1986, Boston, Massachusetts, United States
• 9: Fumas, G.W. (1991 ). Personal communication.
• 10: G. W. Furnas , S. Deerwester , S. T. Dumais , T. K. Landauer , R. A. Harshman , L. A. Streeter , K. E. Lochbaum, Information retrieval using a singular value decomposition model of latent semantic structure, Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval, p.465-480, May 1988, Grenoble, France [doi>10.1145/62437.62487]
• 11: Kane-Esrig, Y., Casella, G., Dumais, S.T., and Streeter, L.A. (1989). Ranking documents for retrieval by modeling of a relevance density. In Proceedings of the 12th IRIS Conference (Information System Research Seminar In Scandinavia), Skagen, Denmark, August 1989.
• 12: Lesk, M.E., and Salton, G. (1969). Relevance assessments and retrieval system evaluation, information Storage and Retrieval 4, 343-359.
• 13: Jakob Nielsen, Hypertext and hypermedia, Academic Press Professional, Inc., San Diego, CA, 1990
• 14: Gary Perlman, The HCI Bibliography project, ACM SIGCHI Bulletin, v.23 n.3, p.15-20, July 1991 [doi>10.1145/126505.126507]
• 15: Roy Rada, Hypertext: from text to Expertext, McGraw-Hill, Inc., New York, NY, 1992
• 16: Gerard Salton , Chris Buckley, Automatic text structuring and retrieval-experiments in automatic encyclopedia searching, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-30, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122863]
• 17: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 18: Ben Shneiderman , Greg Kearsley, Hypertext hands-on—an introduction to a new way of organizing and accessing information, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors; Human information processing
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Human Factors