Evaluating expertise recommendations

ABSTRACT

Finding a person who has the expertise to solve a specific problem is an important application of recommender systems to a difficult organizational problem. Prior systems have made attempts to implement solutions to this problem, but few systems have undergone systematic user evaluation. This work describes a systematic evaluation of the Expertise Recommender (ER), a system that recommends people who are likely to have expertise in a specific problem. ER and the organizational context for which it was designed are described to provide a basis for understanding this evaluation. Prior to conducting the evaluation, a baseline experiment showed that people are relatively good at judging coworkers' expertise when given an appropriate context. This finding provides a way to demonstrate the effectiveness of ER by comparing ER's performance to ratings by coworkers. The evaluation, the design, and results are described in detail. The results suggest that the participants agree with the recommendations made by ER, and that ER significantly outperforms other expertise recommender systems when compared using similar metrics.

REFERENCES

• 1: Allen, T.J. Managing the Flow of Technology. MIT Press, Cambridge, 1977.
• 2: Marko Balabanović , Yoav Shoham, Fab: content-based, collaborative recommendation, Communications of the ACM, v.40 n.3, p.66-72, March 1997 [doi>10.1145/245108.245124]
• 3: John M. Carroll , Mary Beth Rosson, Getting around the task-artifact cycle: how to make claims and design by scenario, ACM Transactions on Information Systems (TOIS), v.10 n.2, p.181-212, April 1992 [doi>10.1145/146802.146834]
• 4: Ehrlich, K. and Cash, D., Turning Information into Knowledge: Information Finding as a Collaborative Activity. in Digital Libraries '94, (College Station, TX, 1994), 119 - 125.
• 5: Leonard N. Foner, Yenta: a multi-agent, referral-based matchmaking system, Proceedings of the first international conference on Autonomous agents, p.301-307, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267732]
• 6: G. W. Furnas , S. Deerwester , S. T. Dumais , T. K. Landauer , R. A. Harshman , L. A. Streeter , K. E. Lochbaum, Information retrieval using a singular value decomposition model of latent semantic structure, Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval, p.465-480, May 1988, Grenoble, France [doi>10.1145/62437.62487]
• 7: Glance, N.S., Aregui, D. and Dardenne, M., Making Recommender Systems Work for Organizations. in Practical Application of Intelligent Agents and Multi- Agents (PAAM'99), (London, UK, 1999).
• 8: David Goldberg , David Nichols , Brian M. Oki , Douglas Terry, Using collaborative filtering to weave an information tapestry, Communications of the ACM, v.35 n.12, p.61-70, Dec. 1992 [doi>10.1145/138859.138867]
• 9: Rebecca E. Grinter, Supporting articulation work using software configuration management systems, Computer Supported Cooperative Work, v.5 n.4, p.447-465, 1996 [doi>10.1007/BF00136714]
• 10: Will Hill , Larry Stead , Mark Rosenstein , George Furnas, Recommending and evaluating choices in a virtual community of use, Proceedings of the SIGCHI conference on Human factors in computing systems, p.194-201, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223929]
• 11: Kautz, H.A., Selman, B. and Shah, M. The Hidden Web. AI Magazine (Summer). (1997). 27 - 36.
• 12: Henry Kautz , Bart Selman , Mehul Shah, Referral Web: combining social networks and collaborative filtering, Communications of the ACM, v.40 n.3, p.63-65, March 1997 [doi>10.1145/245108.245123]
• 13: Barbara A. Kitchenham, Software Metrics: Measurement for Software Process Improvement, Blackwell Publishers, Inc., Cambridge, MA, 1996
• 14: Joseph A. Konstan , Bradley N. Miller , David Maltz , Jonathan L. Herlocker , Lee R. Gordon , John Riedl, GroupLens: applying collaborative filtering to Usenet news, Communications of the ACM, v.40 n.3, p.77-87, March 1997 [doi>10.1145/245108.245126]
• 15: Lutters, W.G., Ackerman, M.S., Boster, J.S. and McDonald, D.W., Mapping Knowledge Networks in Organizations: Creating a Knowledge Mapping Instrument. in Americas Conference on Information Systems, AMCIS'00, (Long Beach, CA, 2000), AIS, 2014-2018.
• 16: Mattox, D., Maybury, M. and Morey, D. Enterprise Expert and Knowledge Discovery. The MITRE Corporation (McLean, VA), September, 2000. http://www.mitre.org/support/papers/tech_papers99_00 /maybury_enterprise/maybury_enterprise.pdf
• 17: Maybury, M., D'Amore, R. and House, D. Awareness of Organizational Expertise. The MITRE Corporation (MacLean, VA), October, 2000. http://www.mitre.org/support/papers/tech_papers99_00 /maybury_awareness/maybury_awareness.pdf
• 18: David Wright Mcdonald , Mark S. Ackerman, Supporting nuance in groupware design: moving from naturalistic expertise location to expertise recommendation, 2000
• 19: David W. McDonald , Mark S. Ackerman, Expertise recommender: a flexible recommendation system and architecture, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.231-240, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358994]
• 20: David W. McDonald , Mark S. Ackerman, Just talk to me: a field study of expertise location, Proceedings of the 1998 ACM conference on Computer supported cooperative work, p.315-324, November 14-18, 1998, Seattle, Washington, United States [doi>10.1145/289444.289506]
• 21: Andreas Paepcke, Information needs in technical work settings and their implications for the design of computer tools, Computer Supported Cooperative Work, v.5 n.1, p.63-92, 1996
• 22: Paul Resnick , Neophytos Iacovou , Mitesh Suchak , Peter Bergstrom , John Riedl, GroupLens: an open architecture for collaborative filtering of netnews, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.175-186, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192905]
• 23: Martin Shepperd, Foundations of software measurement, Prentice Hall International (UK) Ltd., Hertfordshire, UK, 1995
• 24: Streeter, L.A. and Lochbaum, K.E., An Expert/Expert- Locating System Based on Automatic Representation of Semantic Structure. in Fourth Conference on Artificial Intelligence Applications, (San Diego, CA, 1988).
• 25: Streeter, L.A. and Lochbaum, K.E., Who Knows: A System Based on Automatic Representation of Semantic Structure. in RIAO '88, (Cambridge, MA, 1988), 380 - 388.
• 26: Loren G. Terveen , William C. Hill , Brian Amento , David McDonald , Josh Creter, Building task-specific interfaces to high volume conversational data, Proceedings of the SIGCHI conference on Human factors in computing systems, p.226-233, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258712]
• 27: Adriana Vivacqua , Henry Lieberman, Agents to assist in finding help, Proceedings of the SIGCHI conference on Human factors in computing systems, p.65-72, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332408]
• 28: Richard M. Young , Phil Barnard, The use of scenarios in human-computer interaction research: turbocharging the tortoise of cumulative science, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.291-296, April 05-09, 1987, Toronto, Ontario, Canada

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Computer-supported cooperative work

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Groupware
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Organizational design; Evaluation/methodology

General Terms:
Design, Experimentation, Human Factors, Management

Keywords:
CSCW, computer-supported cooperative work, expertise location, recommendation systems, user evaluation