Learning to recognize valuable tags

ABSTRACT

Many websites use tags as a mechanism for improving item metadata through collective user effort. Users of tagging systems often apply far more tags to an item than a system can display. These tags can range in quality from tags that capture a key facet of an item, to those that are subjective, irrelevant, or misleading. In this paper we explore tag selection algorithms that choose the tags that sites display. Based on 225,000 ratings and survey responses, we conduct offline analyses of 21 tag selection algorithms. We select the three best performing algorithms from our offline analysis, and deploy them live on the MovieLens website to 5,695 users for three months. Based on our results, we offer tagging system designers advice about tag selection algorithms.

REFERENCES

• 1: Morgan Ames , Mor Naaman, Why we tag: motivations for annotation in mobile and online media, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240772]
• 2: Andrew Arnt , Shlomo Zilberstein, Learning to Perform Moderation in Online Forums, Proceedings of the 2003 IEEE/WIC International Conference on Web Intelligence, p.637, October 13-17, 2003
• 3: Marko Balabanović , Yoav Shoham, Fab: content-based, collaborative recommendation, Communications of the ACM, v.40 n.3, p.66-72, March 1997 [doi>10.1145/245108.245124]
• 4: J. Besag and P. J. Green. Spatial statistics and bayesian computation. Journal of the Royal Statistical Society. Series B (Methodological), 55(1):25--37, 1993.
• 5: Vincent Bouthors , Olivier Dedieu, Pharos, a Collaborative Infrastructure for Web Knowledge Sharing, Proceedings of the Third European Conference on Research and Advanced Technology for Digital Libraries, p.215-233, September 22-24, 1999
• 6: Bradley P. Carlin , Thomas A. Louis, BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS, Statistics and Computing, v.7 n.2, p.153-154, 1997 [doi>10.1023/A:1018577817064]
• 7: C. Cattuto, V. Loreto, and L. Pietronero. Semiotic dynamics in online social communities. In The European Physical Journal C (accepted). Springer-Verlag, 2006.
• 8: Dan Cosley , Dan Frankowski , Sara Kiesler , Loren Terveen , John Riedl, How oversight improves member-maintained communities, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1054975]
• 9: Dan Cosley , Dan Frankowski , Loren Terveen , John Riedl, Using intelligent task routing and contribution review to help communities build artifacts of lasting value, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124928]
• 10: A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis, Second Edition. Chapman & Hall/CRC, July 2003.
• 11: N. Glance, D. Arregui, and M. Dardenne. Knowledge pump: Supporting the flow and use of knowledge. In Information Technology for Knowledge Management. Springer-Verlag, 1998.
• 12: Scott A. Golder , Bernardo A. Huberman, Usage patterns of collaborative tagging systems, Journal of Information Science, v.32 n.2, p.198-208, April 2006 [doi>10.1177/0165551506062337]
• 13: Jonathan L. Herlocker , Joseph A. Konstan , Loren G. Terveen , John T. Riedl, Evaluating collaborative filtering recommender systems, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.5-53, January 2004 [doi>10.1145/963770.963772]
• 14: Thorsten Joachims, Training linear SVMs in linear time, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA [doi>10.1145/1150402.1150429]
• 15: Cliff Lampe , Paul Resnick, Slash(dot) and burn: distributed moderation in a large online conversation space, Proceedings of the SIGCHI conference on Human factors in computing systems, p.543-550, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985761]
• 16: G. MacGregor and E. McCulloch. Collaborative tagging as a knowledge organisation and resource discovery tool. Library View (accepted), 55(5), 2006.
• 17: M. M. Markus Weimer, Iryna Gurevych. Automatically assesing the post quality in online discussions on software. In Proceedings of the Association for Computational Linguistics, 2007.
• 18: David Millen , Jonathan Feinberg , Bernard Kerr, Social Bookmarking in the Enterprise, Queue, v.3 n.9, November 2005 [doi>10.1145/1105664.1105676]
• 19: Shilad Sen , F. Maxwell Harper , Adam LaPitz , John Riedl, The quest for quality tags, Proceedings of the 2007 international ACM conference on Supporting group work, November 04-07, 2007, Sanibel Island, Florida, USA [doi>10.1145/1316624.1316678]
• 20: Shilad Sen , Shyong K. Lam , Al Mamunur Rashid , Dan Cosley , Dan Frankowski , Jeremy Osterhouse , F. Maxwell Harper , John Riedl, tagging, communities, vocabulary, evolution, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada [doi>10.1145/1180875.1180904]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Collaborative computing

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Design, Experimentation, Human Factors

Keywords:
moderation, tagging, user interfaces