Testing and evaluating tag recommenders in a live system

ABSTRACT

The challenge to provide tag recommendations for collaborative tagging systems has attracted quite some attention of researchers lately. However, most research focused on the evaluation and development of appropriate methods rather than tackling the practical challenges of how to integrate recommendation methods into real tagging systems, record and evaluate their performance. In this paper we describe the tag recommendation framework we developed for our social bookmark and publication sharing system BibSonomy. With the intention to develop, test, and evaluate recommendation algorithms and supporting cooperation with researchers, we designed the framework to be easily extensible, open for a variety of methods, and usable independent from BibSonomy. Furthermore, this paper presents a first evaluation of two exemplarily deployed recommendation methods.

REFERENCES

• 1: 1] P. Basile, D. Gendarmi, F. Lanubile, and G. Semeraro. Recommending smart tags in a social bookmarking system. In Bridging the Gap between Semantic Web and Web 2.0 (SemNet 2007), pages 22--29, 2007.
• 2: R. Burke. Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction, 12(4):331--370, Nov. 2002.
• 3: A. Byde, H. Wan, and S. Cayzer. Personalized tag recommendations via tagging and content-based similarity metrics. In Proceedings of the International Conference on Weblogs and Social Media, March 2007.
• 4: D. Cosley, S. Lawrence, and D. M. Pennock. REFEREE: an open framework for practical testing of recommender systems using ResearchIndex. In VLDB '02: Proceedings of the 28th international conference on Very Large Data Bases, pages 35--46. VLDB Endowment, 2002.
• 5: A. Hotho, R. Jaschke, C. Schmitz, and G. Stumme. BibSonomy: A social bookmark and publication sharing system. In A. de Moor, S. Polovina, and H. Delugach, editors, Proceedings of the Conceptual Structures Tool Interoperability Workshop at the 14th Int. Conf. on Conceptual Structures, Aalborg, Denmark, July 2006. Aalborg University Press.
• 6: A. Hotho, R. Jaschke, C. Schmitz, and G. Stumme. Information retrieval in folksonomies: Search and ranking. In Y. Sure and J. Domingue, editors, The Semantic Web: Research and Applications, volume 4011 of Lecture Notes in Computer Science, pages 411--426, Heidelberg, June 2006. Springer.
• 7: R. Jaschke, L. Marinho, A. Hotho, L. Schmidt-Thieme, and G. Stumme. Tag recommendations in social bookmarking systems. AI Communications, 21(4):231--247, 2008.
• 8: G. Mishne. Autotag: a collaborative approach to automated tag assignment for weblog posts. In WWW '06: Proceedings of the 15th International Conference on World Wide Web, pages 953{954, New York, NY, USA, 2006. ACM Press.
• 9: S. Sood, S. Owsley, K. Hammond, and L. Birnbaum. TagAssist: Automatic tag suggestion for blog posts. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM 2007), 2007.
• 10: M. van Setten. Supporting people in finding information: hybrid recommender systems and goal-based structuring. PhD thesis, University of Twente, Enschede, The Netherlands, Dec. 2005.
• 11: M. Vojnovic, J. Cruise, D. Gunawardena, and P. Marbach. Ranking and suggesting tags in collaborative tagging applications. Technical Report MSR-TR-2007-06, Microsoft Research, 2007.
• 12: Z. Xu, Y. Fu, J. Mao, and D. Su. Towards the semantic web: Collaborative tag suggestions. In Proceedings of the Collaborative Web Tagging Workshop at the WWW 2006, Edinburgh, Scotland, May 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Design, Experimentation, Measurement

Keywords:
framework, social bookmarking, tag recommender