Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Automatic image annotation using supervised learning is performed by concept classifiers trained on labelled example images. This work proposes the use of clickthrough data collected from search logs as a source for the automatic generation of concept training data, thus avoiding the expensive manual annotation effort. We investigate and evaluate this approach using a collection of 97,628 photographic images. The results indicate that the contribution of search log based training data is positive; in particular, the combination of manual and automatically generated training data outperforms the use of manual data alone. It is therefore possible to use clickthrough data to perform large-scale image annotation with little manual annotation effort or, depending on performance, using only the automatically generated training data. The datasets used as well as an extensive presentation of the experimental results can be accessed at http://olympus.ee.auth.gr/~diou/civr2009/.

REFERENCES

• 1: LSCOM Lexicon Definitions and Annotations Version 1.0. Technical report, Columbia University, 2006.
• 2: Ricardo Baeza-Yates , Carlos Hurtado , Marcelo Mendoza, Improving search engines by query clustering, Journal of the American Society for Information Science and Technology, v.58 n.12, p.1793-1804, October 2007 [doi>10.1002/asi.v58:12]
• 3: C.-C. Chang and C.-J. Lin. Libsvm: A library for support vector machines. Available: http://www.csie.ntu.edu.tw/~cjlin/libsvm.
• 4: S.-F. Chang, J. He, Y.-G. Jiang, E. El Khoury, C.-W. Ngo, A. Yanagawa, and E. Zavesky. Columbia University/VIREO-CityU/IRIT TRECVID2008 high-level feature extraction and interactive video search. In Proc. of TRECVID 2008, 2008.
• 5: Nick Craswell , Martin Szummer, Random walks on the click graph, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277784]
• 6: Alexander Hauptmann , Rong Yan , Wei-Hao Lin, How many high-level concepts will fill the semantic gap in news video retrieval?, Proceedings of the 6th ACM international conference on Image and video retrieval, p.627-634, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282369]
• 7: Djoerd Hiemstra, A Linguistically Motivated Probabilistic Model of Information Retrieval, Proceedings of the Second European Conference on Research and Advanced Technology for Digital Libraries, p.569-584, September 21-23, 1998
• 8: D. Hiemstra, H. Rode, R. van Os, and J. Flokstra. PF/Tijah: text search in an XML database system. In Proc. of the 2nd International Workshop on Open Source Information Retrieval (OSIR 2006), pages 12--17, 2006.
• 9: Proceeding of the 17th international conference on World Wide Web, April 2008
• 10: Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
• 11: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007 [doi>10.1145/1229179.1229181]
• 12: Diane Kelly , Jaime Teevan, Implicit feedback for inferring user preference: a bibliography, ACM SIGIR Forum, v.37 n.2, p.18-28, Fall 2003 [doi>10.1145/959258.959260]
• 13: A. Natsev, W. Jiang, M. Merler, J. R. Smith, J. Tešić, L. Xie, and R. Yan. IBM Research TRECVID-2008 Video Retrieval System. In Proc. of TRECVID 2008, 2008.
• 14: M. A. Palomino, M. P. Oakes, and T. Wuytack. Automatic extraction of keywords for a multimedia search engine using the chi-square test. In Proc. of the 9th Dutch-Belgian Information Retrieval Workshop (DIR 2009), pages 3--10, 2009.
• 15: Barbara Poblete , Ricardo Baeza-Yates, Query-sets: using implicit feedback and query patterns to organize web documents, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367504]
• 16: A. S. and G. Quénot. Video corpus annotation using active learning. In Proc. of the 30th European Conference on IR Research, pages 187--198, 2008.
• 17: B. Sigurbjörnsson and R. van Zwol. Flickr tag recommendation based on collective knowledge. In Huai et al. {9}, pages 327--336.
• 18: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 19: A. Ulges, M. Koch, C. Schulze, and T. Breuel. Learning TRECVID'08 high-level features from YouTube#8482;. In Proc. of TRECVID 2008, 2008.
• 20: Jan C. van Gemert , Jan-Mark Geusebroek , Cor J. Veenman , Cees G. M. Snoek , Arnold W. M. Smeulders, Robust Scene Categorization by Learning Image Statistics in Context, Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop, p.105, June 17-22, 2006 [doi>10.1109/CVPRW.2006.177]
• 21: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 22: X.-J. Wang, W.-Y. M. Ma, and X. Li. Exploring statistical correlations for image retrieval. Multimedia Systems, 11(4): 340--351, 2006.
• 23: Jun Yang , Alexander G. Hauptmann, (Un)Reliability of video concept detection, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386367]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Experimentation, Performance

Keywords:
clickthrough data, collective knowledge, concepts, image annotation, implicit feedback, search logs, supervised learning