Exploring multimedia in a keyword space

ABSTRACT

We address the problem of searching multimedia by semantic similarity in a keyword space. In contrast to previous research we represent multimedia content by a vector of keywords instead of a vector of low-level features. This vector of keywords can be obtained through user manual annotations or computed by an automatic annotation algorithm. In this setting, we studied the influence of two aspects of the search by semantic similarity process: (1) accuracy of user keywords versus automatic keywords and (2) functions to compute semantic similarity between keyword vectors of two multimedia documents. We consider these two aspects to be crucial in the design of a keyword space that can exploit social-media information and can enrich applications such as Flickr and YouTube. Experiments were performed on an image and a video dataset with a large number of keywords, with different similarity functions and with two annotation methods. Surprisingly, we found that multimedia semantic similarity with automatic keywords performs as good as or better than 95% accurate user keywords.

REFERENCES

• 1: Gustavo Carneiro , Nuno Vasconcelos, Formulating Semantic Image Annotation as a Supervised Learning Problem, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.163-168, June 20-26, 2005 [doi>10.1109/CVPR.2005.164]
• 2: S.-F. Chang, W. Chen, and H. Sundaram, "Semantic visual templates: linking visual features to semantics," in Int'l Conference on Image Processing, Chicago, IL, USA, 1998.
• 3: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 4: S. L. Feng, V. Lavrenko, and R. Manmatha, "Multiple Bernoulli relevance models for image and video annotation," in IEEE Conf. on Computer Vision and Pattern Recognition, Cambridge, UK, 2004, pp. 1002--1009.
• 5: Myron Flickner , Harpreet Sawhney , Wayne Niblack , Jonathan Ashley , Qian Huang , Byron Dom , Monika Gorkani , Jim Hafner , Denis Lee , Dragutin Petkovic , David Steele , Peter Yanker, Query by Image and Video Content: The QBIC System, Computer, v.28 n.9, p.23-32, September 1995 [doi>10.1109/2.410146]
• 6: A. Haubold, A. Natsev, and M. Naphade, "Semantic multimedia retrieval using lexical query expansion and model-based re-ranking," in IEEE Int'l Conference on Multimedia and Expo Toronto, Canada, 2006.
• 7: Alexander Hauptmann , Rong Yan , Wei-Hao Lin, How many high-level concepts will fill the semantic gap in news video retrieval?, Proceedings of the 6th ACM international conference on Image and video retrieval, p.627-634, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282369]
• 8: X. He, O. King, W.-Y. Ma, M. Li, and H.-J. Zhang, "Learning a semantic space from user's relevance feedback for image retrieval," IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, pp. 39--48, Jan 2003.
• 9: P. Howarth and S. Rüger, "Fractional distance measures for content-based image retrieval," in European Conference on Information Retrieval, Santiago de Compostela, Spain, 2005.
• 10: P. Howarth and S. Rüger, "Trading accuracy for speed," in Int'l Conf. on Image and Video Retrieval Singapore, 2005.
• 11: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 12: J. Lin, "Divergence measures based on the Shannon entropy," IEEE Trans. Inform. Theory, vol. 37, pp. 145--151, January 1991.
• 13: Ye Lu , Chunhui Hu , Xingquan Zhu , HongJiang Zhang , Qiang Yang, A unified framework for semantics and feature based relevance feedback in image retrieval systems, Proceedings of the eighth ACM international conference on Multimedia, p.31-37, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354403]
• 14: J. Magalhães, "Statistical models for semantic-multimedia information retrieval," PhD Thesis, University of London, Imperial College of Science, Technology and Medicine, 2008.
• 15: Joao Magalhaes , Stefan Rueger, High-dimensional visual vocabularies for image retrieval, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277923]
• 16: João Magalhães , Stefan Rüger, Information-theoretic semantic multimedia indexing, Proceedings of the 6th ACM international conference on Image and video retrieval, p.619-626, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282368]
• 17: A. McCallum and K. Nigam, "A comparison of event models for naive Bayes text classification," in AAAI Workshop on Learning for Text Categorization, 1998.
• 18: Florent Monay , Daniel Gatica-Perez, Modeling Semantic Aspects for Cross-Media Image Indexing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.10, p.1802-1817, October 2007 [doi>10.1109/TPAMI.2007.1097]
• 19: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 20: Apostol (Paul) Natsev , Alexander Haubold , Jelena Tešić , Lexing Xie , Rong Yan, Semantic concept-based query expansion and re-ranking for multimedia retrieval, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291448]
• 21: N. Rasiwasia, P. Moreno, and N. Vasconcelos, "Bridiging the gap: Query by semantic example," IEEE Transactions on Multimedia, vol. 9, pp. 923--938, August 2007.
• 22: N. Rasiwasia, N. Vasconcelos, and P. Moreno, "Query by semantic example," in CIVR, Phoenix, AZ, USA, 2006.
• 23: Nicu Sebe , Michael S. Lew , Dionysius P. Huijsmans, Toward Improved Ranking Metrics, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.10, p.1132-1143, October 2000 [doi>10.1109/34.879793]
• 24: Alan F. Smeaton , Ian Quigley, Experiments on using semantic distances between words in image caption retrieval, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.174-180, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243261]
• 25: Cees G. M. Snoek , Marcel Worring , Jan-Mark Geusebroek , Dennis C. Koelma , Frank J. Seinstra , Arnold W. M. Smeulders, The Semantic Pathfinder: Using an Authoring Metaphor for Generic Multimedia Indexing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.10, p.1678-1689, October 2006 [doi>10.1109/TPAMI.2006.212]
• 26: Jelena Tešić , Apostol (Paul) Natsev , John R. Smith, Cluster-based data modeling for semantic video search, Proceedings of the 6th ACM international conference on Image and video retrieval, p.595-602, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282365]
• 27: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109]
• 28: Changbo Yang , Ming Dong , Farshad Fotouhi, Semantic feedback for interactive image retrieval, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101240]
• 29: A. Yavlinsky, E. Schofield, and S. Rüger, "Automated image annotation using global features and robust nonparametric density estimation," in Int'l Conf. on Image and Video Retrieval, Singapore, 2005.
• 30: C. Zhang and T. Chen, "An active learning framework for content-based information retrieval," IEEE Transactions on Multimedia, vol. 4, pp. 260--268, Jun 2002.
• 31: Xiang Sean Zhou , Thomas S. Huang, Unifying Keywords and Visual Contents in Image Retrieval, IEEE MultiMedia, v.9 n.2, p.23-33, April 2002 [doi>10.1109/93.998050]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Abstracting methods

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
automatic keyword annotations, keyword spaces, multimedia, search, user keyword annotations