A mutual semantic endorsement approach to image retrieval and context provision

ABSTRACT

Learning semantics from annotated images to enhance content-based retrieval is an important research direction. In this paper,annotation data are assumed available for only a subset of images inside the database. An on the fly learning method is developed to capture the semantics of query images. Specifically, the semantics of annotated images in a visual proximity of a query are compared with each other to determine the amount of mutual endorsement. An image is considered endorsed by another if they possess similar semantics. Annotations with high mutual endorsement are used to narrow down a candidate pool of images. The new retrieval method is inherently dynamic and treats seamlessly different forms of annotation data. Experiments show that semantic endorsement can increase precision by as much as 70%in average for a wide range of parameter settings. We also develop a context provision mechanism to reveal the relationship between a query and semantic clusters extracted from the database. Context helps users explore the content of a database and provides a platform for them to tailor searches by stressing different perspectives in the interpretation of a query.

REFERENCES

• 1: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 2: N. Boujemaa, J. Fauqueur, and V. Gouet "What's beyond query by example?", Trends and Advances in Content-Based Image and Video Retrieval, L. Shapiro, et al., eds., LNCS, Springer Verlag, 2004.
• 3: T.F. Cox and M.A. Cox, Multidimensional Scaling, Chapman & Hall, 1994.
• 4: Ingrid Daubechies, Ten lectures on wavelets, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1992
• 5: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 6: Anil K. Jain , Richard C. Dubes, Algorithms for clustering data, Prentice-Hall, Inc., Upper Saddle River, NJ, 1988
• 7: J.J. Jiang and D.W. Conrath, "Semantic similarity based on corpus statistics and lexical taxonomy," Proc. Intl. Conf. on Research in Computational Linguistics, 1997.
• 8: Dhiraj Joshi , James Z. Wang , Jia Li, The story picturing engine: finding elite images to illustrate a story using mutual reinforcement, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026732]
• 9: Beitao Li , Kingshy Goh, Confidence-based dynamic ensemble for image annotation and semantics discovery, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957051]
• 10: J. Li, R.M. Gray, and R.A. Olshen, "Multiresolution image classification by hierarchical modeling with two dimensional hidden Markov models," IEEE Trans. on Information Theory, 46(5):1826--41, 2000.
• 11: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 12: Jia Li , James Z. Wang , Gio Wiederhold, IRM: integrated region matching for image retrieval, Proceedings of the eighth ACM international conference on Multimedia, p.147-156, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354452]
• 13: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 14: A. Vailaya, M.A.T. Figueiredo, A.K. Jain, and H.-J. Zhang, "Image classification for content-based indexing," IEEE Trans.Image Processing, 10(1):117--130, 2001.
• 15: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109]
• 16: R. Zhang, Z. Zhang, "Hidden semantic concept discovery in region based image retrieval," CVPR, Washington D.C., June 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.0 General
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

General Terms:
Algorithms, Performance

Keywords:
database context, image retrieval, mutual semantic endorsement, semantic learning