Image retrieval using query by contextual example

ABSTRACT

Current image retrieval techniques have difficulties to retrieve images which exhibit distinct visual patterns but belong to the class of the query image. Previous attempts to improve generalization have shown that the introduction of semantic representations can mitigate this problem. We extend the existing query-by-semantic example (QBSE) retrieval paradigm by adding a second layer of semantic representation. At the first level, the representation is driven by patch-based visual features. Semantic concepts, from a predefined vocabulary, are modeled as Gaussian mixtures on a visual feature space, and images as vectors of posterior probabilities of containing each of the semantic concepts. At the second level, the representation is purely semantic. Semantic concepts are modeled as Dirichlet mixtures on the semantic feature space of QBSE, and images are again represented as vectors of posterior concept probabilities. It is shown that the proposed retrieval strategy, referred to as query-by-contextual-example (QBCE), is able to cope with the ambiguities of patch-based classification, exhibiting significantly better generalization than previous methods. An experimental evaluation on benchmark datasets shows that QBCE retrieval systems can substantially outperform their QBVE and QBSE counterparts, achieving high precision at most levels of recall.

REFERENCES

• 1: David M. Blei , Michael I. Jordan, Modeling annotated data, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860460]
• 2: Gustavo Carneiro , Antoni B. Chan , Pedro J. Moreno , Nuno Vasconcelos, Supervised Learning of Semantic Classes for Image Annotation and Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.3, p.394-410, March 2007 [doi>10.1109/TPAMI.2007.61]
• 3: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 4: S. Feng, R. Manmatha, and V. Lavrenko. Multiple bernoulli relevance models for image and video annotation. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington DC, 2004.
• 5: A. Jain and A. Vailaya. Image retrieval using color and shape. Pattern Recognition Journal, 29, August 1996.
• 6: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2169-2178, June 17-22, 2006 [doi>10.1109/CVPR.2006.68]
• 7: Fei-Fei Li , Pietro Perona, A Bayesian Hierarchical Model for Learning Natural Scene Categories, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.524-531, June 20-26, 2005 [doi>10.1109/CVPR.2005.16]
• 8: J. Magalhães, S. Overell, and S. Rüger. A semantic vector space for query by image example. ACM SIGIR Special Interest Group on Information Retrieval, 2007.
• 9: T. Minka. Estimating a Dirichlet distribution. http://research.microsoft.com/ minka/papers/dirichlet/,1:3, 2000.
• 10: Apostol (Paul) Natsev , Milind R. Naphade , John R. Smith, Semantic representation: search and mining of multimedia content, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014133]
• 11: W. Niblack and et al. The qbic project: Querying images by content using color, texture, and shape. In Storage and Retrieval for Image and Video Databases, pages 173--181, SPIE, Feb. 1993, San Jose, California.
• 12: A. Pentland , R. W. Picard , S. Sclaroff, Photobook: content-based manipulation of image databases, International Journal of Computer Vision, v.18 n.3, p.233-254, June 1996 [doi>10.1007/BF00123143]
• 13: N. Rasiwasia, P. Moreno, and N. Vasconcelos. Bridging the gap: Query by semantic example. Multimedia, IEEE Transactions on, 9(5):923--938, 2007.
• 14: N. Rasiwasia, N. Vasconcelos, and P. J. Moreno. Query by semantic example. In CIVR, pages 51--60, 2006.
• 15: J. Sivic, B. Russell, A. Efros, A. Zisserman, and W. Freeman. Discovering object categories in image collections. Proc. ICCV, 1:65, 2005.
• 16: John R. Smith , Shih-Fu Chang, VisualSEEk: a fully automated content-based image query system, Proceedings of the fourth ACM international conference on Multimedia, p.87-98, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244151]
• 17: J. R. Smith , M. Naphade , A. Natsev, Multimedia semantic indexing using model vectors, Proceedings of the 2003 International Conference on Multimedia and Expo, p.445-448, July 06-09, 2003
• 18: J. R. Smith, C.-Y. Lin, M. R. Naphade, A. Natsev, and B. L. Tseng. Validity-weighted model vector-based retrieval of video. In Proceedings of the SPIE, Volume 5307, pp. 271--279(2003)., pages 271--279, 2003.
• 19: N. Vasconcelos. Minimum probability of error image retrieval. IEEE Trans. on Signal Processing, 52(8), August 2004.
• 20: J. Vogel and B. Schiele. A semantic typicality measure for natural scene categorization. DAGMŠ04 Annual Pattern Recognition Symposium.
• 21: Lior Wolf , Stanley Bileschi, A Critical View of Context, International Journal of Computer Vision, v.69 n.2, p.251-261, August 2006 [doi>10.1007/s11263-006-7538-0]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Computer vision

General Terms:
Algorithms

Keywords:
dirichlet distribution, image retrieval, query-by-example, semantic spaces