Visual & textual fusion for region retrieval: from both fuzzy matching and bayesian reasoning aspects

ABSTRACT

This paper presents a novel visual & textual information fusion framework for region-based image retrieval. We explore the issue of linguistic-integrated region retrieval from both Bayesian Reasoning and Fuzzy Region Matching aspects. Firstly, to associate textual information with image regions, we present a region-based soft annotation strategy. Our method automatically labels each image region with multiple keywords, each of which is assigned a confidence factor to indicate its annotation accuracy. In annotation classifier training, we adopt a pairwise coupling (PWC) SVM bagging network to address the problems of sample insufficiency and sample asymmetry. Consequently, in image retrieval, we fuse regions. visual & textual information to rank image similarities at perceptual level. Two fusion schemes are explored in proposed framework: 1. Semantic-Supervised Integrated Region Matching (SSIRM); 2. Keyword-Integrated Bayesian Reasoning (KIBR). SSIRM is a keyword-integrated fuzzy region matching strategy, which is adopted in the case that the query image is pre-annotated; KIBR is adopted in the case that the query image is non-annotated or poorly-annotated, which supports both query-by-example and query-by-keyword based on statistical text-image translation model. Finally, in relevance feedback (RF) learning, we exploit a unified visual & textual learning algorithm to precisely capture users' retrieval intention. Superior annotation, retrieval (over IRM) and RF performances (Both over IRM + SVM at region-level and SVM & ALSVM & ABSVM at global-level) are presented in our experiments, which demonstrate the efficiency of proposed fusion framework to bridge the semantic gap.

REFERENCES

• 1: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 2: R. C. Veltkamp, M. Tanase, "Content-Based Image Retrieval Systems: A Survey", Technical report UU-CS-2000-34, Dep. of Computing Science, Utrecht University, Oct. 2000. 34.
• 3: Y. Gong, H. J. Zhang and T. C. Chua, "An image database system with content capturing and fast image indexing abilities," Proc. International Conference on Multimedia Computing and Systems, Boston, 14-19 May 1994, pp.121--130.
• 4: Z. Su, H.-J. Zhang, and S. Li, "Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces and Progressive Learning," IEEE Trans. on Image Processing, vol.12, no.3, 2003.8.
• 5: Dacheng Tao , Xiaoou Tang , Xuelong Li , Xindong Wu, Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.7, p.1088-1099, July 2006 [doi>10.1109/TPAMI.2006.134]
• 6: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 7: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 8: Ye Lu, Hongjiang Zhang, Liu Wenyin et. al., "Joint Semantic and Feature Based Image Retrieval Using Relevance Feedback," IEEE Trans. on Multimedia, vol.5, no.3, pp.339--347, Sep. 2003.
• 9: Jingrui He , Hanghang Tong , Mingjing Li , Hong-Jiang Zhang , Changshui Zhang, Mean version space: a new active learning method for content-based image retrieval, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026715]
• 10: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 11: R.M. Haralick, K. Shanmugam, and I. Dinstein, "Texture features for image classification," IEEE Trans. on System, Man, Cybernetics, Vol. 3, pp. 610--621, November 1973.
• 12: King-Shy Goh , Edward Y. Chang , Beitao Li, Using One-Class and Two-Class SVMs for Multiclass Image Annotation, IEEE Transactions on Knowledge and Data Engineering, v.17 n.10, p.1333-1346, October 2005 [doi>10.1109/TKDE.2005.170]
• 13: Md. Mahmudur Rahman, P. Bhattacharya, B. C. Desai, "A Framework for Medical Image Retrieval using Machine Learning & Statistical Similarity Matching Techniques with Relevance Feedback," IEEE Trans. on Information Technology in Biomedicine, Vol.11, Issue 1, pp.58--69, Jan. 2007.
• 14: Ting-Fan Wu , Chih-Jen Lin , Ruby C. Weng, Probability Estimates for Multi-class Classification by Pairwise Coupling, The Journal of Machine Learning Research, 5, p.975-1005, 12/1/2004
• 15: E. Chang, K. Goh et. al., "CBSA: Content-Based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machines," IEEE Trans. on Circuits and Systems for Video Technology, Vol.13, Issue 1, pp.26--38, Jan.2003.
• 16: F. Jing, M. Li et. al., "A Unified Framework for Image Retrieval Using Keyword and Visual Features," IEEE Trans. on Image Processing, Vol.14, No.7, pp.979--989, July 2000.
• 17: Yixin Chen , James Z. Wang, A Region-Based Fuzzy Feature Matching Approach to Content-Based Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.9, p.1252-1267, September 2002 [doi>10.1109/TPAMI.2002.1033216]
• 18: R. Y, T. Huang, Mehrotra S et. al., "Relevance Feedback: A Power Tool for Interactive Content-based Image Retrieval," IEEE Trans. Circuits and Systems for Video Technology, 1998. 8(5): 644--655
• 19: S. Liapis, G. Tziritas, "Color and Textual Image Retrieval Using Chromaticity Histograms and Wavelet Frames," IEEE Transactions on Multimedia, Vol.6, No.5, October, 2004, pp.676--686.
• 20: R. Brunelli and O. Mich, "Image Retrieval by Example," IEEE Trans. on Multimedia, Vol.2, No.3, September, 2000, pp.164--171.
• 21: G. Aggarwal, A. T. V., and S. Ghosal, "An Image Retrieval System With Automatic Query Modification," IEEE Trans. on Multimedia, Vol.4, No.2, June 2002, pp. 201--214.
• 22: K.-M. Lee, and W. N. Street, "Cluster-Driven Refinement for Content-Based Digital Image Retrieval," IEEE Trans. on Multimedia, Vol.6, No.6, December 2004, pp.817--827.
• 23: B.C. Ko, H. Byun, "FRIP: A Region-Based Image Retrieval Tool Using Automatic Image Segmentation and Stepwise Boolean AND Matching," IEEE Trans. on Multimedia, Vol.7, No.1, February 2005, pp.105--113.
• 24: D. Tao, X. Tang, X. Li, and Y. Rui, "Direct Kernel Biased Discriminant Analysis: A New Content-Based Image Retrieval Relevance Feedback Algorithm," IEEE Trans. on Multimedia, Vol.8, No. 4, August 2004, pp.716--727.
• 25: Y. Lu, H.-J. Zhang, L. Wenyin, and C. Hu, "Joint Semantic and Feature Based Image Retrieval Using Relevance Feedback," IEEE Trans. on Multimedia, Vol.5, No.3, September 2003, pp.339--347.
• 26: A. Del Bimbo, E. Vicario, "Weighted Walkthroughs between Extended Entities for Retrieval by Spatial Arrangement," IEEE Trans. on Multimedia, Vol.5, No.1, pp.52--70, 2003.
• 27: Tao Wang , Yong Rui , Jia-Guang Sun, Constraint Based Region Matching for Image Retrieval, International Journal of Computer Vision, v.56 n.1-2, p.37-45, January-February 2004 [doi>10.1023/B:VISI.0000004831.53436.88]
• 28: Jaume Amores , Nicu Sebe , Petia Radeva , Theo Gevers , Arnold Smeulders, Boosting contextual information in content-based image retrieval, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026717]
• 29: I. J. Cox, M. L. Miller, T. P. Minka, T. V. Papathomas, and P. N. Yianilos, "The Bayesian Image Retrieval System, PicHunter: Theory, Implementation, and Psychophysical Experiments," IEEE Trans. on Image Processing, Vol.9, No.1, pp.20--37, 2000.
• 30: R. Zhang and Z. Zhang, "Hidden Semantic Concept Discovery in Region Based Image Retrieval, Computer Vision and Pattern Recognition, Washington, DC, USA, Vol.2, pp.996--1001,June 2004.
• 31: X. S. Zhou, T. S. Huang, "Relevance Feedback in Image Retrieval: A Comprehensive Review," International Journal of Multimedia Systems, Vol.8, pp 536--544, 2003.
• 32: Jing Huang , S. Ravi Kumar , Mandar Mitra , Wei-Jing Zhu , Ramin Zabih, Image Indexing Using Color Correlograms, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.762, June 17-19, 1997
• 33: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 34: Hua Xie , Antonio Ortega, An user preference information based kernel for SVM active learning in content-based image retrieval, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026713]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback; Search process

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
bagging, bayesian reasoning, image annotation, image retrieval, pairwise coupling, relevance feedback, support vector machine