Multimodal concept-dependent active learning for image retrieval

ABSTRACT

It has been established that active learning is effective for learning complex, subjective query concepts for image retrieval. However, active learning has been applied in a concept independent way, (i.e., the kernel-parameters and the sampling strategy are identically chosen) for learning query concepts of differing <i>complexity</i>. In this work, we first characterize a concept's complexity using three measures: <i>hit-rate</i>, <i>isolation</i> and <i>diversity</i>. We then propose a multimodal learning approach that uses images' semantic labels to guide a <i>concept-dependent</i>, <i>active-learning</i> process. Based on the complexity of a concept, we make intelligent adjustments to the sampling strategy and the sampling pool from which images are to be selected and labeled, to improve concept learnability. Our empirical study on a $300$K-image dataset shows that concept-dependent learning is highly effective for image-retrieval accuracy.

REFERENCES

• 1: Rakesh Agrawal , Tomasz Imieliński , Arun Swami, Mining association rules between sets of items in large databases, Proceedings of the 1993 ACM SIGMOD international conference on Management of data, p.207-216, May 25-28, 1993, Washington, D.C., United States
• 2: K. Barnard, P. Duygulu, and D. Forsyth. Exploiting text and image feature co-occurrence statistics in large datasets. Trends and Advances in Content-Based Image and Video Retrieval (To Appear), 2004.
• 3: K. Barnard and D. Forsyth. Learning the semantics of words and pictures. In International Conference on Computer Vision, volume~2, pages 408--415, 2000.
• 4: A. B. Benitez and S.-F. Chang. Image classification using multimedia knowledge networks. Proc. of the Int. Conf. on Image Processing, September 2003.
• 5: A. B. Benitez, J. R. Smith, and S.-F. Chang. Medianet: A multimedia information network for knowledge representation. Proceeding of the SPIE Conference on Internet Multimedia Management Systems, November 2000.
• 6: K. Brinker. Incorporating diversity in active learning with support vector machines. Proceedings of the Twentieth International Conference on Machine Learning, pages 59--66, August 2003.
• 7: Edward Chang , Beitao Li, MEGA---the maximizing expected generalization algorithm for learning complex query concepts, ACM Transactions on Information Systems (TOIS), v.21 n.4, p.347-382, October 2003 [doi>10.1145/944012.944014]
• 8: Wei-Cheng Lai , Kingshy Goh , Edward Y. Chang, On scalability of active learning for formulating query concepts, Proceedings of the 1st international workshop on Computer vision meets databases, June 13-13, 2004, Paris, France [doi>10.1145/1039470.1039477]
• 9: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 10: Anil K. Jain , Robert P. W. Duin , Jianchang Mao, Statistical Pattern Recognition: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.1, p.4-37, January 2000 [doi>10.1109/34.824819]
• 11: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 12: Chen Li , Edward Chang , Hector Garcia-Molina , Gio Wiederhold, Clustering for Approximate Similarity Search in High-Dimensional Spaces, IEEE Transactions on Knowledge and Data Engineering, v.14 n.4, p.792-808, July 2002 [doi>10.1109/TKDE.2002.1019214]
• 13: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 14: Ye Lu , Chunhui Hu , Xingquan Zhu , HongJiang Zhang , Qiang Yang, A unified framework for semantics and feature based relevance feedback in image retrieval systems, Proceedings of the eighth ACM international conference on Multimedia, p.31-37, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354403]
• 15: Y. Mori, H. Takahashi, and R. Oka. Automatic words assignment to images based on image division and vector quantization. In Proc. of RIAO 2000: Content-Based Multimedia Information Access, Apr. 2000.
• 16: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 17: T. Westerveld. Image retrieval: Content versus context. Content-Based Multimedia Information Access, RIAO, pages 276--284, 2000.
• 18: Hongjiang Zhang , Zheng Chen , Mingjing Li , Zhong Su, Relevance Feedback and Learning in Content-Based Image Search, World Wide Web, v.6 n.2, p.131-155, June 2003 [doi>10.1023/A:1023618504691]
• 19: Xiang Sean Zhou , Thomas S. Huang, Unifying Keywords and Visual Contents in Image Retrieval, IEEE MultiMedia, v.9 n.2, p.23-33, April 2002 [doi>10.1109/93.998050]
• 20: Theory of keyblock-based image retrieval, ACM Transactions on Information Systems (TOIS), v.20 n.2, p.224-257, April 2002 [doi>10.1145/506309.506313]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.1 Models
          Subjects: Statistical

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning

General Terms:
Algorithms, Performance

Keywords:
active learning, image retrieval, relevance feedback, support vector machines