Enhancing relevance feedback in image retrieval using unlabeled data

ABSTRACT

Relevance feedback is an effective scheme bridging the gap between high-level semantics and low-level features in content-based image retrieval (CBIR). In contrast to previous methods which rely on labeled images provided by the user, this article attempts to enhance the performance of relevance feedback by exploiting unlabeled images existing in the database. Concretely, this article integrates the merits of semisupervised learning and active learning into the relevance feedback process. In detail, in each round of relevance feedback two simple learners are trained from the labeled data, that is, images from user query and user feedback. Each learner then labels some unlabeled images in the database for the other learner. After retraining with the additional labeled data, the learners reclassify the images in the database and then their classifications are merged. Images judged to be positive with high confidence are returned as the retrieval result, while those judged with low confidence are put into the pool which is used in the next round of relevance feedback. Experiments show that using semisupervised learning and active learning simultaneously in CBIR is beneficial, and the proposed method achieves better performance than some existing methods.

REFERENCES

• 1: Naoki Abe , Hiroshi Mamitsuka, Query Learning Strategies Using Boosting and Bagging, Proceedings of the Fifteenth International Conference on Machine Learning, p.1-9, July 24-27, 1998
• 2: Avrim Blum , Shuchi Chawla, Learning from Labeled and Unlabeled Data using Graph Mincuts, Proceedings of the Eighteenth International Conference on Machine Learning, p.19-26, June 28-July 01, 2001
• 3: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 4: Bookstein, A. 1983. Information retrieval: A sequential learning process. J. American Society Inf. Sci. 34, 4, 331--342.
• 5: Chen, J.-Y., Bouman, C. A., and Dalton, J. 2000. Hierarchical browsing and search of large image databases. IEEE Trans. Image Proces. 9, 3, 442--445.
• 6: G. Ciocca , R. Schettini, A relevance feedback mechanism for content-based image retrieval, Information Processing and Management: an International Journal, v.35 n.5, p.605-632, Sept. 1999 [doi>10.1016/S0306-4573(99)00021-7]
• 7: Ira Cohen , Fabio G. Cozman , Nicu Sebe , Marcelo C. Cirelo , Thomas S. Huang, Semisupervised Learning of Classifiers: Theory, Algorithms, and Their Application to Human-Computer Interaction, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.12, p.1553-1567, December 2004 [doi>10.1109/TPAMI.2004.127]
• 8: Cox, I. J., Miller, M., Minka, T. P., Papathomas, T., and Yianilos, P. 2000. The Bayesian image retrieval system, PicHunter: Theory, implementation, and psychophysical experiments. IEEE Trans. Image Proces. 9, 1, 20--37.
• 9: Fabio G. Cozman , Ira Cohen, Unlabeled Data Can Degrade Classification Performance of Generative Classifiers, Proceedings of the Fifteenth International Florida Artificial Intelligence Research Society Conference, p.327-331, May 14-16, 2002
• 10: Dasgupta, S., Littman, M., and McAllester, D. 2002. PAC generalization bounds for co-training. In Advances in Neural Information Processing Systems 14, T. G. Dietterich et al., eds. MIT Press, Cambridge, MA. 375--382.
• 11: Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statistical Society, Series B 39, 1, 1--38.
• 12: Dong, A. and Bhanu, B. 2003. A new semi-supervised EM algorithm for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (Madison, WI). 662--667.
• 13: Sally A. Goldman , Yan Zhou, Enhancing Supervised Learning with Unlabeled Data, Proceedings of the Seventeenth International Conference on Machine Learning, p.327-334, June 29-July 02, 2000
• 14: Dionysius P. Huijsmans , Nicu Sebe, How to Complete Performance Graphs in Content-Based Image Retrieval: Add Generality and Normalize Scope, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.2, p.245-251, February 2005 [doi>10.1109/TPAMI.2005.30]
• 15: Hwa, R., Osborne, M., Sarkar, A., and Steedman, M. 2003. Corrected co-training for statistical parsers. In Working Notes of the ICML'03 Workshop on the Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining (Washington, DC).
• 16: Yoshiharu Ishikawa , Ravishankar Subramanya , Christos Faloutsos, MindReader: Querying Databases Through Multiple Examples, Proceedings of the 24rd International Conference on Very Large Data Bases, p.218-227, August 24-27, 1998
• 17: Thorsten Joachims, Transductive Inference for Text Classification using Support Vector Machines, Proceedings of the Sixteenth International Conference on Machine Learning, p.200-209, June 27-30, 1999
• 18: Kherfi, M. L., Ziou, D., and Bernardi, A. 2002. Learning from negative example in relevance feedback for content-based image retrieval. In Proceedings of the 16th International Conference on Pattern Recognition (Quebec, Canada). 933--936.
• 19: David Dolan Lewis, Representation and learning in information retrieval, University of Massachusetts, Amherst, MA, 1992
• 20: David D. Lewis , William A. Gale, A sequential algorithm for training text classifiers, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, July 03-06, 1994, Dublin, Ireland
• 21: B. S. Manjunath , W. Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.837-842, August 1996 [doi>10.1109/34.531803]
• 22: Babu M. Mehtre , Mohan S. Kankanhalli , A. Desai Narasimhalu , Guo Chang Man, Color matching for image retrieval, Pattern Recognition Letters, v.16 n.3, p.325-331, March 1995 [doi>10.1016/0167-8655(94)00096-L]
• 23: Miller, D. J. and Uyar, H. S. 1997. A mixture of experts classifier with learning based on both labelled and unlabelled data. In Advances in Neural Information Processing Systems 9, M. Mozer et al., eds. MIT Press, Cambridge, MA. 571--577.
• 24: Henning Müller , Wolfgang Müller , David McG. Squire , Stéphane Marchand-Maillet , Thierry Pun, Performance evaluation in content-based image retrieval: overview and proposals, Pattern Recognition Letters, v.22 n.5, p.593-601, April 2001 [doi>10.1016/S0167-8655(00)00118-5]
• 25: Ion Muslea , Steven Minton , Craig A. Knoblock, Selective Sampling with Redundant Views, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.621-626, July 30-August 03, 2000
• 26: C. Nastar , M. Mitschke , C. Meilhac, Efficient Query Refinement for Image Retrieval, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.547, June 23-25, 1998
• 27: Kamal Nigam , Rayid Ghani, Analyzing the effectiveness and applicability of co-training, Proceedings of the ninth international conference on Information and knowledge management, p.86-93, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354805]
• 28: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 29: Picard, R. W., Minka, T. P., and Szummer, M. 1996. Modeling user subjectivity in image libraries. In Proceedings of the International Conference on Image Processing (Lausanne, Switzerland). 777--780.
• 30: Pierce, D. and Cardie, C. 2001. Limitations of co-training for natural language learning from large data sets. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (Pittsburgh, PA). 1--9.
• 31: Rui, Y., Huang, T. S., Ortega, M., and Mehrotra, S. 1998. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans. Circuits Syst. Video Technol. 8, 5, 644--655.
• 32: Anoop Sarkar, Applying co-training methods to statistical parsing, Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001, p.1-8, June 01-07, 2001, Pittsburgh, Pennsylvania [doi>10.3115/1073336.1073359]
• 33: H. S. Seung , M. Opper , H. Sompolinsky, Query by committee, Proceedings of the fifth annual workshop on Computational learning theory, p.287-294, July 27-29, 1992, Pittsburgh, Pennsylvania, United States [doi>10.1145/130385.130417]
• 34: Shahshahani, B. and Landgrebe, D. 1994. The effect of unlabeled samples in reducing the small sample size problem and mitigating the Hughes phenomenon. IEEE Trans. Geosci. Remote Sensing 32, 5, 1087--1095.
• 35: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 36: Mark Steedman , Miles Osborne , Anoop Sarkar , Stephen Clark , Rebecca Hwa , Julia Hockenmaier , Paul Ruhlen , Steven Baker , Jeremiah Crim, Bootstrapping statistical parsers from small datasets, Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics, April 12-17, 2003, Budapest, Hungary [doi>10.3115/1067807.1067851]
• 37: Tian, Q., Yu, J., Xue, Q., and Sebe, N. 2004. A new analysis of the value of unlabeled data in semi-supervised learning for image retrieval. In Proceedings of the IEEE International Conference on Multimedia Exposition (Taibei). 1019--1022.
• 38: Tieu, K. and Viola, P. 2000. Boosting image retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (Hilton Head, SC). 228--235.
• 39: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 40: Vasconcelos, N. and Lippman, A. 2000. Learning from user feedback in image retrieval systems. In Advances in Neural Information Processing Systems 12, S. A. Solla et al., eds. MIT Press, Cambridge, MA. 977--986.
• 41: Wang, H. F., Jin, X. Y., and Sun, Z. 2002. Semantic image retrieval (in Chinese). J. Comput. Research Development 39, 5, 513--523.
• 42: Wu, Y., Tian, Q., and Huang, T. S. 2000. Discriminant-EM algorithm with application to image retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (Hilton Head, SC). 222--227.
• 43: Jian Yao , Zhongfei (Mark) Zhang, Object Detection in Aerial Imagery Based on Enhanced Semi-Supervised Learning, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1012-1017, October 17-20, 2005 [doi>10.1109/ICCV.2005.172]
• 44: Zhang, C. and Chen, T. 2002. An active learning framework for content-based information retrieval. IEEE Trans. Multimedia 4, 2, 260--268.
• 45: Zhang, R. and Zhang, Z. 2004. Stretching Bayesian learning in the relevance feedback of image retrieval. In Proceedings of the 8th European Conference on Computer Vision (Prague, Czech). 355--367.
• 46: Zhou, X. S. and Huang, T. S. 2001. Small sample learning during multimedia retrieval using BiasMap. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (Kauai, HI). 11--17.
• 47: Zhou, X. S. and Huang, T. S. 2003. Relevance feedback in image retrieval: A comprehensive review. Multimedia Syst. 8, 6, 536--544.
• 48: Zhou, Z.-H., Chen, K.-J., and Jiang, Y. 2004. Exploiting unlabeled data in content-based image retrieval. In Proceedings of the 15th European Conference on Machine Learning (Pisa, Italy). 525--536.
• 49: Zhou, Z.-H. and Li, M. 2005a. Semi-supervised learning with co-training. In Proceedings of the 19th International Joint Conference on Artificial Intelligence (Edinburgh, Scotland). 908--913.
• 50: Zhi-Hua Zhou , Ming Li, Tri-Training: Exploiting Unlabeled Data Using Three Classifiers, IEEE Transactions on Knowledge and Data Engineering, v.17 n.11, p.1529-1541, November 2005 [doi>10.1109/TKDE.2005.186]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases
  H.3 INFORMATION STORAGE AND RETRIEVAL

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Keywords:
Relevance feedback, active learning, content-based image retrieval machine learning, learning with unlabeled data, semisupervised learning