Semi-supervised topic modeling for image annotation

ABSTRACT

We propose a novel technique for semi-supervised image annotation which introduces a harmonic regularizer based on the graph Laplacian of the data into the probabilistic semantic model for learning latent topics of the images. By using a probabilistic semantic model, we connect visual features and textual annotations of images by their latent topics. Meanwhile, we incorporate the manifold assumption into the model to say that the probabilities of latent topics of images are drawn from a manifold, so that for images sharing similar visual features or the same annotations, their probability distribution of latent topics should also be similar. We create a nearest neighbor graph to model the manifold and propose a regularized EM algorithm to simultaneously learn a generative model and assign probability density of latent topics to images discriminatively. In this way, databases with very few labeled images can be annotated better than previous works.

REFERENCES

• 1: K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D. M. Blei, and M. I. Jordan. Matching words and pictures. Journal of Machine Learning Research, 3:1107--1135, 2003.
• 2: M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7:2399--2434, 2006.
• 3: D. M. Blei and M. I. Jordan. Modeling annotated data. In Proc. ACM Int. Conf. on Research and Development in Informaion Retrieval(ACM SIGIR), pages 127--134, 2003.
• 4: D. Cai, Q. Mei, J. Han, and C. Zhai. Modeling hidden topics on document manifold. In Proc. ACM Conf. on Information and knowledge management(CIKM'08), pages 911--920, 2008.
• 5: G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray. Visual categorization with bags of keypoints. In Workshop on Statistical Learning in Computer Vision, ECCV, pages 1--22, 2004.
• 6: X. He, D. Cai, Y. Shao, H. Bao, and J. Han. Laplacian regularized gaussian mixture model for data clustering. Preprint.
• 7: Q. Mei, D. Cai, D. Zhang, and C. Zhai. Topic modeling with network regularization. In Proc. ACM Int. Conf. on World Wide Web (WWW'08), pages 101--110, 2008.
• 8: F. Monay and D. Gatica-Perez. On image auto-annotation with latent space models. In Proc. ACM Int. Conf. on Multimedia (SIGMM'03), pages 275--278, 2003.
• 9: F. Monay and D. Gatica-Perez. Plsa-based image auto-annotation: constraining the latent space. In Proc. ACM Int. Conf. on Multimedia (SIGMM'04), pages 348--351, 2004.
• 10: R. M. Neal and G. E. Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355--368. 1999.
• 11: R. Zhang, Z. M. Zhang, M. Li, W.-Y. Ma, and H.-J. Zhang. A probabilistic semantic model for image annotation and multi-modal image retrieval. In Proc. IEEE Int. Conf. on Computer Vision (ICCV'05), pages 846--851, 2005.
• 12: X. Zhu, J. Lafferty, and Z. Ghahramani. Semi-supervised learning using gaussian fields and harmonic functions. In Proc. Int. Conf. Machine Learning(ICML'05), 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Languages, Theory

Keywords:
automatic image annotation, laplacian regularization, semantic indexing, semi-supervised learning