Modeling annotated data

ABSTRACT

We consider the problem of modeling annotated data---data with multiple types where the instance of one type (such as a caption) serves as a description of the other type (such as an image). We describe three hierarchical probabilistic mixture models which aim to describe such data, culminating in correspondence latent Dirichlet allocation, a latent variable model that is effective at modeling the joint distribution of both types and the conditional distribution of the annotation given the primary type. We conduct experiments on the Corel database of images and captions, assessing performance in terms of held-out likelihood, automatic annotation, and text-based image retrieval.

REFERENCES

• 1: H. Attias. A variational Bayesian framework for graphical models. In Advances in Neural Information Processing Systems 12, 2000.
• 2: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 3: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003
• 4: D. Cohn and T. Hofmann. The missing link-A probabilistic model of document content and hypertext connectivity. In Advances in Neural Information Processing Systems 13, 2001.
• 5: A. Goodrum. Image information retrieval: An overview of current research. Informing Science, 3(2):63--67, 2000.
• 6: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 7: Michael I. Jordan , Zoubin Ghahramani , Tommi S. Jaakkola , Lawrence K. Saul, An Introduction to Variational Methods for Graphical Models, Machine Learning, v.37 n.2, p.183-233, Nov.1.1999 [doi>10.1023/A:1007665907178]
• 8: Carlo Meghini , Fabrizio Sebastiani , Umberto Straccia, A model of multimedia information retrieval, Journal of the ACM (JACM), v.48 n.5, p.909-970, September 2001 [doi>10.1145/502102.502103]
• 9: C. Morris. Parametric empirical Bayes inference: Theory and applications. Journal of the American Statistical Association, 78(381):47--65, 1983.
• 10: M. Naphade and T. Huang. A probabilistic framework for semantic video indexing, filtering and retrieval. IEEE Transactions on Multimedia, 3(1):141--151, March 2001.
• 11: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 12: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688]
• 13: B. Taskar, E. Segal, and D. Koller. Probabilistic clustering in relational data. In IJCAI-01, pages 870--876, 2001.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Multivariate statistics

General Terms:
Algorithms, Experimentation

Keywords:
automatic image annotation, empirical Bayes, image retrieval, probabilistic graphical models, variational methods