EXTENT: fusing context, content, and semantic ontology for photo annotation

ABSTRACT

This architecture paper presents EXTENT, a probabilistic framework that uses influence diagrams to fuse metadata of multiple modalities for photo annotation. EXTENT fuses contextual information (location, time, and camera parameters), photo content (perceptual features), and semantic ontology in a synergistic way. It uses causal strengths to encode causalities between variables, and between variables and semantic labels. Through a landmark-recognition case study, we show that EXTENT can provide high-quality annotation, substantially better than any traditional unimodal methods.

REFERENCES

• 1: K. Barnard and D. Forsyth. Learning the semantics of words and pictures. In International Conference on Computer Vision, volume 2, pages 408--415, 2000.
• 2: E. Y. Chang. Extent: Combining context, content, and semantic ontology for photo annotation. US Provisional Patent, 2005.
• 3: E. Y. Chang, K. Goh, G. Sychay, and G. Wu. Content-based soft annotation for multimodal image retrieval using bayes point machines. IEEE Trans. on Circuits and Systems for Video Technology Special Issue on Conceptual and Dynamical Aspects of Multimedia Content Description, 13(1):26--38, 2003.
• 4: Marc Davis , Simon King , Nathan Good , Risto Sarvas, From context to content: leveraging context to infer media metadata, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027572]
• 5: A. Deshpande, C. Guestrin, S. Madden, and W. Hong. Beyong pixels: Exploiting camera metadata for photo classification. IEEE CVPR, 2004.
• 6: Anind K. Dey, Understanding and Using Context, Personal and Ubiquitous Computing, v.5 n.1, p.4-7, February 2001 [doi>10.1007/s007790170019]
• 7: Diomidis D. Spinellis, Position-Annotated Photographs: A Geotemporal Web, IEEE Pervasive Computing, v.2 n.2, p.72-79, April 2003 [doi>10.1109/MPRV.2003.1203756]
• 8: N. Friedman and D. Koller. Learning bayesian networks from data (tutorial). NIPS, 2000.
• 9: King-Shy Goh , Edward Y. Chang , Beitao Li, Using One-Class and Two-Class SVMs for Multiclass Image Annotation, IEEE Transactions on Knowledge and Data Engineering, v.17 n.10, p.1333-1346, October 2005 [doi>10.1109/TKDE.2005.170]
• 10: King-Shy Goh , Edward Chang , Kwang-Ting Cheng, SVM binary classifier ensembles for image classification, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502652]
• 11: D. Heckerman. A bayesian approach to learning causal networks. Conference on Uncertainty in Artificial Intelligence, pages 107--118, 1995.
• 12: D. Heckerman and R. Shachter. Decision-theoretic foundations for causal reasoning. MSR-TR-94-11, 1994.
• 13: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 14: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 15: Mor Naaman , Susumu Harada , QianYing Wang , Hector Garcia-Molina , Andreas Paepcke, Context data in geo-referenced digital photo collections, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027573]
• 16: M. Naaman, A. Paepcke, and H. Garcia-Molina. From where to what: Metadata sharing for digital photographs with geographic coordinates. International Conference on Cooperative Information Systems (CoopIS), 2003.
• 17: L. R. Novick and P. W. Cheng. Assessing interactive causal influence. Psychological Review, 111(2):455--485, 2004.
• 18: J. B. Tenenbaum and T. L. Griffiths. Generalization, similarity, and bayesian inference. Behavioral and Brain Sciences, 24:629--641, 2001.
• 19: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 20: J. Williamson. Causality, in Dov Gabbay & F. Guenthner (eds.): Handbook of Philosophical Logic. Kluwer (to appear), 2005.

INDEX TERMS

Keywords:
causal strength, image annotation, inference diagram, information integration