Multimodal metadata fusion using causal strength

ABSTRACT

We propose a probabilistic framework that uses influence diagrams to fuse metadata of multiple modalities for photo annotation. We fuse contextual information (location, time, and camera parameters), visual content (holistic and local perceptual features), and semantic ontology in a synergistic way. We use causal strengths to encode causalities between variables, and between variables and semantic labels. Through analytical and empirical studies, we demonstrate that our fusion approach can achieve high-quality photo annotation and good interpretability, substantially better than traditional methods.

REFERENCES

• 1: http://www.exif.org.
• 2: K. Barnard and D. Forsyth. Learning the semantics of words and pictures. In International Conference on Computer Vision, volume 2, pages 408--415, 2000.
• 3: M. Boutell and J. Luo. Bayesian fusion of camera metadata cues in semantic scene classification. IEEE CVPR, 2004.
• 4: E. Y. Chang. Extent: Combining context, content, and semantic ontology for photo annotation. Second International Workshop on Computer Vision meets Databases, 2005.
• 5: Marc Davis , Simon King , Nathan Good , Risto Sarvas, From context to content: leveraging context to infer media metadata, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027572]
• 6: Anind K. Dey, Understanding and Using Context, Personal and Ubiquitous Computing, v.5 n.1, p.4-7, February 2001 [doi>10.1007/s007790170019]
• 7: T. Dietterich and G. Bakiri. Solving multiclass learning problems via error-correcting output codes. Artifical Intelligence Research, 2:263--286, 1995.
• 8: Diomidis D. Spinellis, Position-Annotated Photographs: A Geotemporal Web, IEEE Pervasive Computing, v.2 n.2, p.72-79, April 2003 [doi>10.1109/MPRV.2003.1203756]
• 9: P. J. Doshi, L. G. Greenwald, and J. R. Clarke. Using bayesian networks for cleansing trauma data. American Association for Artificial Intelligence, 2003.
• 10: Nir Friedman , Dan Geiger , Moises Goldszmidt, Bayesian Network Classifiers, Machine Learning, v.29 n.2-3, p.131-163, Nov./Dec. 1997 [doi>10.1023/A:1007465528199]
• 11: Dan Geiger , David Heckerman, Knowledge representation and inference in similarity networks and Bayesian multinets, Artificial Intelligence, v.82 n.1-2, p.45-74, April 1996 [doi>10.1016/0004-3702(95)00014-3]
• 12: E. B. Goldstein. Senstation and perception (5th edition). 1999.
• 13: R. M. Haralick, K. Shanmugam, and I. Dinstein. Texture features for image classification. IEEE Trans. on Sys. Man. and Cyb, 3(6), 1973.
• 14: D. Heckerman. A bayesian approach to learning causal networks. Conference on Uncertainty in Artificial Intelligence, pages 107--118, 1995.
• 15: D. Heckerman and R. Shachter. Decision-theoretic foundations for causal reasoning.MSR-TR-94-11, 1994.
• 16: Y. Ke and R. Sukthankar. Pca-sift: A more distinctive representation for local image descriptors. IEEE Computer Vision and Pattern Recognition, 2004.
• 17: L. Khan and D. McLeod. Disambiguation of annotated text of audio using ontologies. SIGKDD, 2002.
• 18: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 19: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 20: B. S. Manjunath , W. Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.837-842, August 1996 [doi>10.1109/34.531803]
• 21: Mor Naaman , Susumu Harada , QianYing Wang , Hector Garcia-Molina , Andreas Paepcke, Context data in geo-referenced digital photo collections, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027573]
• 22: M. Naaman, A. Paepcke, and H. Garcia-Molina. From where to what: Metadata sharing for digital photographs with geographic coordinates. International Conference on Cooperative Information Systems (CoopIS), 2003.
• 23: NIST. Common evaluation measures. 2001.
• 24: L. R. Novick and P. W. Cheng. Assessing interactive causal influence. Psychological Review, 111(2):455--485, 2004.
• 25: Judea Pearl, Causality: models, reasoning, and inference, Cambridge University Press, New York, NY, 2000
• 26: J. Pearl. Causal inference in the health sciences: A conceptual introduction. Special issue on causal inference, Kluwer Academic Publishers, Health Services and Outcomes Research Methodology, 2:189--220, 2001.
• 27: J. Platt. Probabilistic outputs for svms and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 1999.
• 28: Y. Rui, T. S. Huang, and S.-F. Chang. Image retrieval: Current techniques, promising directions and open issues. Journal of Visual Communication and Image Representation, 1999.
• 29: Y. Rui, A. C. She, and T. S. Huang. Modified fourier descriptors for shape representations- a practical approach. Proc. of First International Workshop on Image Databases and Multi Media Search, 1996.
• 30: J. R. Smith and S. F. Chang. Transform features for texture classification and discrimination in large image databases. Proc. IEEE Int. Conf. on Image Proc., 1994.
• 31: J. R. Smith and S.-F. Chang. Tools and techniques for color image retrieval. Proc. SPIE Proceedings Storage and Retrieval for Image and Video Databases IV, 2670, 1995.
• 32: John R. Smith , Shih-Fu Chang, Visually Searching the Web for Content, IEEE MultiMedia, v.4 n.3, p.12-20, July 1997 [doi>10.1109/93.621578]
• 33: M. Stricker and M. Orengo. Similarity of color images. Proc. SPIE Storage and Retrieval for Image and Video Databases, 1995.
• 34: H. Tamura, S. Mori, and T. Yamawaki. Texture features corresponding to visual perception. IEEE Trans. on Sys., Man. and Cyb, 3(6), 1978.
• 35: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 36: J. Z. Wang, J. Li, and G. Wiederhold. Simplicity: Semantics-sensitive integrated matching for picture libraries. ACM Multimedia Conference, 2000.
• 37: J. Williamson. Causality, in Dov Gabbay & F. Guenthner (eds.): Handbook of Philosophical Logic. Kluwer (to appear), 2005.
• 38: Y. Wu, B. L. Tseng, and J. R. Smith. Ontology-based multi-classification learning for video concept detection. IEEE International Conference on Multimedia and Expo, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms

Keywords:
causal strength, influence diagram, multimodal fusion, photo annotation