Incorporating camera metadata for attended region detection and consumer photo classification

ABSTRACT

Photos taken by human beings significantly differ from the pictures that are taken by a surveillance camera or a vision sensor on a robot, e.g., human beings may intentionally capture photos to express his/her feeling or record a memorial scene. Such a creative photo capture process is accomplished by adjusting two factors: (1) the parameters setting of a camera; and (2) the position between the camera and the interesting objects or scenes. To enable automatic understanding and interpretation of the semantics of photos, it is very important to take all these factors into account. Unfortunately, most existing algorithms for image understanding focus on only the content of the images while completely ignoring these two important factors. In this paper, we have developed a new algorithm to calculate what the interestingness of the photographer is and what the core content of a photo is. The gained information (i.e., attended regions and attention of the photographer) is further used to support more effective photo classification and retrieval. Our experiments on 70,000+ photos taken by 200+ different models of cameras have obtained very positive results.

REFERENCES

• 1: A. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain, Content-based image retrieval at the end of the early years, PAMI, vol.22, pp.1349--1380, 2000.
• 2: R. Desimone, J. Duncan,"Neural mechanisms of selective visual attention, annual reviews", Neuroscience, vol.18, pp.193--222, 1995.
• 3: C. M. Privitera, L. W. Stark, "Algorithms for defining visual region-of-interest: Comparison with eye fixations", PAMI, vol.22, pp.970--982, 2000.
• 4: C. Siagian, L. Itti, "Rapid biologically-inspired scene classification using features shared with visual attention", PAMI, vol.29, pp.300--312, 2007.
• 5: Y. Ma, H. Zhang, "Contrast-based image attention analysis by using fuzzy growing", ACM Multimedia, pp.374--381, 2003.
• 6: J. Luo, A.E. Savakis, A. Singhal, "A Bayesian network-based framework for semantic image understanding", Pattern Recognition, vol.38, pp.919--934, 2005.
• 7: M. Boutell, J. Luo, "Bayesian fusion of camera metadata cues in semantic scene classification", IEEE CVPR, pp.623--630, 2004.
• 8: J. Fan, Y. Gao, H. Luo, "Integrating concept ontology and multi-task learning to achieve more effective classifier training for multi-level image annotation", IEEE Trans. on Image Processing, vol. 17, no.3, pp.407--426, 2008.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Feature representation

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.0 General

General Terms:
Algorithms

Keywords:
attended regions, camera metadata, image classification