Classification and annotation of digital photos using optical context data

ABSTRACT

Other than the pixel information, a digital photo of today has a host of other information regarding the photo shooting event. These information are captured by different sensors present on the camera and are stored as metadata. In this paper we exploit this meta information and derive useful semantics about the digital photo. We also compare our results with classical relevance models used for automatic photo annotation. We create a dataset of digital photos containing all information and report results on it. We also make the dataset available to the community for further experiments.

REFERENCES

• 1: http://http://www.flickr.com/photos/tags/.
• 2: http://www.flickr.com/.
• 3: http://www.ics.uci.edu/~psinha/dataset.html.
• 4: EXIF: Exchangeable image file format for digital cameras:version 2.2. Technical report, JEITI Association, 2002.
• 5: K. Barnard and D. Forsyth. Learning the semantics of words and pictures. In Proc of IEEE Intl Conf Computer Vision, volume 2, pages 408--415, 2000.
• 6: David M. Blei , Michael I. Jordan, Modeling annotated data, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860460]
• 7: M. Boutell and J. Luo. Bayesian fusion of camera metadata cues in semantic scene classification. In Proc. IEEE CVPR, 2004.
• 8: Matthew Boutell , Jiebo Luo, Photo Classification by Integrating Image Content and Camera Metadata, Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 4, p.901-904, August 23-26, 2004 [doi>10.1109/ICPR.2004.693]
• 9: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800]
• 10: Matthew Cooper , Jonathan Foote , Andreas Girgensohn , Lynn Wilcox, Temporal event clustering for digital photo collections, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957093]
• 11: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 12: C. Fraley and A. Raftery. How many clusters? which clustering method? answers via model based cluster analysis. The Computer Journal, 1998.
• 13: Yihong Gong, Advancing content-based image retrieval by exploiting image color and region features, Multimedia Systems, v.7 n.6, p.449-457, Nov. 1999 [doi>10.1007/s005300050145]
• 14: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 15: V. Lavrenko, R. Manmatha, and J. Jeon. A model for learning the semantics of pictures. In Proc. NIPS, 2003.
• 16: X. Liu, L. Zhang, M. Li, H. Zhang, and D. Wang. Boosting image classification with lda based feature combination for digital photograph management. Pattern Recognition, 38:887--901, 2005.
• 17: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 18: B. S. Manjunath , W. Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.837-842, August 1996 [doi>10.1109/34.531803]
• 19: G. Schwarz. Estimating dimensions of a model. The Annals of Statistics, 6:461--464, 1978.
• 20: P. Sinha and R. Jain. Concept annotation and search space decrement of digital photos using optical context information. In Proc of Multimedia Content Access: Algorithms and Systems, SPIE Electronic Imaging, 2008.
• 21: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 22: M. Tuffield, S. Harris, D. Dupplaw, A.Chakravarthy, B. Brewster, N. Gibbins, K. O'Hara, F. Ciravegna, D. Sleeman, N. Shadbolt, and Y. Wilks. Image annotation with photocopain. In International Workshop on Semantic Web Annotations for Multimedia, 2006.
• 23: Thijs Westerveld , Arjen P. de Vries, Experimental result analysis for a generative probabilistic image retrieval model, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860461]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Algorithms, Design, Performance

Keywords:
automatic annotation, camera metadata, context, digital photo, search