Annotating photo collections by label propagation according to multiple similarity cues

ABSTRACT

This paper considers the emerging problem of annotating personal photo collections that are taken by digital cameras and may have been subsequently organized by customers. Unlike the images from the web searching engine or commercial image banks (e.g. the Corel database), the photos in the same personal collection are related to each other in time, location, and content. Advanced technologies can record the GPS coordinates for each photo, and thus provide a richer source of context to model and enforce the correlation between the photos in the same collection. Recognizing the well-known limitations ("semantic gap") of visual recognition algorithms, we exploit the correlation between the photos to enhance the annotation performance. In our approach, high-confidence annotation labels are first obtained for certain photos and then propagated to the remaining photos in the same collection, according to time, location, and visual proximity (or similarity). A novel generative probabilistic model is employed, which outperforms the pervious linear propagation scheme. Experimental results have shown the advantages of the proposed annotation scheme.

REFERENCES

• 1: Flickr, http://www.flickr.com
• 2: Picasa Web Album, http://picasaweb.google.com
• 3: Corel stock photo database, http://www.corel.com
• 4: Fei-Fei Li , Pietro Perona, A Bayesian Hierarchical Model for Learning Natural Scene Categories, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.524-531, June 20-26, 2005 [doi>10.1109/CVPR.2005.16]
• 5: A. K. Jain and A. Vailaya, Image retrieval using color and shape, Pattern Recognition, 29 (8), 1233--1244, 1996.
• 6: Y. Rui, T. S. Huang, S Mehrotra, Content-based image retrieval with relevance feedback in MARS, IEEE Proc. Image Processing, 1997.
• 7: A. Blum, T. Mitchell, Combining labeled and unlabeled data with co-training, Annual Conference on Learning Theory, 1998.
• 8: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 9: Steven C. H. Hoi , Michael R. Lyu, A Semi-Supervised Active Learning Framework for Image Retrieval, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.302-309, June 20-26, 2005 [doi>10.1109/CVPR.2005.44]
• 10: Gustavo Carneiro , Antoni B. Chan , Pedro J. Moreno , Nuno Vasconcelos, Supervised Learning of Semantic Classes for Image Annotation and Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.3, p.394-410, March 2007 [doi>10.1109/TPAMI.2007.61]
• 11: X. Zhu, Z. Ghahramani, and J. Lafferty, Semi-supervised learning using Gaussian fields and harmonic functions, International Conference on Machine Learning, 2003.
• 12: Fei Wang , Changshui Zhang , Helen C. Shen , Jingdong Wang, Semi-Supervised Classification Using Linear Neighborhood Propagation, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.160-167, June 17-22, 2006 [doi>10.1109/CVPR.2006.272]
• 13: D. Zhou, O. Bousquet, T. N. Lal, J. Weston and B. Scholkopf, Learning with Local and Global Consistency. Neural Information Processing Systems, 2004.
• 14: Jingrui He , Mingjing Li , Hong-Jiang Zhang , Hanghang Tong , Changshui Zhang, Manifold-ranking based image retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027531]
• 15: Jing Liu , Mingjing Li , Wei-Ying Ma , Qingshan Liu , Hanqing Lu, An adaptive graph model for automatic image annotation, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178689]
• 16: Hanghang Tong , Jingrui He , Mingjing Li , Changshui Zhang , Wei-Ying Ma, Graph based multi-modality learning, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101337]
• 17: Jingjing Liu , Wei Lai , Xian-Sheng Hua , Yalou Huang , Shipeng Li, Video search re-ranking via multi-graph propagation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291279]
• 18: Dengyong Zhou , Christopher J. C. Burges, Spectral clustering and transductive learning with multiple views, Proceedings of the 24th international conference on Machine learning, p.1159-1166, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273642]
• 19: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 20: Matthew Brown , David G. Lowe, Automatic Panoramic Image Stitching using Invariant Features, International Journal of Computer Vision, v.74 n.1, p.59-73, August 2007 [doi>10.1007/s11263-006-0002-3]
• 21: M. Boutell, and J. Luo. Beyond pixels: Exploiting camera metadata for photo classification. Pattern Recognition 38(6): 935--946, 2005.
• 22: Y. Aytar, O. B. Orhan, and M. Shah. Improving Semantic Concept Detection and Retrieval Using Contextual Estimates, IEEE International Conference on Multimedia & Expo, 2007.
• 23: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 24: Yongmei Michelle Wang , Hongjiang Zhang, Detecting image orientation based on low-level visual content, Computer Vision and Image Understanding, v.93 n.3, p.328-346, March 2004 [doi>10.1016/j.cviu.2003.10.006]
• 25: Dacheng Tao , Xiaoou Tang , Xuelong Li , Xindong Wu, Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.7, p.1088-1099, July 2006 [doi>10.1109/TPAMI.2006.134]
• 26: P. Quelhas , F. Monay , J.-M. Odobez , D. Gatica-Perez , T. Tuytelaars , L. Van Gool, Modeling Scenes with Local Descriptors and Latent Aspects, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.883-890, October 17-20, 2005 [doi>10.1109/ICCV.2005.152]
• 27: S. F. Chang, et. al., Columbia Univ. TRECVID-2005 Video Search and High-Level Feature Extraction, Proc. TREC Video Retrieval Evaluation, 2005.
• 28: M. Campbell, et. al., IBM Research TRECVID-2006 Video Retrieval System, Proc. TREC Video Retrieval Evaluation, 2006.
• 29: M. Boutell and J. Luo, Beyond Pixels: Exploiting camera metadata for photo classification, Pattern Recognition 38(6): 935--946, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
        I.4.10 Image Representation
      I.4.7 Feature Measurement

General Terms:
Algorithms, Experimentation

Keywords:
color histogram, gps, label propagation, photo collection, photo similarities, sift, timestamp