Leveraging probabilistic season and location context models for scene understanding

ABSTRACT

Recent research has shown the power of context-aware scene understanding in bridging the semantic gap between high-level semantic concepts and low-level image features. In this paper, we present a new method to exploit nonvisual context information from the season and location proximity in which pictures were taken to facilitate region (object) annotation in consumer photos. Our method does not require precise time and location from the capture device or user input. Instead, it learns from rough location (e.g., states in the US) and time (e.g., seasons) information, which can be obtained through picture metadata automatically or through minimal user input (e.g., grouping). In addition, the visual context within the image is obtained by analyzing the spatial relationships between different regions (objects) in the scene. Both visual and nonvisual context information are fused using a probabilistic graphical model to improve the accuracy of object region recognition. Our method has been evaluated on a database that consists of over 10,000 regions in more than 1000 images collected from both the Web and consumers. Experimental results show that incorporating the season and location context significantly improves the performance of region recognition.

REFERENCES

• 1: Berretti, S., Bimbo, A. D., and Vicario, E. 2002. Spatial arrangement of color in retrieval by visual similarity. Pattern Recog. 35 (2002) 1661--1674.
• 2: Boutell, M., Luo, J., and Brown, C. M. 2004. Learning spatial configuration models using modified Dirichlet priors. In Proceedings of the Workshop on Statistical Relational Learning (in Conjunction with ICML) (2004).
• 3: Boutell, M., Choudhury, A., Luo, J., and Brown, C. M. 2006. Using semantic features for scene classification: How good do they need to be? In Proceedings of the IEEE International Conference on Multimedia and Exposition. (Jul. 2006).
• 4: Boutell, M., Luo, J., and Brown, C. 2007. Scene parsing using region-based generative models. IEEE Trans. Multimedia.
• 5: Yixin Chen , James Z. Wang, Image Categorization by Learning and Reasoning with Regions, The Journal of Machine Learning Research, 5, p.913-939, 12/1/2004
• 6: Christopher M. Christoudias, Synergism in Low Level Vision, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 4, p.40150, August 11-15, 2002
• 7: Dorin Comaniciu , Peter Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.603-619, May 2002 [doi>10.1109/34.1000236]
• 8: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 9: Flickr, http://www.flickr.com
• 10: Niels Haering , Niels de Vitoria Lobo, Features and classification methods to locate deciduous trees in images, Computer Vision and Image Understanding, v.75 n.1-2, p.133-149, July/Aug. 1999 [doi>10.1006/cviu.1999.0769]
• 11: Kohavi, R. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the International Joint Conference on Artificial Intelligence (1995).
• 12: Kschischang, F. R., Frey, B. H., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Info. Theory. 47(2) (Feb. 2001) 498--519.
• 13: Sanjiv Kumar , Martial Hebert, A Hierarchical Field Framework for Unified Context-Based Classification, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1284-1291, October 17-20, 2005 [doi>10.1109/ICCV.2005.9]
• 14: P. Meer , B. Georgescu, Edge Detection with Embedded Confidence, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.12, p.1351-1365, December 2001 [doi>10.1109/34.977560]
• 15: Murphy, K., Torralba, A., and Freeman, W. T. 2003. Using the forest to see the trees: A graphical model relating features, objects, and scenes. In Proceedings of the Neural Information Processing Systems (2003).
• 16: Naphade, M. R. and Huang, T. S. 2001. A probabilistic framework for semantic video indexing, filtering, and retrieval. IEEE Trans. Multimedia 3(1) (Mar. 2001) 141--151.
• 17: Naphade, M. and Huang, T. S. 2002. A factor graph framework for semantic video indexing. IEEE Trans. Circ. Syst. Video Technol. 12(1) (Jan. 2002).
• 18: Picasa, http://picasa.google.com/
• 19: Rui, Y., Huang, T. S., and Chang, S. F. 1999. Image retrieval: Current techniques, promising directions and open issues. J. Visual Commun. Image Represent. (1999).
• 20: Eli Saber , A. Murat Tekalp , Reiner Eschbach , Keith Knox, Automatic image annotation using adaptive color classification, Graphical Models and Image Processing, v.58 n.2, p.115-126, March 1996 [doi>10.1006/gmip.1996.0010]
• 21: Singhal, A., Luo, J., and Zhu, W. Probabilistic spatial context models for scene content understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Madison, WI, Jun. 2003).
• 22: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 23: J. R. Smith , C. -S. Li, Decoding Image Semantics Using Composite Region Templates, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.9, June 21-21, 1998
• 24: Vailaya, A. and Jain, A. 2000. "Detecting sky and vegetation in outdoor images," In Proceedings of the SPIE. 3972 (2000).
• 25: Wang, W., Song, Y., and Zhang, A. 2002. Semantics retrieval by content and context of image regions. In Proceedings of the 15th International Conference on Vision Interface. (2002).
• 26: Jinhui Yuan , Jianmin Li , Bo Zhang, Exploiting spatial context constraints for automatic image region annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291379]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Experimentation, Measurement, Performance, Theory

Keywords:
context-aware scene understanding, image retrieval, object/ region detection, season and location proximity, spatial context