Event recognition: viewing the world with a third eye

ABSTRACT

Semantic event recognition based only on vision cues is a challenging problem. This problem is particularly acute when the application domain is unconstrained still images available on the Internet or in personal repositories. In recent years, it has been shown that metadata captured with pictures can provide valuable contextual cues complementary to the image content and can be used to improve classification performance. With the recent geotagging phenomenon, an important piece of metadata available with many geotagged pictures now on the World Wide Web is GPS information. In this study, we obtain satellite images corresponding to picture location data and investigate their novel use to recognize the picture-taking environment, as if through a third eye above the object. Additionally, we combine this inference with classical vision-based event detection methods and study the synergistic fusion of the two approaches. We employ both color- and structure-based visual vocabularies for characterizing ground and satellite images, respectively. Training of satellite image classifiers is done using a multiclass AdaBoost engine while the ground image classifiers are trained using SVMs. Modeling and prediction involve some of the most interesting semantic event-activity classes encountered in consumer pictures, including those that occur in residential areas, commercial areas, beaches, sports venues, and parks. The powerful fusion of the complementary views achieves significant performance improvement over the ground view baseline. With integrated GPS-capable cameras on the horizon, we believe that our line of research can revolutionize event recognition and media annotation in years to come.

REFERENCES

• 1: Vailaya, A., Figueiredo, M., Jain, A., and Zhang, H.-J. 1999. Content-based hierarchical classification of vacation images. In Proceedings of IEEE Multimedia Systems, 1999.
• 2: Li, L.-J. and Fei-Fei, L. 2007. What, where and who? Classifying event by scene and object recognition. In Proceedings of International Conference on Computer Vision, 2007.
• 3: Jain, V. and Singhal, A. 2008. Selective hidden random fields: Exploiting domain specific saliency for event classification. In Proceedings of International Conference on Computer Vision and Pattern Recognition, 2008.
• 4: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 5: Christopher M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 6: Luo, J., Boutell, M., and Brown, C. 2006. Pictures are not taken in a vacuum: An overview of exploiting context for semantic scene content understanding. IEEE Signal Processing Magazine 23(2):101--114, 2006.
• 7: Swain, P. H. 1978. Fundamentals of Pattern Recognition in Remote Sensing. McGraw-Hill, 1978.
• 8: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2169-2178, June 17-22, 2006 [doi>10.1109/CVPR.2006.68]
• 9: Zhu, J., Rosset. S., Zou, H., and Hastie, T. Multi-class AdaBoost Technique. Technical Report. Stanford University, 2005.
• 10: Wei Hao , Jiebo Luo, Generalized Multiclass AdaBoost and Its Applications to Multimedia Classification, Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop, p.113, June 17-22, 2006 [doi>10.1109/CVPRW.2006.87]
• 11: Morgan Ames , Mor Naaman, Why we tag: motivations for annotation in mobile and online media, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240772]
• 12: Einat Amitay , Nadav Har'El , Ron Sivan , Aya Soffer, Web-a-where: geotagging web content, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009040]
• 13: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 14: Shih-Fu Chang , Dan Ellis , Wei Jiang , Keansub Lee , Akira Yanagawa , Alexander C. Loui , Jiebo Luo, Large-scale multimodal semantic concept detection for consumer video, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290118]
• 15: Y. Chen , X. Y. Chen , F. Y. Rao , X. L. Yu , Y. Li , D. Liu, LORE: an infrastructure to support location-aware services, IBM Journal of Research and Development, v.48 n.5/6, p.601-615, September/November 2004
• 16: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Tagging over time: real-world image annotation by lightweight meta-learning, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291328]
• 17: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 18: Micah Dubinko , Ravi Kumar , Joseph Magnani , Jasmine Novak , Prabhakar Raghavan , Andrew Tomkins, Visualizing tags over time, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135810]
• 19: Hinze A. and Voisard, A. 2003. Location and time-based information delivery in tourism. Advances in Spatial and Temporal Databases. Lecture Notes in Computer Science, 2750 (2003) 489--507.
• 20: Alexandar Jaffe , Mor Naaman , Tamir Tassa , Marc Davis, Generating summaries and visualization for large collections of geo-referenced photographs, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178692]
• 21: Dhiraj Joshi , Milind Naphade , Apostol Natsev, Semantics reinforcement and fusion learning for multimedia streams, Proceedings of the 6th ACM international conference on Image and video retrieval, p.309-316, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282328]
• 22: Lyndon Kennedy , Mor Naaman , Shane Ahern , Rahul Nair , Tye Rattenbury, How flickr helps us make sense of the world: context and content in community-contributed media collections, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291384]
• 23: M. L. Kherfi , D. Ziou , A. Bernardi, Image Retrieval from the World Wide Web: Issues, Techniques, and Systems, ACM Computing Surveys (CSUR), v.36 n.1, p.35-67, March 2004 [doi>10.1145/1013208.1013210]
• 24: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 25: Jia Li , James Z. Wang, Real-Time Computerized Annotation of Pictures, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.6, p.985-1002, June 2008 [doi>10.1109/TPAMI.2007.70847]
• 26: Juhong Liu , Ouri Wolfson , Huabei Yin, Extracting Semantic Location from Outdoor Positioning Systems, Proceedings of the 7th International Conference on Mobile Data Management, p.73, May 10-12, 2006 [doi>10.1109/MDM.2006.87]
• 27: Jie Yu , Jiebo Luo, Leveraging probabilistic season and location context models for scene understanding, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386379]
• 28: Alexander Loui , Jiebo Luo , Shih-Fu Chang , Dan Ellis , Wei Jiang , Lyndon Kennedy , Keansub Lee , Akira Yanagawa, Kodak's consumer video benchmark data set: concept definition and annotation, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290117]
• 29: Florent Monay , Daniel Gatica-Perez, On image auto-annotation with latent space models, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957070]
• 30: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 31: Jochen Schiller , Agnès Voisard, Location Based Services, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2004
• 32: Cees G. M. Snoek , Marcel Worring , Arnold W. M. Smeulders, Early versus late fusion in semantic video analysis, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101236]
• 33: Yanagawa, A., Chang, S.-F., Kennedy, L., and Hsu, W. 2007. Columbia University's Baseline Detectors for 374 LSCOM Semantic Visual Concepts. Columbia University ADVENT Technical Report, 2007.
• 34: Jun Yang , Rong Yan , Alexander G. Hauptmann, Cross-domain video concept detection using adaptive svms, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291276]
• 35: Dhiraj Joshi , Jiebo Luo, Inferring generic activities and events from image content and bags of geo-tags, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386361]
• 36: http://code.google.com/apis/kml/documentation.
• 37: Guo-Jun Qi , Xian-Sheng Hua , Yong Rui , Jinhui Tang , Tao Mei , Hong-Jiang Zhang, Correlative multi-label video annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291245]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.2 Information Storage
          Subjects: File organization
  H.4 INFORMATION SYSTEMS APPLICATIONS

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
event recognition, geotagged photographs, visual vocabulary