From context to content: leveraging context to infer media metadata

ABSTRACT

The recent popularity of mobile camera phones allows for new opportunities to gather important metadata at the point of capture. This paper describes a method for generating metadata for photos using spatial, temporal, and social context. We describe a system we implemented for inferring location information for pictures taken with camera phones and its performance evaluation. We propose that leveraging contextual metadata at the point of capture can address the problems of the semantic and sensory gaps. In particular, combining and sharing spatial, temporal, and social contextual metadata from a given user and across users allows us to make inferences about media content.

REFERENCES

• 1: Aigrain, P., Zhang, H. and Petkovic, D. Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review. Multimedia Tools and Applications, 3, 3 (Nov. 1996), 179--202.
• 2: Marc Davis, Media Streams: an iconic visual language for video representation, Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 3: Davis, M. Active Capture: Integrating Human-Computer Interaction and Computer Vision/Audition to Automate Media Capture. In Proc. of 2003 IEEE International Conference on Multimedia and Expo (ICME2003) Special Session on Moving from Features to Semantics Using Computational Media Aesthetics (Baltimore, MD, July 6-9, 2003). IEEE Computer Society Press, New York, NY, 2003, Vol. II, 185--188.
• 4: Marc Davis, Active capture: automatic direction for automatic movies, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957141]
• 5: Davis, M. and Sarvas, R. Mobile Media Metadata for Mobile Imaging. In Proc. of 2004 IEEE International Conference on Multimedia and Expo (ICME2004) Special Session on Mobile Imaging (Taipei, Taiwan, June 27-30, 2004). IEEE Computer Society Press, New York, NY, 2004.
• 6: Marc Davis, Mobile media metadata: metadata creation system for mobile images, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027741]
• 7: Anind K. Dey, Understanding and Using Context, Personal and Ubiquitous Computing, v.5 n.1, p.4-7, February 2001 [doi>10.1007/s007790170019]
• 8: Chitra Dorai , Svetha Venkatesh, Computational Media Aesthetics: Finding Meaning Beautiful, IEEE MultiMedia, v.8 n.4, p.10-12, October 2001 [doi>10.1109/93.959093]
• 9: Ken Haase , David Tamés, BabelVision: better image searching through shared annotations, interactions, v.11 n.2, March + April 2004 [doi>10.1145/971258.971266]
• 10: Hull, R., Kumar, B., Lieuwen, D., Patel-Schneider, P. F., Sahuguet, A., Varadarajan, S., and Vyas, A. "Enabling Context-Aware and Privacy-Conscious User Data Sharing. In Proc. of 2004 IEEE International Conference on Mobile Data Management (MDM'04) (Berkeley, CA, January 19-22, 2004). IEEE Computer Society Press, New York, NY, 2004, 187--198.
• 11: Henry Lieberman , Elizabeth Rozenweig , Push Singh, Aria: An Agent for Annotating and Retrieving Images, Computer, v.34 n.7, p.57-62, July 2001 [doi>10.1109/2.933504]
• 12: Naaman, M., Paepcke, A., and Garcia-Molina, H. From Where to What: Metadata Sharing for Digital Photographs with Geographic Coordinates. In Proc. of 10th International Conference on Cooperative Information Systems (CoopIS) (Catania, Sicily, November 3-7, 2003). Springer-Verlag, Heidelberg, Germany, 2003, 196--217.
• 13: Risto Sarvas , Erick Herrarte , Anita Wilhelm , Marc Davis, Metadata creation system for mobile images, Proceedings of the 2nd international conference on Mobile systems, applications, and services, June 06-09, 2004, Boston, MA, USA [doi>10.1145/990064.990072]
• 14: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 15: Kentaro Toyama , Ron Logan , Asta Roseway, Geographic location tags on digital images, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957046]
• 16: Vartiainen, P. Using Metadata and Context Information in Sharing Personal Content of Mobile Users, Master's Thesis, University of Helsinki, Finland, 2003.
• 17: Anita Wilhelm , Yuri Takhteyev , Risto Sarvas , Nancy Van House , Marc Davis, Photo annotation on a camera phone, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986075]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
      H.3.3 Information Search and Retrieval
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
      H.5.3 Group and Organization Interfaces

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.m Miscellaneous

General Terms:
Algorithms, Design, Human Factors

Keywords:
content-based image retrieval, context-to-content inference, contextual metadata, location-based services, mobile camera phones, wireless multimedia applications