Spirittagger: a geo-aware tag suggestion tool mined from flickr

ABSTRACT

Geographically referenced, or "geo-tagged," photo data sets offer tantalizing potential for automated knowledge discovery in the world. By combining tag reranking based on geographic context with content-based image analysis we are able to suggest geographically relevant tags for photos newly tagged with GPS coordinates. These tag suggestions could be used to help users organize their photo collections or improve retrieval systems. Our algorithm weights labels that correspond to pertinent objects, events, neighborhoods, and activities in a region. While previous work with geo-tagged images has focused on representative views of landmarks or estimating location, our tag suggestion tool, SpiritTagger, suggests tags that reveal an insight into the spirit, or genius loci, of a city or region. Experiments on a data set consisting of over 100,000 Flickr photos in Los Angeles and Southern California show that our geographically relevant tag suggestion tool provides a significant improvement in precision-recall performance over baseline image-based similarity suggestion.

REFERENCES

• 1: Shane Ahern , Mor Naaman , Rahul Nair , Jeannie Hui-I Yang, World explorer: visualizing aggregate data from unstructured text in geo-referenced collections, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255177]
• 2: Flickr. http://www.flickr.com/. Website.
• 3: J. Hays and A. A. Efros. IM2GPS: Estimating Geographic Information From a Single Image. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, 2008.
• 4: E. Kasutani and A. Yamada. The MPEG-7 Color Layout Descriptor: A Compact Image Feature Description for High-Speed Image/Video Segment Retrieval. In IEEE International Conference on Image Processing, pages 674--677, 2001.
• 5: Lyndon Kennedy , Mor Naaman , Shane Ahern , Rahul Nair , Tye Rattenbury, How flickr helps us make sense of the world: context and content in community-contributed media collections, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291384]
• 6: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 7: E. Moxley, T. Mei, X.-S. Hua, W.-Y. Ma, and B. Manjunath. Automatic Video Annotation Through Search and Mining. In International Conference on Multimedia and Expo, June 2008.
• 8: M. Naaman, A. Paepcke, and H. Garcia-Molina. From Where to What: Metadata Sharing for Digital Photographs with Geographic Coordinates. In R. Meersman, Z. Tari, and D. C. Schmidt, editors, CoopIS/DOA/ODBASE, volume 2888 of Lecture Notes in Computer Science, pages 196--217. Springer, 2003.
• 9: David Nister , Henrik Stewenius, Scalable Recognition with a Vocabulary Tree, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2161-2168, June 17-22, 2006 [doi>10.1109/CVPR.2006.264]
• 10: E. Nowak, F. Jurie, and B. Triggs. Sampling Strategies for Bag-of-Features Image Classification. In European Conference on Computer Vision. Springer, 2006.
• 11: N. O'Hare, C. Gurrin, G. Jones, and A. F. Smeaton. Combination of Content Analysis and Context Features for Digital Photograph Retrieval. In 2nd IEE European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies, pages 323--328, 2005.
• 12: Panoramio. http://www.panoramio.com/. Website.
• 13: B.S. Manjunath , Phillipe Salembier , Thomas Sikora, Introduction to MPEG-7: Multimedia Content Description Interface, John Wiley & Sons, Inc., New York, NY, 2002
• 14: C. Shirky. Ontology is Overrated - Categories, Links, and Tags.
• 15: Gang Wang , Ye Zhang , Li Fei-Fei, Using Dependent Regions for Object Categorization in a Generative Framework, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1597-1604, June 17-22, 2006 [doi>10.1109/CVPR.2006.324]
• 16: Xin-Jing Wang , Lei Zhang , Feng Jing , Wei-Ying Ma, AnnoSearch: Image Auto-Annotation by Search, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1483-1490, June 17-22, 2006 [doi>10.1109/CVPR.2006.58]
• 17: Webshots. http://www.webshots.com/. Website.
• 18: Hao Zhang , Alexander C. Berg , Michael Maire , Jitendra Malik, SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2126-2136, June 17-22, 2006 [doi>10.1109/CVPR.2006.301]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
data mining, geotagging, image annotation, photo collections, social media