Generating diverse and representative image search results for landmarks

ABSTRACT

Can we leverage the community-contributed collections of rich media on the web to automatically generate representative and diverse views of the world's landmarks? We use a combination of context- and content-based tools to generate representative sets of images for location-driven features and landmarks, a common search task. To do that, we using location and other metadata, as well as tags associated with images, and the images' visual features. We present an approach to extracting tags that represent landmarks. We show how to use unsupervised methods to extract representative views and images for each landmark. This approach can potentially scale to provide better search and representation for landmarks, worldwide. We evaluate the system in the context of image search using a real-life dataset of 110,000 images from the San Francisco area.

REFERENCES

• 1: S. Ahern, S. King, M. Naaman, R. Nair, and J. H.-I. Yang. ZoneTag: Rich, community-supported context-aware media capture and annotation. In Workshop on Mobile Spatial Interaction (MSI) at the SIGCHI conference on Human Factors in computing systems (CHI '07), 2007.
• 2: Shane Ahern , Mor Naaman , Rahul Nair , Jeannie Hui-I Yang, World explorer: visualizing aggregate data from unstructured text in geo-referenced collections, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255177]
• 3: T. L. Berg and D. A. Forsyth. Automatic ranking of iconic images. Technical report, U.C. Berkeley, January 2007.
• 4: Deng Cai , Xiaofei He , Zhiwei Li , Wei-Ying Ma , Ji-Rong Wen, Hierarchical clustering of WWW image search results using visual, textual and link information, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027747]
• 5: C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
• 6: S. Chang, W. Hsu, L. Kennedy, L. Xie, A. Yanagawa, E. Zavesky, and D. Zhang. Columbia University TRECVID-2005 Video Search and High-Level Feature Extraction. NIST TRECVID Workshop, Gaithersburg, MD, November, 2005.
• 7: M. Davis, M. Smith, F. Stentiford, A. Bambidele, J. Canny, N. Good, S. King, and R. Janakiraman. Using context and similarity for face and location identification. In Proceedings of the IS&T/SPIE 18th Annual Symposium on Electronic Imaging Science and Technology, 2006.
• 8: Flickr.com, yahoo! inc. http://www.flickr.com.
• 9: Feng Jing , Lei Zhang , Wei-Ying Ma, VirtualTour: an online travel assistant based on high quality images, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180762]
• 10: Lyndon S. Kennedy , Shih-Fu Chang , Igor V. Kozintsev, To search or to label?: predicting the performance of search-based automatic image classifiers, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178712]
• 11: Lyndon Kennedy , Mor Naaman , Shane Ahern , Rahul Nair , Tye Rattenbury, How flickr helps us make sense of the world: context and content in community-contributed media collections, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291384]
• 12: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 13: B. S. Manjunath , W. Y. Ma, Texture Features for Browsing and Retrieval of Image Data, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.837-842, August 1996 [doi>10.1109/34.531803]
• 14: M. Naaman, A. Paepcke, and H. Garcia-Molina. From where to what: Metadata sharing for digital photographs with geographic coordinates. In 10th International Conference on Cooperative Information Systems (CoopIS), 2003.
• 15: Apostol (Paul) Natsev , Milind R. Naphade , Jelena TešiĆ, Learning the semantics of multimedia queries and concepts from a small number of examples, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101288]
• 16: N. O'Hare, C. Gurrin, G. J. Jones, and A. F. Smeaton. Combination of content analysis and context features for digital photograph retrieval. In 2nd IEE European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies, 2005.
• 17: M. S. M. Orengo. Similarity of color images. Proc. SPIE Storage and Retrieval for Image and Video Databases, 2420:381--392, 1995.
• 18: S. Palmer, E. Rosch, and P. Chase. Canonical perspective and the perception of objects. Attention and Performance IX, pages 135--151, 1981.
• 19: Tye Rattenbury , Nathaniel Good , Mor Naaman, Towards automatic extraction of event and place semantics from flickr tags, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277762]
• 20: I. Simon, N. Snavely, and S. M. Seitz. Scene summarization for online image collections. In ICCV '07: Proceedings of the 11th IEEE international Conference on Computer Vision. IEEE, 2007.
• 21: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 22: S. Sontag. On Photography. Picador USA, 2001.
• 23: Kentaro Toyama , Ron Logan , Asta Roseway, Geographic location tags on digital images, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957046]
• 24: C.-M. Tsai, A. Qamra, and E. Chang. Extent: Inferring image metadata from context and content. In IEEE International Conference on Multimedia and Expo, 2005.
• 25: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 26: Shuo Wang , Feng Jing , Jibo He , Qixing Du , Lei Zhang, IGroup: presenting web image search results in semantic clusters, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240718]
• 27: Yi Wu , Edward Y. Chang , Belle L. Tseng, Multimodal metadata fusion using causal strength, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101338]
• 28: Youtube.com, google inc. http://www.youtube.com.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.m MISCELLANEOUS

General Terms:
Algorithms, Human Factors

Keywords:
geo-referenced photographs, photo collections, social media