World-scale mining of objects and events from community photo collections

ABSTRACT

In this paper, we describe an approach for mining images of objects (such as touristic sights) from community photo collections in an unsupervised fashion. Our approach relies on retrieving geotagged photos from those web-sites using a grid of geospatial tiles. The downloaded photos are clustered into potentially interesting entities through a processing pipeline of several modalities, including visual, textual and spatial proximity. The resulting clusters are analyzed and are automatically classified into objects and events. Using mining techniques, we then find text labels for these clusters, which are used to again assign each cluster to a corresponding Wikipedia article in a fully unsupervised manner. A final verification step uses the contents (including images) from the selected Wikipedia article to verify the cluster-article assignment. We demonstrate this approach on several urban areas, densely covering an area of over 700 square kilometers and mining over 200,000 photos, making it probably the largest experiment of its kind to date.

REFERENCES

• 1: Rakesh Agrawal , Tomasz Imieliński , Arun Swami, Mining association rules between sets of items in large databases, Proceedings of the 1993 ACM SIGMOD international conference on Management of data, p.207-216, May 25-28, 1993, Washington, D.C., United States
• 2: M. Aurnhammer, P. Hanappe, and L. Steels. Integrating collaborative tagging and emergent semantics for image retrieval. In Collaborative Web Tagging Workshop (WWW'06), 2006.
• 3: H. Bay, T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. In ECCV'06, 2006.
• 4: Christian Borgelt, An implementation of the FP-growth algorithm, Proceedings of the 1st international workshop on open source data mining: frequent pattern mining implementations, p.1-5, August 21-21, 2005, Chicago, Illinois [doi>10.1145/1133905.1133907]
• 5: A. Bosch, A. Zisserman, and X. Muñoz. Scene classification via pLSA. In ECCV'06, 2006.
• 6: W. B. Croft and D. J. Harper. Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35, 1997.
• 7: R. Fergus, P. Perona, and A. Zisserman. Object class recognition by unsupervised scale-invariant learning. In CVPR'03, 2003.
• 8: Martin A. Fischler , Robert C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, v.24 n.6, p.381-395, June 1981 [doi>10.1145/358669.358692]
• 9: M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. Seitz. Multi-view stereo for community photo collections. In ICCV'07, 2007.
• 10: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer visiond, Cambridge University Press, New York, NY, 2001
• 11: Alexandar Jaffe , Mor Naaman , Tamir Tassa , Marc Davis, Generating summaries and visualization for large collections of geo-referenced photographs, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178692]
• 12: Bastian Leibe , Edgar Seemann , Bernt Schiele, Pedestrian Detection in Crowded Scenes, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.878-885, June 20-26, 2005 [doi>10.1109/CVPR.2005.272]
• 13: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 14: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 15: J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust wide-baseline stereo from maximally stable extremal regions. In BMVC'02, 2002.
• 16: Krystian Mikolajczyk , Cordelia Schmid, Scale & Affine Invariant Interest Point Detectors, International Journal of Computer Vision, v.60 n.1, p.63-86, October 2004 [doi>10.1023/B:VISI.0000027790.02288.f2]
• 17: David Nister , Henrik Stewenius, Scalable Recognition with a Vocabulary Tree, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2161-2168, June 17-22, 2006 [doi>10.1109/CVPR.2006.264]
• 18: L. Paletta, G. Fritz, C. Seifert, P. Luley, and A. Almer. A mobile vision service for multimedia tourist applications in urban environments. In IEEE Intel. Transp. Syst. Conf., 2006.
• 19: J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR'07, 2007.
• 20: T. Quack, H. Bay, and L. Van Gool. Object recognition for the internet of things. In Internet of Things 2008, 2008.
• 21: T. Quack, V. Ferrari, and L. Van Gool. Video mining with frequent itemset configurations. In CIVR'06, 2006.
• 22: J. R. Quinlan, Induction of Decision Trees, Machine Learning, v.1 n.1, p.81-106 [doi>10.1023/A:1022643204877]
• 23: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 24: S. Savarese and L. Fei-Fei. 3d generic object categorization, localization and pose estimation. In ICCV'07, 2007.
• 25: I. Simon, N. Snavely, and S. M. Seitz. Scene summarization for online image collections. In ICCV'07, 2007.
• 26: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 27: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 28: J. Sivic and A. Zisserman. Video data mining using configurations of viewpoint invariant regions. In CVPR'04, 2004.
• 29: J. Sivic and A. Zisserman. Video data mining using configurations of viewpoint invariant regions. In CVPR'04, 2004.
• 30: T. Tuytelaars and L. Van Gool. Wide baseline stereo matching based on local, affinely invariant regions. In BMVC'00, 2000.
• 31: Maarten Vergauwen , Luc Van Gool, Web-based 3D Reconstruction Service, Machine Vision and Applications, v.17 n.6, p.411-426, October 2006 [doi>10.1007/s00138-006-0027-1]
• 32: A. Webb. Statistical Pattern Recognition. Wiley, second edition, 2002.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Design, Measurement, Theory

Keywords:
database, geo-referenced, image, mining, object recognition, photo collection, retrieval, web