A game based approach to assign geographical relevance to web images

ABSTRACT

Geographical context is very important for images. Millions of images on the Web have been already assigned latitude and longitude information. Due to the rapid proliferation of such images with geographical context, it is still difficult to effectively search and browse them, since we do not have ways to decide their relevance. In this paper, we focus on the geographical relevance of images, which is defined as to what extent the main objects in an image match landmarks at the location where the image was taken. Recently, researchers have proposed to use game based approaches to label large scale data such as Web images. However, previous works have not shown the quality of collected game logs in detail and how the logs can improve existing applications. To answer these questions, we design and implement a Web-based and multi-player game to collect human knowledge while people are enjoying the game. Then we thoroughly analyze the game logs obtained during a three week study with 147 participants and propose methods to determine the image geographical relevance. In addition, we conduct an experiment to compare our methods with a commercial search engine. Experimental results show that our methods dramatically improve image search relevance. Furthermore, we show that we can derive geographically relevant objects and their salient portion in images, which is valuable for a number of applications such as image location recognition.

REFERENCES

• 1: Luis von Ahn, Games with a Purpose, Computer, v.39 n.6, p.92-94, June 2006 [doi>10.1109/MC.2006.196]
• 2: Luis von Ahn , Ruoran Liu , Manuel Blum, Peekaboom: a game for locating objects in images, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124782]
• 3: Luis von Ahn , Mihir Kedia , Manuel Blum, Verbosity: a game for collecting common-sense facts, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124784]
• 4: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 5: Brockett, C. and Dolan, C. B. Echo Chamber: A game for eliciting a colloquial paraphrase corpus, In Proc. of KCVC'05.
• 6: Gustavo Carneiro , Nuno Vasconcelos, A database centric view of semantic image annotation and retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076129]
• 7: Casey Dugan , Michael Muller , David R. Millen , Werner Geyer , Beth Brownholtz , Marty Moore, The dogear game: a social bookmark recommender system, Proceedings of the 2007 international ACM conference on Supporting group work, November 04-07, 2007, Sanibel Island, Florida, USA [doi>10.1145/1316624.1316683]
• 8: Boris Epshtein , Eyal Ofek , Yonatan Wexler , Pusheng Zhang, Hierarchical photo organization using geo-relevance, Proceedings of the 15th annual ACM international symposium on Advances in geographic information systems, November 07-09, 2007, Seattle, Washington [doi>10.1145/1341012.1341036]
• 9: Flickr http://www.flickr.com/
• 10: Geocaching http://www.geocaching.com/
• 11: Google Earth http://earth.google.com/
• 12: GPS Mission http://gpsmission.com/
• 13: Hays, J. and Efros, A. A. IM2GPS: Estimating geographic information from a single image. In Proc. of CVPR'08, pp. 1--8.
• 14: He, X., Cai, D., Wen, J.-R., Ma. W.-Y. and Zhang, H.-J. ImageSeer: Clustering and searching WWW images using link and page layout analysis. Microsoft Technical Report, MSR-TR-2004-38, 2004.
• 15: Yan Huang , Shashi Shekhar , Hui Xiong, Discovering Colocation Patterns from Spatial Data Sets: A General Approach, IEEE Transactions on Knowledge and Data Engineering, v.16 n.12, p.1472-1485, December 2004 [doi>10.1109/TKDE.2004.90]
• 16: Yushi Jing , Shumeet Baluja, Pagerank for product image search, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367540]
• 17: Yan Ke , Xiaoou Tang , Feng Jing, The Design of High-Level Features for Photo Quality Assessment, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.419-426, June 17-22, 2006 [doi>10.1109/CVPR.2006.303]
• 18: Lavrenko, V., Manmatha, R. and Jeon, J. A model for learning the semantics of pictures, NIPS'03.
• 19: Ronny Lempel , Aya Soffer, PicASHOW: pictorial authority search by hyperlinks on the Web, Proceedings of the 10th international conference on World Wide Web, p.438-448, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372098]
• 20: Live Image Search http://www.live.com/?scope=images
• 21: Yu-Fei Ma , Hong-Jiang Zhang, Contrast-based image attention analysis by using fuzzy growing, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957094]
• 22: David A. Shamma , Bryan Pardo, Karaoke callout: using social and collaborative cell phone networking for new entertainment modalities and data collection, Proceedings of the 1st ACM workshop on Audio and music computing multimedia, October 27-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178723.1178743]
• 23: Epimenides Voutsakis , Euripides G. M. Petrakis , Evangelos Milios, Weighted Link Analysis for Logo and Trademark Image Retrieval on the Web, Proceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence, p.581-585, September 19-22, 2005 [doi>10.1109/WI.2005.162]
• 24: Lee Wang , Chuang Wang , Xing Xie , Josh Forman , Yansheng Lu , Wei-Ying Ma , Ying Li, Detecting dominant locations from search queries, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076107]
• 25: Xin-Jing Wang , Lei Zhang , Feng Jing , Wei-Ying Ma, AnnoSearch: Image Auto-Annotation by Search, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1483-1490, June 17-22, 2006 [doi>10.1109/CVPR.2006.58]
• 26: Wei Zhang , Jana Kosecka, Image Based Localization in Urban Environments, Proceedings of the Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), p.33-40, June 14-16, 2006 [doi>10.1109/3DPVT.2006.80]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Web-based interaction

General Terms:
Algorithms, Design, Human Factors

Keywords:
geographical relevance, human computation, image annotation, image search