A picture is worth a thousand keywords: exploring mobile image-based web search

ABSTRACT

Images of objects as queries is a new approach to search for information on the web. Image-based information retrieval goes beyond only matching images, as information in other modalities also can be extracted from data collections using image search. We have developed a new system that uses images to search for web-based information. This paper has a particular focus towards exploring user's experience of general mobile image-based Web searches to find what issues and phenomena it contains. This was achieved in a multi-part study by creating and letting respondents test prototypes of mobile image-based search systems and collecting data using interviews, observations, video-observations, and questionnaires. We observed that searching for information only based on visual similarity and without any assistance is sometimes difficult, especially on mobile devices with limited interaction bandwidth. Most of our subjects preferred a search tool that guides the users through the search result based on contextual information, compared to presenting the search result as a plain ranked list.

REFERENCES

• 1: Yuksel Alp Aslandogan , Clement T. Yu, Evaluating strategies and systems for content based indexing of person images on the Web, Proceedings of the eighth ACM international conference on Multimedia, p.313-321, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354514]
• 2: Berners-Lee, T., Hendler, J. & Lassila, O.: The semantic Web. Scientific American 284, 5, p. 34--43, 2000.
• 3: Biever, C.: See it, snap it, and let your phone find it on the web. New Scientist, p. 20--21, 2006.
• 4: Orkut Buyukkokten , Hector Garcia-Molina , Andreas Paepcke , Terry Winograd, Power browser: efficient Web browsing for PDAs, Proceedings of the SIGCHI conference on Human factors in computing systems, p.430-437, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332470]
• 5: Nigel Davies , Keith Cheverst , Alan Dix , Andre Hesse, Understanding the role of image recognition in mobile tour guides, Proceedings of the 7th international conference on Human computer interaction with mobile devices & services, September 19-22, 2005, Salzburg, Austria [doi>10.1145/1085777.1085809]
• 6: Xin Fan , Xing Xie , Zhiwei Li , Mingjing Li , Wei-Ying Ma, Photo-to-search: using multimodal queries to search the web from mobile devices, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101851]
• 7: Jennifer Fleming, Web navigation: designing the user experience, O'Reilly & Associates, Inc., Sebastopol, CA, 1998
• 8: Anne Kaikkonen , Virpi Roto, Navigating in a mobile XHTML application, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642669]
• 9: Loel Kim , Michael J. Albers, Web design issues when searching for information in a small screen display, Proceedings of the 19th annual international conference on Computer documentation, October 21-24, 2001, Sante Fe, New Mexico, USA [doi>10.1145/501516.501555]
• 10: Alexandros Labrinidis , Nick Roussopoulos, WebView materialization, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.367-378, May 15-18, 2000, Dallas, Texas, United States
• 11: Maarek, Y., Soffer, A. and Chang, B.: Working Notes of the WWW2002 Workshop On Mobile Search, 2002.
• 12: Sharon McDonald , Ting-Sheng Lai , John Tait, Evaluating a content based image retrieval system, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-240, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383993]
• 13: Sharon McDonald , John Tait, Search strategies in content-based image retrieval, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860452]
• 14: Miles, M. B. & Huberman, A. M.: Qualitative Data Analysis. New Bury, CA: Sage, 1994.
• 15: Erica Newcomb , Toni Pashley , John Stasko, Mobile computing in the retail arena, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642670]
• 16: Niblack W., R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos and G. Taubin.: The QBIC project, SPIE Storage and Retrieval for Image and Video Databases, p. 173--187, 1993.
• 17: Stan Sclaroff , Leonid Taycher , Marco La Cascia, ImageRover: A Content-Based Image Browser for the World Wide Web, Proceedings of the 1997 Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '97), p.2, June 20-20, 1997
• 18: Smith, J. R. And Chang, S.-F.: An image and video search engine for the World-Wide Web. In Proceedings of the SPIE Conference on Storage and Retrieval for Image and Video Databases V (IS&T/SPIE, San Jose, CA), p. 84--95, 1997.
• 19: Tollmar K., Yeh T, and Darrell T.: Searching the Web with Mobile Images for Location Recognition, Proceedings of MOBILE HCI 2004.
• 20: Allison Woodruff , Andrew Faulring , Ruth Rosenholtz , Julie Morrsion , Peter Pirolli, Using thumbnails to search the Web, Proceedings of the SIGCHI conference on Human factors in computing systems, p.198-205, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365098]
• 21: Tom Yeh , Kristen Grauman , Konrad Tollmar , Trevor Darrell, A picture is worth a thousand keywords: image-based object search on a mobile platform, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057083]
• 22: Yeh T, Tollmar K. and Darrell T.: Searching the Web with Mobile Images, IEEE Computer Vision and Pattern Recognition, In Proceedings of CVPR 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Design, Human Factors

Keywords:
content-based image retrieval, mobile information retrieval, mobile interface, object recognition