Photo-based question answering

ABSTRACT

Photo-based question answering is a useful way of finding information about physical objects. Current question answering (QA) systems are text-based and can be difficult to use when a question involves an object with distinct visual features. A photo-based QA system allows direct use of a photo to refer to the object. We develop a three-layer system architecture for photo-based QA that brings together recent technical achievements in question answering and image matching. The first, template-based QA layer matches a query photo to online images and extracts structured data from multimedia databases to answer questions about the photo. To simplify image matching, it exploits the question text to filter images based on categories and keywords. The second, information retrieval QA layer searches an internal repository of resolved photo-based questions to retrieve relevant answers. The third, human-computation QA layer leverages community experts to handle the most difficult cases. A series of experiments performed on a pilot dataset of 30,000 images of books, movie DVD covers, grocery items, and landmarks demonstrate the technical feasibility of this architecture. We present three prototypes to show how photo-based QA can be built into an online album, a text-based QA, and a mobile application.

REFERENCES

• 1: 199QUERY: text any question and get the answer by SMS! http://www.199query.com/.
• 2: Amazon.com: Online shopping. http://www.amazon.com/.
• 3: Amazon Mechanical Turk. http://www.mturk.com/.
• 4: Andrea Andrenucci , Eriks Sneiders, Automated Question Answering: Review of the Main Approaches, Proceedings of the Third International Conference on Information Technology and Applications (ICITA'05) Volume 2, p.514-519, July 04-07, 2005 [doi>10.1109/ICITA.2005.78]
• 5: Ask.com search engine -- better web search. http://www.ask.com/.
• 6: AskMeNow -- get answers with search designed for mobile. http://www.askmenow.com/.
• 7: Tamara L. Berg , David A. Forsyth, Animals on the Web, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1463-1470, June 17-22, 2006 [doi>10.1109/CVPR.2006.57]
• 8: ChaCha: Good answer. http://www.chacha.com/.
• 9: C.Y. Chen, T. Kurozumi, and J. Yamato. Poster image matching by color scheme and layout information. In Proc. of ICME '06, pages 345--348, 2006.
• 10: Xin Fan , Xing Xie , Zhiwei Li , Mingjing Li , Wei-Ying Ma, Photo-to-search: using multimodal queries to search the web from mobile devices, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101851]
• 11: Li Fei-Fei , Rob Fergus , Pietro Perona, Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12, p.178, June 27-July 02, 2004
• 12: Christiane Fellbaum, editor. WordNet: An Electronic Lexical Database (Language, Speech, and Communication). The MIT Press, May 1998.
• 13: Flickr: Photo sharing. http://www.flickr.com/.
• 14: ArchitectureWeek great buildings collection. http://www.greatbuildings.com/.
• 15: Daniel Jurafsky , James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, Prentice Hall PTR, Upper Saddle River, NJ, 2000
• 16: B. Katz. Annotating the world wide web using natural language. In Proc. of RIAO '97, 1997.
• 17: Boris Katz , Sue Felshin , Deniz Yuret , Ali Ibrahim , Jimmy J. Lin , Gregory Marton , Alton Jerome McFarland , Baris Temelkuran, Omnibase: Uniform Access to Heterogeneous Data for Question Answering, Proceedings of the 6th International Conference on Applications of Natural Language to Information Systems-Revised Papers, p.230-234, June 27-28, 2002
• 18: B. Katz, J. Lin, C. Stauffer, and E. Grimson. Answering questions about moving objects in surveillance videos. In Proc. of AAAI Spring Symposium on New Directions in Question Answering '03, 2003.
• 19: Y. Ke and R. Sukthankar. PCA-SIFT: a more distinctive representation for local image descriptors. In Proc. of CVPR '04, volume 2, pages 506--513, 2004.
• 20: Asian online market and groceries superstore. http://www.koamart.com/.
• 21: J. Matas, O. Chum, U. Martin, and T. Pajdla. Robust wide baseline stereo from maximally stable extremal regions. In Proc. of BMVC, volume 1, pages 384--393, 2002.
• 22: Naver knowledge search. http://kin.naver.com/.
• 23: David Nister , Henrik Stewenius, Scalable Recognition with a Vocabulary Tree, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2161-2168, June 17-22, 2006 [doi>10.1109/CVPR.2006.264]
• 24: J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In Proc. of CVPR '07, pages 1--8, 2007.
• 25: B. Platel, E. Balmachnova, L.M.J. Florack, and Ter. Top-points as interest points for image matching. In Proc. of ECCV '06, pages 418--429, 2006.
• 26: B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for image annotation. MIT AI Lab Memo AIM-2005-025, 2005.
• 27: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 28: SnapTell: Image recognition based mobile marketing. http://www.snaptell.com/.
• 29: Eriks Sneiders, Automated Question Answering Using Question Templates That Cover the Conceptual Model of the Database, Proceedings of the 6th International Conference on Applications of Natural Language to Information Systems-Revised Papers, p.235-239, June 27-28, 2002
• 30: Wanpeng Song , Min Feng , Naijie Gu , Liu Wenyin, Question Similarity Calculation for FAQ Answering, Proceedings of the Third International Conference on Semantics, Knowledge and Grid, p.298-301, October 29-31, 2007 [doi>10.1109/SKG.2007.247]
• 31: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 32: E. Voorhees. The TREC-8 question answering track report, 1999.
• 33: Wikipedia: the free encyclopedia. http://www.wikipedia.com/.
• 34: Yahoo! Answers. http://answers.yahoo.com/.
• 35: Hui Yang , Lekha Chaisorn , Yunlong Zhao , Shi-Yong Neo , Tat-Seng Chua, VideoQA: question answering on news video, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957146]
• 36: B. Yao, X. Yang, and S.C. Zhu. Introduction to a large-scale general purpose ground truth database: methodology, annotation tool and benchmarks. In Proc. of EMMCVPR '04, pages 169--183, 2007.
• 37: Tom Yeh , Trevor Darrell, Multimodal question answering for mobile devices, Proceedings of the 13th international conference on Intelligent user interfaces, January 13-16, 2008, Gran Canaria, Spain [doi>10.1145/1378773.1378841]
• 38: Tom Yeh , Kristen Grauman , Konrad Tollmar , Trevor Darrell, A picture is worth a thousand keywords: image-based object search on a mobile platform, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057083]
• 39: T. Yeh, J.J. Lee, and T. Darrell. Adaptive vocabulary forests br dynamic indexing and category learning. In Proc. of ICCV '07, pages 1--8, 2007.
• 40: T. Yeh, K. Tollmar, and T. Darrell. Searching the web with mobile images for location recognition. In Proc. of CVPR '04, volume 2, pages 76--81, 2004.
• 41: Y. Zhang, L. Wang, R. Hartley, and H. Li. Where's the weet-bix? In Proc. of ACCV '07, pages 800--810, 2007.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

General Terms:
Algorithms, Design, Human Factors

Keywords:
computer vision, information retrieval, question answering