Multimodal question answering for mobile devices

ABSTRACT

This paper introduces multimodal question answering, a new interface for community-based question answering services. By offering users an extra modality---photos---in addition to the text modality to formulate queries, multimodal question answering overcomes the limitations of text-only input methods when the users ask questions regarding visually distinctive objects. Such interface is especially useful when users become curious about an interesting object in the environment and want to know about it---simply by taking a photo and asking a question in a situated (from a mobile device) and intuitive (without describing the object in words) manner. We propose a system architecture for multimodal question answering, describe an algorithm for searching the database, and report on the findings of two prototype studies.

REFERENCES

• 1: Maria-Elena Nilsback , Andrew Zisserman, A Visual Vocabulary for Flower Classification, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1447-1454, June 17-22, 2006 [doi>10.1109/CVPR.2006.42]
• 2: Günther Pospischil , Martina Umlauft , Elke Michlmayr, Designing LoL@, a Mobile Tourist Guide for UMTS, Proceedings of the 4th International Symposium on Mobile Human-Computer Interaction, p.140-154, September 18-20, 2002
• 3: H. Sonobe, S. Takagi, and F. Yoshimoto. Mobile computing system for fish image retrieval. In IWAIT 04, 2004.
• 4: T. Yeh, J. Lee, and T. Darrell. Adaptive vocabulary forests for dynamic indexing and category learning. In ICCV 07, 2007, to appear.
• 5: T. Yeh, K. Tollmar, and T. Darrell. Searching the web with mobile images for location recognition. CVPR 04, 02:76--81, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.3 Microcomputers
          Subjects: Portable devices (e.g., laptops, personal digital assistants)

  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.2 Nonnumerical Algorithms and Problems
          Subjects: Pattern matching

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Design, Human Factors, Performance

Keywords:
information retrieval, mobile application, pattern matching, question answering