Robust gesture processing for multimodal interaction

ABSTRACT

With the explosive growth in mobile computing and communication over the past few years, it is possible to access almost any information from virtually anywhere. However, the efficiency and effectiveness of this interaction is severely limited by the inherent characteristics of mobile devices, including small screen size and the lack of a viable keyboard or mouse. This paper concerns the use of multimodal language processing techniques to enable interfaces combining speech and gesture input that overcome these limitations. Specifically we focus on robust processing of pen gesture inputs in a local search application and demonstrate that edit-based techniques that have proven effective in spoken language processing can also be used to overcome unexpected or errorful gesture input. We also examine the use of a bottom-up gesture aggregation technique to improve the coverage of multimodal understanding.

REFERENCES

• 1: Aaron Adler , Randall Davis, Speech and sketching for multimodal design, Proceedings of the 9th international conference on Intelligent user interfaces, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964483]
• 2: Toward conversational human-computer interaction, AI Magazine, v.22 n.4, p.27-37, Winter 2001
• 3: E. André. Natural language in multimedia/multimodal systems. In R. Mitkov, editor, Handbook of Computational Linguistics, pages 650--669. Oxford University Press, 2002.
• 4: S. Bangalore and M. Johnston. Balancing data-driven and rule-based approaches in the context of a multimodal conversational system. In Proceedings of North American Association for Computational Linguistics/Human Language Technology, pages 33--40, Boston, USA, 2004.
• 5: S. Bangalore and M. Johnston. Robust understanding in multimodal interfaces. Accepted for publication in Computational Linguistics, 2008.
• 6: M. Boros, W. Eckert, F. Gallwitz, G. Gğrz, G. Hanrieder, and H. Niemann. Towards understanding spontaneous speech: word accuracy vs. concept accuracy. In Proceedings of International Conference on Spoken Language Processing, pages 41--44, Philadelphia, USA, 1996.
• 7: Justine Cassell, Embodied conversational agents: representation and intelligence in user interfaces, AI Magazine, v.22 n.4, p.67-83, Winter 2001
• 8: Adam Cheyer , Luc Julia, Multimodal Maps: An Agent-Based Approach, Multimodal Human-Computer Communication, Systems, Techniques, and Experiments, p.111-121, January 1998
• 9: A. Ciaramella. A Prototype Performance Evaluation Report. Technical Report WP8000-D3, Project Esprit 2218 SUNDIAL, 1993.
• 10: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, Multimodal interaction for distributed interactive simulation, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 11: P. R. Cohen, M. Johnston, D. McGee, S. L. Oviatt, J. Clow, and I. Smith. The efficiency of multimodal interaction: a case study. In Proceedings of International Conference on Spoken Language Processing, pages 249--252, Sydney, Australia, 1998.
• 12: John Dowding , Jean Mark Gawron , Doug Appelt , John Bear , Lynn Cherny , Robert Moore , Douglas Moran, Gemini: a natural language system for spoken-language understanding, Proceedings of the 31st annual meeting on Association for Computational Linguistics, p.54-61, June 22-26, 1993, Columbus, Ohio [doi>10.3115/981574.981582]
• 13: P. Ehlen, M. Johnston, and G. Vasireddy. Collecting mobile multimodal data for MATCH. In Proceedings of International Conference on Spoken Language Processing, pages 2557--2560, Denver, CO, USA, 2002.
• 14: A. L. Gorin , G. Riccardi , J. H. Wright, How may I help you?, Speech Communication, v.23 n.1-2, p.113-127, Oct. 1997 [doi>10.1016/S0167-6393(97)00040-X]
• 15: J. Gustafson, L. Bell, J. Beskow, J. Boye, R. Carlson, J. Edlund, B. Granstrm, D. House, and M. Wirén. Adapt - a multimodal conversational dialogue system in an apartment domain. In International Conference on Spoken Language Processing, pages 134--137, Beijing, China, 2000.
• 16: Michael Johnston, Deixis and conjunction in multimodal systems, Proceedings of the 18th conference on Computational linguistics, p.362-368, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990873]
• 17: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 18: Michael Johnston , Srinivas Bangalore, Finite-state multimodal integration and understanding, Natural Language Engineering, v.11 n.2, p.159-187, June 2005 [doi>10.1017/S1351324904003572]
• 19: Michael Johnston , Srinivas Bangalore , Gunaranjan Vasireddy , Amanda Stent , Patrick Ehlen , Marilyn Walker , Steve Whittaker , Preetam Maloor, MATCH: an architecture for multimodal dialogue systems, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073146]
• 20: James A. Landay , Brad A. Myers, Sketching Interfaces: Toward More Human Interface Design, Computer, v.34 n.3, p.56-64, March 2001
• 21: Sharon Oviatt, Multimodal interactive maps: designing for human performance, Human-Computer Interaction, v.12 n.1, p.93-129, March 1997 [doi>10.1207/s15327051hci1201&2_4]
• 22: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 23: W. Wahlster. SmartKom: Fusion and fission of speech, gestures, and facial expressions. In Proceedings of the 1st International Workshop on Man-Machine Symbiotic Systems, pages 213--225, Kyoto, Japan, 2002.
• 24: W. Ward, Understanding spontaneous speech: the Phoenix system, Proceedings of the Acoustics, Speech, and Signal Processing, 1991. ICASSP-91., 1991 International Conference, p.365-367, April 14-17, 1991

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
finite-state methods, local search, mobile, multimodal interfaces, robustness, speech-gesture integration