Skipping spare information in multimodal inputs during multimodal input fusion

ABSTRACT

In a multimodal interface, a user can use multiple modalities, such as speech, gesture, and eye gaze etc., to communicate with a system. As a critical component in a multimodal interface, multimodal input fusion explores the ways to effectively interpret the combined semantic interpretation of user's multimodal inputs. Although multimodal inputs may contain spare information, few multimodal input fusion approaches have tackled how to deal with spare information in multimodal inputs. This paper proposes a novel multimodal input fusion approach to flexibly skip spare information in multimodal inputs and derive semantic interpretation of them. The evaluation about the proposed approach confirms that the approach makes human-computer interaction more natural and smooth.

REFERENCES

• 1: Bangalore S., and Johnston M. Integrating multimodal language processing with speech recognition. In Proceedings of Sixth International Conference on Spoken Language Processing. October 16-20, Beijing, China, 2000.
• 2: Robert J. K. Jacob, The use of eye movements in human-computer interaction techniques: what you look at is what you get, ACM Transactions on Information Systems (TOIS), v.9 n.2, p.152-169, April 1991 [doi>10.1145/123078.128728]
• 3: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 4: Ed Kaiser , David Demirdjian , Alexander Gruenstein , Xiaoguang Li , John Niekrasz , Matt Wesson , Sanjeev Kumar, A multimodal learning interface for sketch, speak and point creation of a schedule chart, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027992]
• 5: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 6: Rudzicz F. Clavius: Bi-Directional Parsing for Generic Multimodal Interaction. In Proceedings of Joint Conference of the International Committee on Computational Linguistics and the Association for Computational Linguistics (COLING/ACL-06 SRW), July 17-21, Sydney, Australia 2006.
• 7: Yong Sun , Helmut Prendinger , Yu (David) Shi , Fang Chen , Vera Chung , Mitsuru Ishizuka, The hinge between input and output: understanding the multimodal input fusion results in an agent-based multimodal presentation system, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358878]
• 8: Yong Sun , Yu Shi , Fang Chen , Vera Chung, An efficient unification-based multimodal language processor in multimodal input fusion, Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces, November 28-30, 2007, Adelaide, Australia [doi>10.1145/1324892.1324936]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

General Terms:
Algorithms, Design, Theory

Keywords:
processing of multimodal input, spare information in multimodal input