Beyond attention: the role of deictic gesture in intention recognition in multimodal conversational interfaces

ABSTRACT

In a multimodal conversational interface supporting speech and deictic gesture, deictic gestures on the graphical display have been traditionally used to identify user attention, for example, through reference resolution. Since the context of the identified attention can potentially constrain the associated intention, our hypothesis is that deictic gestures can go beyond attention and apply to intention recognition. Driven by this assumption, this paper systematically investigates the role of deictic gestures in intention recognition. We experiment with different model-based methods and instancebased methods to incorporate gestural information for intention recognition. We examine the effects of utilizing gestural information in two different processing stages: speech recognition stage and language understanding stage. Our empirical results have shown that utilizing gestural information improves intention recognition. The performance is further improved when gestures are incorporated in both speech recognition and language understanding stages compared to either stage alone.

REFERENCES

• 1: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 2: J. Chai, S. Pan, and M. Zhou. Mind: A context-based multimodal interpretation framework in conversational systems. In O. Bernsen, L. Dybkjaer, and J. van Kuppevelt, editors, Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems. Kluwer Academic Publishers, 2005.
• 3: J. Chai, Z. Prasov, and S. Qu. Cognitive principles in robutst multimodal interpretation. Journal of Artificial Intelligence Research, 27:55--83, 2006.
• 4: Joyce Y. Chai , Shaolin Qu, A salience driven approach to robust input interpretation in multimodal conversational systems, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.217-224, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220603]
• 5: Joyce Y. Chai , Pengyu Hong , Michelle X. Zhou, A probabilistic approach to reference resolution in multimodal user interfaces, Proceedings of the 9th international conference on Intelligent user interfaces, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964457]
• 6: A. Chotimongkol and A. Rudnicky. N-best speech hypotheses reordering using linear regression. In Proceedings of 7th EUROSPEECH, pages 1829--1832, 2001.
• 7: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 8: Corinna Cortes , Vladimir Vapnik, Support-Vector Networks, Machine Learning, v.20 n.3, p.273-297, Sept. 1995 [doi>10.1023/A:1022627411411]
• 9: J. Eisenstein and C. M. Christoudias. A salience-based approach to gesture-speech alignment. In Proceedings of HLT/NAACL'04, 2004.
• 10: Malte Gabsdil , Oliver Lemon, Combining acoustic and pragmatic features to predict recognition performance in spoken dialogue systems, Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, p.343-es, July 21-26, 2004, Barcelona, Spain [doi>10.3115/1218955.1218999]
• 11: Barbara J. Grosz , Candace L. Sidner, Attention, intentions, and the structure of discourse, Computational Linguistics, v.12 n.3, p.175-204, July-September 1986
• 12: A. Gruenstein, C. Wang, and S. Seneff. Context-sensitive statistical language modeling. In Proceedings of Eurospeech'05, 2005.
• 13: J. Gustafson, L. Bell, J. Beskow, B. J., R. Carlson, J. Edlund, B. Granstrom, H. D., and M. Wiren. Adapt - a multimodal conversational dialogue system in an apartment domain. In Proceedings of 6th International Conference on Spoken Language Processing (ICSLP), 2000.
• 14: C.-W. Hsu and C.-J. Lin. A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks, 13:415--425, 2002.
• 15: Carla Huls , Wim Claassen , Edwin Bos, Automatic referent resolution of deictic and anaphoric expressions, Computational Linguistics, v.21 n.1, p.59-79, March 1995
• 16: S. Iba, C. Paredis, and P. Khosla. Intention aware interactive multi-modal robot programming. In Proceedings of IEEE/RSJ International Conference on - Intelligent Robots and Systems, 2003.
• 17: Michael Johnston , Srinivas Bangalore , Gunaranjan Vasireddy , Amanda Stent , Patrick Ehlen , Marilyn Walker , Steve Whittaker , Preetam Maloor, MATCH: an architecture for multimodal dialogue systems, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073146]
• 18: Z. Kazi, S. Chen, M. Beitler, D. Chester, and R. Foulds. Multimodal hci for robot control: Towards an intelligent robotic assistant for people with disablities. In Proceedings of AAAI'96 Fall Symposium on Developing AI Applications for the Disabled, 1996.
• 19: Andrew Kehler, Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.685-690, July 30-August 03, 2000
• 20: P. Kiefer and C. Schlieder. Exploring context-sensitivity in spatial intention recognition. In Proceedings of the Workshop on Behaviour Monitoring and Interpretation (BMI'07), 2007.
• 21: Oliver Lemon , Alexander Gruenstein, multithreaded context for robust conversational interfaces: Context-sensitive speech recognition and interpretation of corrective fragments, ACM Transactions on Computer-Human Interaction (TOCHI), v.11 n.3, p.241-267, September 2004 [doi>10.1145/1017494.1017496]
• 22: J. G. Neal , C. Y. Thielman , Z. Dobes , S. M. Haller , S. C. Shapiro, Natural language with integrated deictic and graphic gestures, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 23: S. Oviatt. Mulitmodal interactive maps: Designing for human performance. Human-Computer Interaction, 12:93--129, 1997.
• 24: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 25: Shaolin Qu , Joyce Y. Chai, Salience modeling based on non-verbal modalities for spoken language understanding, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181036]
• 26: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 27: Pernilla Qvarfordt , Shumin Zhai, Conversing with the user based on eye-gaze patterns, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055004]
• 28: D. Roy and N. Mukherjee. Towards situated speech understanding: Visual context priming of language models. Computer Speech and Language, 19(2):227--248, 2005.
• 29: R. A. Solsona, E. Fosler-Lussier, H.-K. J. Kuo, A. Potamianos, and I. Zitouni. Adaptive language models for spoken dialogue systems. In Proceedings of ICASSP, 2002.
• 30: Karim A. Tahboub, Intelligent Human-Machine Interaction Based on Dynamic Bayesian Networks Probabilistic Intention Recognition, Journal of Intelligent and Robotic Systems, v.45 n.1, p.31-52, January 2006 [doi>10.1007/s10846-005-9018-0]
• 31: W. Walker, P. Lamere, P. Kwok, B. Raj, R. Singh, E. Gouvea, P. Wolf, and J. Woelfel. Sphinx-4: A flexible open source framework for speech recognition. Technical Report TR-2004-139, Sun Microsystems Laboratories, 2004.
• 32: S.-J. Youn and K.-W. Oh. Intention recognition using a graph representation. International Journal of Applied Science, Engineering and Techcnology, 4:13--18, 2007.
• 33: M. Zancanaro, O. Stock, and C. Strapparava. Multimodal interaction for information access: Exploiting cohesion. Computational Intelligence, 13(7):439--464, 1997.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Keywords:
gesture, language understanding, multimodal interface, speech