Ambiguity detection in multimodal systems

ABSTRACT

Multimodal systems support users to communicate in a natural way according to their needs. However, the naturalness of the interaction implies that it is hard to find one and only one interpretation of the users' input. Consequently the necessity to define methods for users' input interpretation and ambiguity detection is arising. This paper proposes a theoretical approach based on a Constraint Multiset Grammar combined with Linear Logic, for representing and detecting ambiguities, and in particular semantic ambiguities, produced by the user's input. It considers user's input as a set of primitives defined as terminal elements of the grammar, composing multimodal sentences. The Linear Logic is used to define rules that allow detecting ambiguities connected to the semantics of the user's input. In particular, the paper presents the main features of the user's input and connections between the elements belonging to a multimodal sentence, and it enables to detect ambiguities that can arise during their interpretation process.

REFERENCES

• 1: Jean-Yves Girard, Linear logic, Theoretical Computer Science, v.50 n.1, p.1-102, Jan. 30, 1987 [doi>10.1016/0304-3975(87)90045-4]
• 2: Sitt Sen Chok , K. Marriott, Automatic construction of user interfaces from constraint multiset grammars, Proceedings of the 11th International IEEE Symposium on Visual Languages, p.242, September 05-09, 1995
• 3: Michael Johnston , Srinivas Bangalore, Finite-state multimodal integration and understanding, Natural Language Engineering, v.11 n.2, p.159-187, June 2005 [doi>10.1017/S1351324904003572]
• 4: Johnston, M. and S. Bangalore. 2005. "Combining Stochastic and Grammar-based Language Processing with Finite-state Edit Machines". In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop.
• 5: Joyce Yue Chai, Zahar Prasov, Shaolin Qu. 2006. Cognitive Principles in Robust Multimodal Interpretation. J. Artif. Intell. Res. (JAIR) 27: pp. 55--83.
• 6: Tsai, W. H., & Fu, K. S. 1979. Error-correcting isomorphism of attributed relational graphs for pattern analysis. IEEE Trans. Sys., Man and Cyb., 9, pp. 757--768.
• 7: Joyce Y. Chai , Pengyu Hong , Michelle X. Zhou, A probabilistic approach to reference resolution in multimodal user interfaces, Proceedings of the 9th international conference on Intelligent user interfaces, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964457]
• 8: Michael Collins, Three generative, lexicalised models for statistical parsing, Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, p.16-23, July 07-12, 1997, Madrid, Spain
• 9: Marsal Gavaldà , Alex Waibel, Growing semantic grammars, Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 10: Bob Carpenter, The logic of typed feature structures, Cambridge University Press, New York, NY, 1992
• 11: D. M. Berry, E. Kamsties, M. M. Krieger. 2003. From contract drafting to software specification: Linguistic sources of ambiguity. A Handbook. University of Waterloo, Waterloo, Ontario, Canada, 2003.
• 12: Kim Marriott , Bernd Meyer , Kent B. Wittenburg, A survey of visual language specification and recognition, Visual language theory, Springer-Verlag New York, Inc., New York, NY, 1998
• 13: D'Ulizia A, Ferri F., Grifoni P. 2007. A Hybrid Grammar-Based Approach to Multimodal Languages Specification, OTM 2007 Workshop Proceedings, 25--30 November 2007, Vilamoura, Portugal, Springer-Verlag, LNCS 4805. pp 367--376.
• 14: Martin J. C. (1997). Toward Intelligent Cooperation Between Modalities: The Example of a System Enabling Multimodal Interaction with a Map. Proceedings of International Joint Conference on Artificial Intelligence (IJCAI'97) Workshop on "Intelligent Multimodal Systems." Nagoya, Japan.
• 15: Caschera M. C., Ferri F., Grifoni P. 2007. An Approach for Managing 45 Ambiguities in Multimodal Interaction. OTM 2007 Ws, Part I, LNCS 4805. 45 Springer-Verlag Berlin Heidelberg 2007. pp. 387--397.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  F. Theory of Computation
  F.4 MATHEMATICAL LOGIC AND FORMAL LANGUAGES
      F.4.2 Grammars and Other Rewriting Systems
          Subjects: Grammar types (e.g., context-free, context-sensitive)

General Terms:
Design, Human Factors

Keywords:
grammar-based language, interpretation of multimodal input, multimodal ambiguity, multimodal interfaces