Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

We present MultiML, a markup language for the annotation of multimodal human utterances. MultiML is able to represent input from several modalities, as well as the relationships between these modalities. Since MultiML separates general parts of representation from more context-specific aspects, it can easily be adapted for use in a wide range of contexts. This paper demonstrates how speech and gestures are described with MultiML, showing the principles - including hierarchy and underspecification - that ensure the quality and extensibility of MultiML. As a proof of concept, we show how MultiML is used to annotate a sample human-robot interaction in the domain of a multimodal joint-action scenario.

REFERENCES

• 1: A. Kranstedt, S. Kopp, and I. Wachsmuth "Murml: A multimodal utterance representation markup language for conversational agents," in Proc. of the AAMAS Workshop on "Embodied conversational agents Let's specify and evaluate them", 2002.
• 2: S. Prillwitz, R. Leven, H. Zienert, T. Hanke, and J. Henning, HamNoSys. Version 2.0; Hamburger Notationssystem für Gebärdensprache. Eine Einführung. Hamburg: Signum, 1989.
• 3: F. Landragin, A. Denis, A. Ricci, and L. Romary, "Multimodal meaning representation for generic dialogue systems architectures," in Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004), 2004, pp. 521--524.
• 4: D. Gibbon, U. Gut, B. Hell, K. Looks, and A. T. T. Trippel, "A computational model of arm gestures in conversation," in EUROSPEECH-2003, 2003, pp. 813--816.
• 5: Joyce Chai, Semantics-based representation for multimodal interpretation in conversational systems, Proceedings of the 19th international conference on Computational linguistics, p.1-7, August 24-September 01, 2002, Taipei, Taiwan [doi>10.3115/1072228.1072263]
• 6: Mary Ellen Foster , Tomas By , Markus Rickert , Alois Knoll, Human-Robot dialogue for joint construction tasks, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181009]
• 7: T. Müller, P. Ziaie, and A. Knoll, "A wait-free realtime system for optimal distribution of vision tasks on multicore architectures," in Proc. 5th International Conference on Informatics in Control, Automation and Robotics, May 2008.
• 8: P. Ziaie, T. Müller, M. E. Foster, and A. Knoll, "Using a naïve Bayes classifier based on k-nearest neighbors with distance weighting for static hand-gesture recognition in a human-robot dialog system," in Proceedings of CSICC 2008, Kish Island, Iran, Mar. 2008.
• 9: Michi Henning, A New Approach to Object-Oriented Middleware, IEEE Internet Computing, v.8 n.1, p.66-75, January 2004 [doi>10.1109/MIC.2004.1260706]
• 10: M. Giuliani and A. Knoll, "Integrating multimodal cues using grammar based models," in Proceedings of HCI International 2007, Beijing, China, July 2007, pp. 858--867.
• 11: M. Rickert, M. E. Foster, M. Giuliani, T. By, G. Panin, and A. Knoll, "Integrating language, vision and action for human robot dialog systems," in Proceedings of the International Conference on Human-Computer Interaction, C. Stephanidis, Ed. Beijing: Springer, July 2007, pp. 987--995.
• 12: J. F. Allen and G. Ferguson, "Actions and events in interval temporal logic," Journal of Logic and Computation, vol. 4, pp. 531--579, 1994.
• 13: A. E. Ades and M. J. Steedman, "On the order of words," Linguistics and philosophy, vol. 4, pp. 517--558, 1982.
• 14: Mark Steedman, The syntactic process, MIT Press, Cambridge, MA, 2000
• 15: K. Ajdukiewicz, "Die syntaktische Konnexität," Studia Philosophica, vol. 1, pp. 1--27, 1935.
• 16: Y. Bar-Hillel, "A quasi-arithmetic notation for syntactic description," Language, vol. 29, pp. 47--58, 1953.
• 17: M. White, "Effcient realization of coordinate structures in combinatory categorial grammar," Research on Language & Computation, vol. 4, no. 1, pp. 39--75, 2006.
• 18: Jason Baldridge , Geert-Jan M. Kruijff, Coupling CCG and hybrid logic dependency semantics, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073137]
• 19: D. McNeill, Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press, 1992.
• 20: A. Kendon, Gesture: Visible Action as Utterance. Cambridge University Press, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.0 General

General Terms:
Languages, Standardization

Keywords:
human-robot interaction, multimodal, representation