Using multimodal interaction to navigate in arbitrary virtual VRML worlds

ABSTRACT

In this paper we present a multimodal interface for navigating in arbitrary virtual VRML worlds. Conventional haptic devices like keyboard, mouse, joystick and touchscreen can freely be combined with special Virtual-Reality hardware like spacemouse, data glove and position tracker. As a key feature, the system additionally provides intuitive input by command and natural speech utterances as well as dynamic head and hand gestures. The commuication of the interface components is based on the abstract formalism of a context-free grammar, allowing the representation of device-independent information. Taking into account the current system context, user interactions are combined in a semantic unification process and mapped on a model of the viewer's functionality vocabulary. To integrate the continuous multimodal information stream we use a straight-forward rule-based approach and a new technique based on evolutionary algorithms. Our navigation interface has extensively been evaluated in usability studies, obtaining excellent results.

REFERENCES

• 1: F. Althoff, G. McGlaun, and M. Lang. Combining multiple input modalities for VR navigation - A user study. In 9.th Int. Conf. on HCI, August 2001.
• 2: F. Althoff, T. Volk, G. McGlaun, and M. Lang. A generic user interface framework for VR applications. In 9.th Int. Conf. on HCI, New Orleans, August 2001.
• 3: H. J. Bullinger. Virtual reality as a focal point between new media and telecomunication. VR World 1995 - Conference Documentation, IDG 1995.
• 4: A. Cheyer and L. Julia. Designing, developing and evaluating multimodal applications. In WS on Pen/Voice Interfaces (CHI 99), Pittsburgh 1999.
• 5: K.-H. Engelmeier et al. Virtual reality and multimedia human-computer interaction in medicine. IEEE WS on Multimedia Signal Processing, pages 88--97, Los Angeles, December 1998.
• 6: Lernout & Houspie Speech Products N.V. Lernout & Houspie - Software Developers Kit, 1998.
• 7: Details of the MIVIS system (October 2001). Internet-Publication, http://www.mivis.de.
• 8: Developer site of blaxxun interactive (July 2001). http://www.blaxxun.com/developer/contact/3d.
• 9: Specification of VRML 97. ISO/IEC 14772-1:1997, http://www.web3d.org/technicalinfo/Specifications/vrml97/index.html, July 2001.
• 10: M. Latoschik et al. Multimodale interaktion mit einem system zur virtuellen konstruktion. Informatik '99, 29. Jahrestagung der Gesellschaft fr Informatik, Paderborn, pages 88--97, October 1999.
• 11: Z. Michalewicz. Genetic Algoithms and Data Structures. Springer-Verlag, New York, 1999.
• 12: P. Morguet. Stochastische Modellierung von Bildsequenzen zur Segmentierung und Erkennung dynamischer Gesten. PhD thesis, Technical University of Munich, Germany, Januar, 2001.
• 13: P. Morguet et al. Comparison of approaches to continuous hand gesture recognition for a visual dialog system. Proc. of ICASSP 99, pages 3549--3552, 1999.
• 14: J. Müller and H. Stahl. Speech understanding and speech translation in various domains by maximum a-posteriori semantic decoding. In Proc. EIS 98, pages 256--267, La Laguna, Spain 1998.
• 15: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 16: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 17: S. L. Oviatt. Multimodal interface research: A science without borders. Proc. of 6th Int. Conference on Spoken Language Processing (ICSLP 2000), 2000.
• 18: V. Pavlovic, G. Berry, and T. Huang. BattleView: A multimodal hci research application. In Workshop on Perceptual User Interfaces (PUI 98), November 1998.
• 19: I. Poupyrev , N. Tomokazu , S. Weghorst, Virtual Notepad: Handwriting in Immersive VR, Proceedings of the Virtual Reality Annual International Symposium, p.126, March 14-18, 1998
• 20: B. Schuller, F. Althoff, G. McGlaun, and M. Lang. Navigating in virtual worlds via natural speech. In 9.th Int. Conf. on HCI, New Orleans, August 2001.
• 21: J. Stewart. FreeWRL homepage. Internet-Publication, http://www-ext.crc.ca/FreeWRL, June 2001.
• 22: Alex Waibel , Minh Tue Vo , Paul Duchnowski , Stefan Manke, Multimodal interfaces, Artificial Intelligence Review, v.10 n.3-4, p.299-319, Aug. 1996 [doi>10.1007/BF00127684]