A user interface framework for multimodal VR interactions

ABSTRACT

This article presents a User Interface (UI) framework for multimodal interactions targeted at immersive virtual environments. Its configurable input and gesture processing components provide an advanced behavior graph capable of routing continuous data streams asynchronously. The framework introduces a Knowledge Representation Layer which augments objects of the simulated environment with Semantic Entities as a central object model that bridges and interfaces Virtual Reality (VR) and Artificial Intelligence (AI) representations. Specialized node types use these facilities to implement required processing tasks like gesture detection, preprocessing of the visual scene for multimodal integration, or translation of movements into multimodally initialized gestural interactions. A modified Augmented Transition Nettwork (ATN) approach accesses the knowledge layer as well as the preprocessing components to integrate linguistic, gestural, and context information in parallel. The overall framework emphasizes extensibility, adaptivity and reusability, e.g., by utilizing persistent and interchangeable XML-based formats to describe its processing stages.

REFERENCES

• 1: Frank Althoff , Gregor McGlaun , Björn Schuller , Peter Morguet , Manfred Lang, Using multimodal interaction to navigate in arbitrary virtual VRML worlds, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971494]
• 2: George N. Phillips Jr., Modular Approach of Multimodal Integration in a Virtual Environment, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.331, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167017]
• 3: K. Böhm, W. Hübner, and K. Väänänen. Given: Gesture driven interactions in virtual environments; a toolkit approach to 3D interactions. In Interfaces to Real and Virtual Worlds, 1992.
• 4: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 5: R. Carey, G. Bell, and C. Marrin. ISO/IEC 14772-1:1997 virtual reality modeling language (VRML). Technical report, The VRML Consortium Incorporated, 1997.
• 6: M. Cavazza, X. Pouteau, and D. Pernel. Multimodal communication in virtual environments. In Symbiosis of Human and Artifact, pages 597--604. Elsevier Science B. V., 1995.
• 7: Philip Cohen , David McGee , Sharon Oviatt , Lizhong Wu , Joshua Clow , Robert King , Simon Julier , Lawrence Rosenblum, Multimodal Interaction for 2D and 3D Environments, IEEE Computer Graphics and Applications, v.19 n.4, p.10-13, July 1999 [doi>10.1109/38.773958]
• 8: Alexander G. Hauptmann , Paul McAvinney, Gestures with speech for graphic manipulation, International Journal of Man-Machine Studies, v.38 n.2, p.231-249, Feb. 1993 [doi>10.1006/imms.1993.1011]
• 9: Guido Heumer , Malte Schilling, Automatic Data Exchange and Synchronization for Knowledge-Based Intelligent Virtual Environments, Proceedings of the 2005 IEEE Conference 2005 on Virtual Reality, p.43-50, March 12-16, 2005 [doi>10.1109/VR.2005.10]
• 10: Michael Johnston, Unification-based multimodal parsing, Proceedings of the 36th annual meeting on Association for Computational Linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 11: M. Johnston and S. Bangalore. Finite-state methods for multimodal parsing and integration. In Finite-state Methods Workshop, ESSLLI Summer School on Logic Language and Information,Helsinki, Finland, august 2001.
• 12: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 13: Ed Kaiser , Alex Olwal , David McGee , Hrvoje Benko , Andrea Corradini , Xiaoguang Li , Phil Cohen , Steven Feiner, Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958438]
• 14: David B. Koons , Carlton J. Sparrell , Kristinn R. Thorisson, Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 15: Frédéric Landragin, Referring to Objects with Spoken and Haptic Modalities, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.99, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166976]
• 16: Marc Erich Latoschik, A gesture processing framework for multimodal interaction in virtual reality, Proceedings of the 1st international conference on Computer graphics, virtual reality and visualisation, November 05-07, 2001, Camps Bay, Cape Town, South Africa [doi>10.1145/513867.513888]
• 17: Designing Transition Networks for Multimodal VR-Interactions Using a Markup Language, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.411, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167030]
• 18: M. E. Latoschik and M. Schilling. Incorporating VR Databases into AI Knowledge Representations: A Framework for Intelligent Graphics Applications. In Proceedings of the Sixth International Conference on Computer Graphics and Imaging. IASTED, ACTA Press, 2003.
• 19: B. Lenzmann. Benutzeradaptive und multimodale Interface-Agenten. PhD thesis, Technische Fakultät, Universität Bielefeld, 1998.
• 20: M. Lucente, G.-J. Zwart, and A. D. George. Visualization space: A testbed for deviceless multimodal user interface. In Intelligent Environments Symposium, American Assoc. for Artificial Intelligence Spring Symposium Series, Mar. 1998.
• 21: Mark T. Maybury, Research in multimedia and multimodal parsing and generation, Artificial Intelligence Review, v.9 n.2-3, p.103-127, June 1995 [doi>10.1007/BF00849175]
• 22: Jeannette G. Neal , Stuart C. Shapiro, Intelligent multi-media interface technology, Intelligent user interfaces, ACM Press, New York, NY, 1991 [doi>10.1145/107215.128690]
• 23: Sharon Oviatt, Multimodal interfaces, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 24: Thies Pfeiffer , Marc Erich Latoschik, Resolving Object References in Multimodal Dialogues for Immersive Virtual Environments, Proceedings of the IEEE Virtual Reality 2004 (VR'04), p.35, March 27-31, 2004 [doi>10.1109/VR.2004.67]
• 25: C. J. Sparrell and D. B. Koons. Interpretation of coverbal depictive gestures. In AAAI Spring Symposium Series, pages 8--12. Stanford University, March 1994.
• 26: Paul S. Strauss , Rikk Carey, An object-oriented 3D graphics toolkit, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.341-349, July 1992
• 27: Daniel Thalmann, The virtual human as a multimodal interface, Proceedings of the working conference on Advanced visual interfaces, p.14-20, May 2000, Palermo, Italy [doi>10.1145/345513.345232]
• 28: Damien Touraine , Patrick Bourdot , Yacine Bellik , Laurence Bolot, A framework to manage multimodal fusion of events for advanced interactions within virtual environments, Proceedings of the workshop on Virtual environments 2002, May 30-31, 2002, Barcelona, Spain
• 29: Henrik Tramberend, Avocado: A Distributed Virtual Reality Framework, Proceedings of the IEEE Virtual Reality, p.14, March 13-17, 1999
• 30: M. Vo and C. Wood. Building an application framework for speech and pen input integration in multimodal learning interfaces. In Proceedings of International Conference on Acoustics, Speech, and Signal Processing, 1996.
• 31: E. V. Zudilova, A Multi-Modal Interface for an Interactive Simulated Vascular Reconstruction System, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.313, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167013]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.1 Applications and Expert Systems
          Subjects: Natural language interfaces
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality

General Terms:
Algorithms, Design

Keywords:
gesture and speech processing, multimodal interaction, semantic scene description, user interface framework, virtual reality