The virtual human as a multimodal interface

ABSTRACT

This paper discusses the main issues for creating Interactive Virtual Environments with Virtual Humans emphasizing the following aspects: creation of Virtual Humans, gestures, interaction with objects, multimodal communication.

REFERENCES

• 1: Won-Sook Lee , Nadia Magnenat-Thalmann, Head Modeling from Pictures and Morphing in 3D with Image Metamorphosis Based on Triangulation, Proceedings of the International Workshop on Modelling and Motion Capture Techniques for Virtual Environments, p.254-267, January 1998
• 2: P. Fua , R. Plaenkers , D. Thalmann, From Synthesis to Analysis: Fitting Human Animation Models to Image Data, Proceedings of the International Conference on Computer Graphics, p.4, June 07-11, 1999
• 3: Pascal Fua, Face Models from Uncalibrated Video Sequences, Proceedings of the International Workshop on Modelling and Motion Capture Techniques for Virtual Environments, p.214-228, January 1998
• 4: R. Boulic et. al. Goal Oriented Design and Correction of Articulated Figure Motionwith the TRACK system, Computer and Graphics, Pergamon Press, Vol. 18, No. 4., pp. 443-452, 1994.
• 5: Tom Molet , Ronan Boulic , Daniel Thalmann, A real time anatomical converter for human motion capture, Proceedings of the Eurographics workshop on Computer animation and simulation '96, p.79-94, December 1996, Poitiers, France
• 6: P. Kalra, A. Mangili, N. Magnenat Thalmann, D. Thalmann (1992) Simulation of Facial Muscle Actions Based on Rational Free Form Deformation, Proc. Eurographics '92, pp. 59-69.
• 7: Norman Badler, Virtual Humans for Animation, Ergonomics, and Simulation, Proceedings of the 1997 IEEE Workshop on Motion of Non-Rigid and Articulated Objects (NAM '97), p.28, June 16-July 16, 1997
• 8: W. Lewis Johnson , Jeff Rickel, Steve: an animated pedagogical agent for procedural training in virtual environments, ACM SIGART Bulletin, v.8 n.1-4, p.16-21, 1997 [doi>10.1145/272874.272877]
• 9: Libby Levison, Connecting planning and acting via object-specific reasoning, University of Pennsylvania, Philadelphia, PA, 1996
• 10: M. Kallmann, D. Thalmann, Modeling Objects for Interaction Tasks, Proc. Eurographics Workshop on Animation and Simulation, Springer, 1998
• 11: Marcelo Kallmann , Daniel Thalmann, A Behavioral Interface to Simulate Agent-Object Interactions in Real Time, Proceedings of the Computer Animation, p.138, May 26-28, 1999
• 12: Jami J. Shah , Jamie J. Shah , Martti Mantyla, Parametric and Feature Based CAD/Cam: Concepts, Techniques, and Applications, John Wiley & Sons, Inc., New York, NY, 1995
• 13: R. Boulic, S. Rezzonico, D. Thalmann, 1996. Multi- Finger Manipulation of Virtual Objects, Proc. ACM Symposium on Virtual Reality Software and Technology VRST'96, Hong-Kong, 1996, pp 67-74.
• 14: Serge Rezzonico , Ronan Boulic , Zhiyong Huang , Nadia Magnenat Thalmann , Daniel Thalmann, Consistent grasping in virtual environments based on the interactive grasping automata, Selected papers of the Eurographics workshops on Virtual environments '95, p.107-118, December 1995, Barcelona, Spain
• 15: Luc Emering , Ronan Boulic , Daniel Thalmann, Interacting with Virtual Humans through Body Actions, IEEE Computer Graphics and Applications, v.18 n.1, p.8-11, January 1998 [doi>10.1109/38.637261]
• 16: C. Ciressan, D.Thalmann, 3D Model-Based Upper-Body Tracking Using Monocular Camera, internal report, Computer Graphics Lab, EPFL.
• 17: Ernst Dieter Dickmanns , Volker Graefe, Applications of dynamic monocular machine vision, Machine Vision and Applications, v.1 n.4, p.241-261, 1988 [doi>10.1007/BF01212362]
• 18: M. Argyle, Bodily Communication, New York: Methuen & Co., I988.
• 19: A. Black, P. Taylor (1997) Festival Speech Synthesis System: System Documentation (1.1.1), Technical Report HCRC/TR-83, Human Communication Research Center, University of Edinburgh.
• 20: Hansrudi Noser , Daniel Thalmann, A Rule-Based Interactive Behavioral Animation System for Humanoids, IEEE Transactions on Visualization and Computer Graphics, v.5 n.4, p.281-307, October 1999 [doi>10.1109/2945.817347]
• 21: Hennebert J. and Delacr6taz D.P., (1996) POST: Parallel Object-Oriented Speech Toolkit, Proc. ICSLP 96, Philadelphia
• 22: G. Sannier, S. Balcisoy, N. Magnenat-Thalmann, D. Thalmann, VHD: A System for Directing Real-Time Virtual Actors, The Visual Computer, Springer, Vol.15, No 7/8, i999, pp.320-329.
• 23: Ronan Boulic , Pascal Bécheiraz , Luc Emering , Daniel Thalmann, Integration of motion control techniques for virtual human and avatar real-time animation, Proceedings of the ACM symposium on Virtual reality software and technology, p.111-118, September 1997, Lausanne, Switzerland [doi>10.1145/261135.261156]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Motion

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking

General Terms:
Design, Human Factors, Languages, Management, Performance, Theory

Keywords:
action recognition, gestures, multimodal communication, virtual humans