Elvis: situated speech and gesture understanding for a robotic chandelier

ABSTRACT

We describe a home lighting robot that uses directional spotlights to create complex lighting scenes. The robot senses its visual environment using a panoramic camera and attempts to maintain its target goal state by adjusting the positions and intensities of its lights. Users can communicate desired changes in the lighting environment through speech and gesture (e.g., "Make it brighter over there"). Information obtained from these two modalities are combined to form a goal, a desired change in the lighting of the scene. This goal is then incorporated into the system's target goal state. When the target goal state and the world are out of alignment, the system formulates a sensorimotor plan that acts on the world to return the system to homeostasis.

REFERENCES

• 1: P. Lamere, P. Kwok, et al. Design of the CMU Sphinx-4 Decoder. Eurospeech, September 2003.
• 2: M. Kuperstein. Neural model of adaptive hand-eye coordination for single postures. Science, 239. 1308--1311, 1988.
• 3: M. Petrou, L. Shafarenko, and J. Kittler. Histogram-based segmentation in a perceptually uniform color space. IEEE Transactions on Image Processing, vol. 7, pp. 1354--1358, September 1998.
• 4: A. Kendon. Conducting Interaction. Cambridge: Cambridge University Press 1990.
• 5: D. McNeill. Hand and Mind. The University of Chicago Press, Chicago, 1992.
• 6: Rajeev Sharma , Jiongyu Cai , Srivat Chakravarthy , Indrajit Poddar , Yogesh Sethi, Exploiting Speech/Gesture Co-occurrence for Improving Continuous Gesture Recognition in Weather Narration, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.422, March 26-30, 2000
• 7: S. Kettebekov and R. Sharma. Understanding Gestures in Multimodal Human-Computer Interaction. International Journal on Artificial Intelligence Tools, vol. 9, no. 2, pp. 205--224, June 2000.
• 8: P. R. Cohen , M. Dalrymple , D. B. Moran , F. C. Pereira , J. W. Sullivan, Synergistic use of direct manipulation and natural language, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.227-233, March 1989
• 9: Sotaro Kita , Ingeborg van Gijn , Harry van der Hulst, Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders, Proceedings of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction, p.23-35, September 17-19, 1997
• 10: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen); Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Design, Human Factors

Keywords:
gesture, grounded, input methods, lighting, multimodal, natural interaction, situated, speech