A multimodal learning interface for grounding spoken language in sensory perceptions

ABSTRACT

Most speech interfaces are based on natural language processing techniques that use pre-defined symbolic representations of word meanings and process only linguistic information. To understand and use language like their human counterparts in multimodal human-computer interaction, computers need to acquire spoken language and map it to other sensory perceptions. This paper presents a multimodal interface that learns to associate spoken language with perceptual features by being situated in users' everyday environments and sharing user-centric multisensory information. The learning interface is trained in unsupervised mode in which users perform everyday tasks while providing natural language descriptions of their behaviors. We collect acoustic signals in concert with multisensory information from non-speech modalities, such as user's perspective video, gaze positions, head directions and hand movements. The system firstly estimates users' focus of attention from eye and head cues. Attention, as represented by gaze fixation, is used for spotting the target object of user interest. Attention switches are calculated and used to segment an action sequence into action units which are then categorized by mixture hidden Markov models. A multimodal learning algorithm is developed to spot words from continuous speech and then associate them with perceptually grounded meanings extracted from visual perception and action. Successful learning has been demonstrated in the experiments of three natural tasks: "unscrewing a jar", "stapling a letter" and "pouring water".

REFERENCES

• 1: R. Adams , L. Bischof, Seeded Region Growing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.16 n.6, p.641-647, June 1994 [doi>10.1109/34.295913]
• 2: Charu C. Aggarwal , Philip S. Yu, Finding generalized projected clusters in high dimensional spaces, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.70-81, May 15-18, 2000, Dallas, Texas, United States
• 3: D. H. Ballard and C. Yu. A multimodal learning interface for word acquisition. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Hong Kong, April 2003.
• 4: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 5: L. W. Campbell , D. A. Becker , A. Azarbayejani , A. F. Bobick , A. Pentland, Invariant features for 3-D gesture recognition, Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG '96), p.157, October 14-16, 1996
• 6: John A. Hartigan, Clustering Algorithms, John Wiley & Sons, Inc., New York, NY, 1975
• 7: M. Hayhoe. Visual routines: A functional account of vision. Visual Cognition, 7:43--64, 2000.
• 8: M. Land, N. Mennie, and J. Rusted. The roles of vision and eye movements in the control of activities of daily living. Perception, 28:1311--1328, 1999.
• 9: Richard P. Lippmann, Review of neural networks for speech recognition, Neural Computation, v.1 n.1, p.1-38, Spring 1990 [doi>10.1162/neco.1989.1.1.1]
• 10: Bartlett W. Mel, SEEMORE: combining color, shape, and texture histogramming in a neurally inspired approach to visual object recognition, Neural Computation, v.9 n.4, p.777-804, May 15, 1997 [doi>10.1162/neco.1997.9.4.777]
• 11: T. Oates, L. Firoiu, and P. R. Cohen. Clustering time series with hidden Markov models and dynamic time warping. In Proceedings of the IJCAI-99 Workshop on Neural, Symbolic and Reinforcement Learning Methods for Sequence Learning, pages 17--21, 1999.
• 12: S. Oviatt. Multimodal interfaces. In J. Jacko and A. Sears, editors, Handbook of Human-Computer Interaction. Lawrence Erlbaum, New Jersey, 2002.
• 13: L. R. Rabiner and B. Juang. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257--286, 1989.
• 14: T. Robinson. An application of recurrent nets to phone probability estimation. IEEE Transactions on Neural Networks, 5(2):298--305, 1994.
• 15: D. Roy and A. Pentland. Learning words from sights and sounds: A computational model. Cognitive Science, 26(1):113--146, 2002.
• 16: D. D. Salvucci and J. Anderson. Tracking eye movement protocols with cognitive process models. In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society, pages 923--928, LEA: Mahwah, NJ, 1998.
• 17: Bernt Schiele , James L. Crowley, Recognition without Correspondence using MultidimensionalReceptive Field Histograms, International Journal of Computer Vision, v.36 n.1, p.31-50, Jan. 2000 [doi>10.1023/A:1008120406972]
• 18: J. M. Siskind. Grounding language in perception. artificial Intelligence Review, 8:371--391, 1995.
• 19: P. Smyth. Clustering sequences with hidden markov models. In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, page 648. The MIT Press, 1997.
• 20: Michael J. Swain , Dana H. Ballard, Color indexing, International Journal of Computer Vision, v.7 n.1, p.11-32, Nov. 1991 [doi>10.1007/BF00130487]
• 21: S. Wang and J. M. Siskind. Image segmentation with ratio cut. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003.
• 22: Learning to Recognize Human Action Sequences, Proceedings of the 2nd International Conference on Development and Learning, p.28, June 12-15, 2002

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Language acquisition

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.0 General
          Subjects: Cognitive simulation

General Terms:
Human Factors, Languages

Keywords:
language acquisition, machine learning, multimodal integration