Estimating focus of attention based on gaze and sound

ABSTRACT

Estimating a person's focus of attention is useful for various human-computer interaction applications, such as smart meeting rooms, where a user's goals and intent have to be monitored. In work presented here, we are interested in modeling focus of attention in a meeting situation. We have developed a system capable of estimating participants' focus of attention from multiple cues. We employ an omnidirectional camera to simultaneously track participants' faces around a meeting table and use neural networks to estimate their head poses. In addition, we use microphones to detect who is speaking. The system predicts participants' focus of attention from acoustic and visual information separately, and then combines the output of the audio- and video-based focus of attention predictors. We have evaluated the system using the data from three recorded meetings. The acoustic information has provided 8% error reduction on average compared to using a single modality.

REFERENCES

• 1: Gregory D. Abowd , Christopher G. Atkeson , Ami Feinstein , Cindy Hmelo , Rob Kooper , Sue Long , Nitin Sawhney , Mikiya Tani, Teaching and learning as multimedia authoring: the classroom 2000 project, Proceedings of the fourth ACM international conference on Multimedia, p.187-198, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244191]
• 2: M. Argyle and M. Cook. Gaze and Mutual Gaze. Cambridge University Press, 1976.
• 3: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 4: M. Black, F. Brard, A. Jepson, W. Newman, E. Saund, G. Socher, and M. Taylor. The digital office: Overview. In Proceedings of the 1998 AAAI Spring Symposium on Intelligent Environments, volume AAAI Technical Report SS-98-02. AAAI, AAAI Press, March 1998.
• 5: Patrick Chiu , Ashutosh Kapuskar , Sarah Reitmeier , Lynn Wilcox, Room with a Rear View: Meeting Capture in a Multimedia Conference Room, IEEE MultiMedia, v.7 n.4, p.48-54, October 2000 [doi>10.1109/93.895154]
• 6: A. J. Diebold. Animal Communication - Techniques of Study and Results of Research, chapter Anthropology of the comparative psychology of communicative behavior. Bloomington: Indiana University Press, 1968.
• 7: A. H. Gee and R. Cipolla. Non-intrusive gaze tracking for human-computer interaction. In Proc. Mechatronics and Machine Vision in Practise, pages 112--117, 1994.
• 8: D. Gopher. The Blackwell dictionary of Cognitive Psychology, chapter Attention, pages 23--28. Basil Blackwell Inc., 1990.
• 9: T. S. Jebara , A. Pentland, Parametrized structure from motion for 3D adaptive feedback tracking of faces, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.144, June 17-19, 1997
• 10: S. R. Langton, R. J. Watt, and V. Bruce. Do the eyes have it? cues to the direction of social attention. Trends in Cognitive Neuroscience, 4(2), 2000.
• 11: M. Mozer. The neural network house: An environment that adapts to its inhibitants. In Intelligent Environments, Papers from the 1998 AAAI Spring Symposium, number Technical Report SS-98-92, pages 110--114. AAAI, AAAI Press, 1998.
• 12: D. Perret and N. Emery. Understanding the intentions of others from visual signals: neurophysiological evidence. Cahiers de Psychologie Cognitive, 13:683-694, 1994.
• 13: R. Stiefelhagen , Jie Yang , A. Waibel, A model-based gaze tracking system, Proceedings of the 1996 IEEE International Joint Symposia on Intelligence and Systems, p.304, November 04-05, 1996
• 14: Rainer Stiefelhagen , Jie Yang , Alex Waibel, Modeling focus of attention for meeting indexing, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.3-10, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319464]
• 15: A. Waibel, M. Bett, M. Finke, and R. Stiefelhagen. Meeting browser: Tracking and summarizing meetings. In D. E. M. Penrose, editor, Proceedings of the Broadcast News Transcription and Understanding Workshop, pages 281--286, Lansdowne, Virginia, February. 8-11 1998. DARPA, Morgan Kaufmann.
• 16: Jie Yang , A. Waibel, A real-time face tracker, Proceedings of the 3rd IEEE Workshop on Applications of Computer Vision (WACV '96), p.142, December 02-04, 1996

INDEX TERMS

Keywords:
focus of attention, gaze tracking, intelligent environments, meeting analysis