A perceptual user interface for recognizing head gesture acknowledgements

ABSTRACT

We present the design and implementation of a perceptual user interface for a responsive dialog-box agent that employs real-time computer vision to recognize user acknowledgements from head gestures (e.g., nod = yes). IBM Pupil-Cam technology together with anthropometric head and face measures are used to first detect the location of the user's face. Salient facial features are then identi ed and tracked to compute the global 2-D motion direction of the head. For recognition, timings of natural gesture motion are incorporated into a state-space model. The interface is presented in the context of an enhanced text editor employing a perceptual dialog-box agent.

REFERENCES

• 1: H. Anton. Calculus with Analytic Geometry. John Wiley & Sons, New York, 1988.
• 2: A. Azarbayejani , T. Starner , B. Horowitz , A. Pentland, Visually Controlled Graphics, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.6, p.602-605, June 1993 [doi>10.1109/34.216730]
• 3: R. Birdwhistell. Kinesics and context: Essays on Body Motion Communication. Univ. Penn. Press, Philadelphia, PA, 1970.
• 4: A. Bobick, J. Davis, and S. Intille. The KidsRoom: an example application using a deep perceptual interface. In Proc. Wkshp. Percept. User Interfaces, pages 1--4, 1997.
• 5: A. Bobick, S. Intille, J. Davis, F. Baird, L. Campbell, Y. Ivanov, C. Pinhanez, A. Schutte, and A. Wilson. The KidsRoom: action recognition in an interactive story environment. Presence: Teleoperators and Virtual Environments, 8(4):367--391, 1999.
• 6: Gary R. Bradski, Real Time Face and Object Tracking as a Component of a Perceptual User Interface, Proceedings of the 4th IEEE Workshop on Applications of Computer Vision (WACV'98), p.214, October 19-21, 1998
• 7: J. Cooper , S. Venkatesh , L. Kitchen, Early Jump-Out Corner Detectors, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.8, p.823-828, August 1993 [doi>10.1109/34.236246]
• 8: James L. Crowley , Francois Berard, Multi-Modal Tracking of Faces for Video Communications, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.640, June 17-19, 1997
• 9: T. Darrell, G. Gordon, M. Marville, and J. Woodfill. Robust visual person tracking for interactive displays. In Proc. Wkshp. Percept. User Interfaces, pages 91--94, 1998.
• 10: J. Davis and A. Bobick. Virtual PAT: a virtual personal aerobics trainer. In Proc. Wkshp. Percept. User Interfaces, pages 13--18, 1998.
• 11: L. Farkas, editor. Anthropometry of the Head and Face. Raven Press, New York, 1994.
• 12: S. Gokturk, J. Bouguet, and R. Grzeszczuk. A data-driven model for monocular face tracking. In Proc. Int. Conf. Comp. Vis., pages 701--708. IEEE, 2001.
• 13: Intel Open Source Computer Vision Library, www.intel.com/research/mrl/research/opencv.
• 14: L. Kitchen and A. Rosenfeld. Gray level corner detection. Patt. Recogn. Lett., 1:95--102, 1982.
• 15: D. Lyons, D. Pelletier, and D. Knapp. Multimodal interactive advertising. In Proc. Wkshp. Percept. User Interfaces, pages 83--86, 1998.
• 16: Mimic. Lipsinc, Inc.
• 17: H. Moravec. Towards automatic visual obstacle avoidance. In Proc. Int. Joint Conf. Artif. Intell., 1977.
• 18: C. Morimoto, D. Koons, A. Amir, and M. Flickner. Real-time detection of eyes and faces. In Proc. Wkshp. Percept. User Interfaces, pages 117--120, 1998.
• 19: C. Morimoto, D. Koons, A. Amir, and M. Flickner. Pupil detection and tracking using multiple light sources. Image and Vision Comp., 18(4):331--335, 2000.
• 20: D. Morris, P. Collett, and P. Marsh. Gestures: Their Origins and Distribution. Stein and Day, New York, 1979.
• 21: J. Alison Noble, Finding corners, Image and Vision Computing, v.6 n.2, p.121-128, May 1988 [doi>10.1016/0262-8856(88)90007-8]
• 22: Poser. Curious Labs, Inc.
• 23: J. M. Rehg , M. Loughlin , K. Waters, Vision for a smart kiosk, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.690, June 17-19, 1997
• 24: Henry A. Rowley , Shumeet Baluja , Takeo Kanade, Neural Network-Based Face Detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.1, p.23-38, January 1998 [doi>10.1109/34.655647]
• 25: L. Sproull, M. Subramani, S. Kiesler, J. Walker, and K. Waters. When the interface is a face. Human-Computer Interaction, 11:97--124, 1996.
• 26: Kah-Kay Sung , Tomaso Poggio, Example-Based Learning for View-Based Human Face Detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.1, p.39-51, January 1998 [doi>10.1109/34.655648]
• 27: K. Toyama. Look, ma -- no hands! Hands free cursor control with real-time 3D face tracking. In Proc. Wkshp. Percept. User Interfaces, pages 49--54, 1998.
• 28: Matthew Turk , George Robertson, Perceptual user interfaces (introduction), Communications of the ACM, v.43 n.3, p.32-34, March 2000 [doi>10.1145/330534.330535]
• 29: Haiyuan Wu , Tadayoshi Shioyama , Hirotomo Kobayashi, Spotting Recognition of Head Gestures from Color Image Series, Proceedings of the 14th International Conference on Pattern Recognition-Volume 1, p.83, August 16-20, 1998