Automatic facial expression recognition on a single 3D face by exploring shape deformation

ABSTRACT

Facial expression recognition has many applications in multimedia processing and the development of 3D data acquisition techniques makes it possible to identify expressions using 3D shape information. In this paper, we propose an automatic facial expression recognition approach based on a single 3D face. The shape of an expressional 3D face is approximated as the sum of two parts, a basic facial shape component (BFSC) and an expressional shape component (ESC). The BFSC represents the basic face structure and neutral-style shape and the ESC contains shape changes caused by facial expressions. To separate the BFSC and ESC, our method firstly builds a reference face for each input 3D non-neutral face by a learning method, which well represents the basic facial shape. Then, based on the BFSC and the original expressional face, a facial expression descriptor is designed. The surface depth changes are considered in the descriptor. Finally, the descriptor is input into an SVM to recognize the expression. Unlike previous methods which recognize a facial expression with the help of manually labeled key points and/or a neutral face, our method works on a single 3D face without any manual assistance. Extensive experiments are carried out on the BU-3DFE database and comparisons with existing methods are conducted. The experimental results show the effectiveness of our method.

REFERENCES

• 1: SHREC: Shape Retrieval Contest. http://www.aim-at-shape.net.
• 2: P. J. Besl and N. D. McKay. A Method for Registration of 3-D Shapes. IEEE Trans. on Pattern Analysis and Machine Intelligence, 14(2):239--256, 1992.
• 3: B. Fasel and J. Luettin. Automatic Facial Expression Analysis: A Survey. Pattern Recognition, 36(1):259--275, 2003.
• 4: K. Fukunaga. Introduction to Statistical Pattern Recognition. Academic Press, 1990.
• 5: A. Jaimes and N. Sebe. Multimodal Human-Computer Interaction: A Survey. Computer Vision and Image Understanding, 108(1-2):116--134, 2007.
• 6: C. Padgett and G. Cottrell. Representing Face Images for Emotion Classification. Advances in Neural Information Processing Systems, pages 894--900, 1997.
• 7: M. Pantic and L. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(12):1424--1445, 2000.
• 8: P. Phillips, P. Flynn, T. Scruggs, K. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek. Overview of the face recognition grand challenge. In IEEE Conf. on Computer Vision and Pattern Recognition, pages 947--954, 2005.
• 9: T. Russ, C. Boehnen, and T. Peters. 3D Face Recognition Using 3D Alignment for PCA. In IEEE Conf. on Computer Vision and Pattern Recognition, volume 2, 2006.
• 10: P. Sinha, B. Balas, Y. Ostrovsky, and R. Russell. Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About. Proceedings of the IEEE, 94(11):1948, 2006.
• 11: H. Soyel and H. Demirel. Facial Expression Recognition Using 3D Facial Feature Distances. In Int'l Conf. on Image Analysis and Recognition, pages 17--22, 2006.
• 12: H. Tanaka, M. Ikeda, and H. Chiaki. Curvature-Based Face Surface Recognition Using Spherical Correlation. Principal Directions for Curved Object Recognition. In IEEE Int'l Conf. on Automatic Face and Gesture Recognition, pages 372--377, 1998.
• 13: H. Tang and T. S. Huang. 3D Facial Expression Recognition Based on Automatically Selected Features. In IEEE Conf. on Computer Vision and Pattern Recognition Workshops, pages 1--8, 2008.
• 14: J. Wang, L. Yin, X. Wei, and Y. Sun. 3D Facial Expression Recognition Based on Primitive Surface Feature Distribution. In IEEE Conf. on Computer Vision and Pattern Recognition, pages 17--22, 2006.
• 15: Y. Wang, X. Tang, J. Liu, G. Pan, and R. Xiao. 3D Face Recognition by Local Shape Difference Boosting. In European Conference on Computer Vision, 2008.
• 16: L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato. A 3D Facial Expression Database for Facial Behavior Research. In Int'l Conf. on Automatic Face and Gesture Recognition, 2006.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Size and shape

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

General Terms:
Algorithms, Design, Experimentation

Keywords:
3D facial expression recognition, basic facial shape component, expressional shape component, shape deformation