Semantic 3D motion retargeting for facial animation

ABSTRACT

We present a system for realistic facial animation that decomposes facial motion capture data into semantically meaningful motion channels based on the Facial Action Coding System. A captured performance is retargeted onto a morphable 3D face model based on a semantic correspondence between motion capture and 3D scan data. The resulting facial animation reveals a high level of realism by combining the high spatial resolution of a 3D scanner with the high temporal accuracy of motion capture data that accounts for subtle facial movements with sparse measurements.Such an animation system allows us to systematically investigate human perception of moving faces. It offers control over many aspects of the appearance of a dynamic face, while utilizing as much measured data as possible to avoid artistic biases. Using our animation system, we report results of an experiment that investigates the perceived naturalness of facial motion in a preference task. For expressions with small amounts of head motion, we find a benefit for our part-based generative animation system over an example-based approach that deforms the whole face at once.

REFERENCES

• 1: K. S. Arun , T. S. Huang , S. D. Blostein, Least-squares fitting of two 3-D point sets, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.9 n.5, p.698-700, Sept. 1987 [doi>10.1109/TPAMI.1987.4767965]
• 2: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 3: Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Comput. Graph. Forum 22, 3, 641--650.
• 4: Brainard, D. H. 1997. The psychophysics toolbox. Spatial Vision 10, 433--436.
• 5: Breidt, M., Wallraven, C., Cunningham, D. W., and Bülthoff, H. H. 2003. Facial animation based on 3d scans and motion capture. In SIGGRAPH '03 Sketches & Applications, ACM Press, New York, NY, USA, N. Campbell, Ed.
• 6: Choe, B., and Ko, H.-S. 2001. Analysis and synthesis of facial expressions with hand-generated muscle actuation basis. In Proceedings of Computer Animation, 12--19.
• 7: Erika Chuang , Christoph Bregler, Mood swings: expressive speech animation, ACM Transactions on Graphics (TOG), v.24 n.2, p.331-347, April 2005 [doi>10.1145/1061347.1061355]
• 8: Douglas W. Cunningham , Mario Kleiner , Heirich H. Bülthoff , Christian Wallraven, The components of conversational facial expressions, Proceedings of the 1st Symposium on Applied perception in graphics and visualization, August 07-08, 2004, Los Angeles, California [doi>10.1145/1012551.1012578]
• 9: Peter Eisert , Bernd Girod, Analyzing Facial Expressions for Virtual Conferencing, IEEE Computer Graphics and Applications, v.18 n.5, p.70-78, September 1998 [doi>10.1109/38.708562]
• 10: Ekman, P., and Friesen, W. V. 1978. Facial Action Coding System: A Technique for the Measurements of Facial Movement. Palo Alto, CA, USA.
• 11: Essa, I., and Pentland, A. 1994. A vision system for observing and extracting facial action parameters. In Computer Vision and Pattern Recognition Conference, 76--83.
• 12: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 13: Geiger, G., Ezzat, T., and Poggio, T. 2003. CBCL paper #224/AI memo #2003-003. Tech. rep., Massachusetts Institute of Technology, Cambridge, MA, USA.
• 14: Brian Guenter , Cindy Grimm , Daniel Wood , Henrique Malvar , Fredric Pighin, Making faces, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.55-66, July 1998 [doi>10.1145/280814.280822]
• 15: Hill, H., and Johnston, A. 2001. Categorizing sex and identity from the biological motion of faces. Current Biology 11, 11, 880--885.
• 16: Pushkar Joshi , Wen C. Tien , Mathieu Desbrun , Frédéric Pighin, Learning controls for blend shape based realistic facial animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 17: Kalberer, G. A., and Gool, L. V. 2002. Realistic face animation for speech. Journal of Visualization and Computer Animation 13, 97--106.
• 18: Mario Kleiner , Adrian Schwaninger , Douglas Cunningham , Barbara Knappmeyer, Using facial texture manipulation to study facial motion perception, Proceedings of the 1st Symposium on Applied perception in graphics and visualization, August 07-08, 2004, Los Angeles, California [doi>10.1145/1012551.1012602]
• 19: Knappmeyer, B., Thornton, I., and Bülthoff, H. H. 2001. Facial motion can determine facial identity. J. Vis. 1, 3 (12), 338a.
• 20: Knappmeyer, B., Giese, M., and Bülthoff, H. H. 2003. Spatio-temporal caricature effects for facial motion. J. Vis. 3, 9 (10), 304.
• 21: J. J. Lien , T. Kanade , J. F. Cohn , C. -C. Li, Automated Facial Expression Recognition Based on FACS Action Units, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.390, April 14-16, 1998
• 22: Parke, F. I. 1972. Computer generated animation of faces. PhD thesis, University of Utah, Salt Lake City.
• 23: Frédéric Pighin , Richard Szeliski , David H. Salesin, Modeling and Animating Realistic Faces from Images, International Journal of Computer Vision, v.50 n.2, p.143-169, November 2002 [doi>10.1023/A:1020393915769]
• 24: Eftychios Sifakis , Igor Neverov , Ronald Fedkiw, Automatic determination of facial muscle activations from sparse motion capture marker data, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 25: Ying-li Tian , Takeo Kanade , Jeffrey F. Cohn, Recognizing Lower Face Action Units for Facial Expression Analysis, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.484, March 26-30, 2000
• 26: Troje, N. F., and Bülthoff, H. H. 1996. Face recognition under varying poses: The role of texture and shape. Vision Research 36, 1761--1771.
• 27: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 28: Christian Wallraven , Martin Breidt , Douglas W. Cunningham , Heinrich H. Bülthoff, Psychophysical evaluation of animated facial expressions, Proceedings of the 2nd symposium on Applied perception in graphics and visualization, August 26-28, 2005, A Coroña, Spain [doi>10.1145/1080402.1080405]
• 29: Lance Williams, Performance-driven facial animation, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.235-242, September 1990, Dallas, TX, USA

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing

Keywords:
facial animation, facial motion, human perception, motion retargeting, performance capture