eFASE: expressive facial animation synthesis and editing with phoneme-isomap controls

ABSTRACT

This paper presents a novel data-driven system for expressive facial animation synthesis and editing. Given novel phoneme-aligned speech input and its emotion modifiers (specifications), this system automatically generates expressive facial animation by concatenating captured motion data while animators establish constraints and goals. A constrained dynamic programming algorithm is used to search for best-matched captured motion nodes by minimizing a cost function. Users optionally specify "hard constraints" (motion-node constraints for expressing phoneme utterances) and "soft constraints" (emotion modifiers) to guide the search process. Users can also edit the processed facial motion node database by inserting and deleting motion nodes via a novel phoneme-Isomap interface. Novel facial animation synthesis experiments and objective trajectory comparisons between synthesized facial motion and captured motion demonstrate that this system is effective for producing realistic expressive facial animations.

REFERENCES

• 1: {BBPV03} Blanz V., Basso C., Poggio T., Vetter T.: Reanimating faces in images and video. Computer Graphics Forum 22, 3 (2003).
• 2: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 3: Carlos Busso , Zhigang Deng , Ulrich Neumann , Shrikanth Narayanan, Natural head motion synthesis driven by acoustic prosodic features: Virtual Humans and Social Agents, Computer Animation and Virtual Worlds, v.16 n.3-4, p.283-290, July 2005 [doi>10.1002/cav.v16:3/4]
• 4: Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
• 5: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 6: Erika Chuang , Christoph Bregler, Mood swings: expressive speech animation, ACM Transactions on Graphics (TOG), v.24 n.2, p.331-347, April 2005 [doi>10.1145/1061347.1061355]
• 7: Erika S. Chuang , Hrishikesh Deshpande , Chris Bregler, Facial Expression Space Learning, Proceedings of the 10th Pacific Conference on Computer Graphics and Applications, p.68, October 09-11, 2002
• 8: Yong Cao , Petros Faloutsos , Eddie Kohler , Frédéric Pighin, Real-time speech motion synthesis from recorded motions, Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, August 27-29, 2004, Grenoble, France [doi>10.1145/1028523.1028570]
• 9: {CG00} Cosatto E., Graf H. P.: Audio-visual unit selection for the synthesis of photo-realistic talking-heads. In Proc. of ICME (2000), pp. 619--622.
• 10: {CM93} Cohen M. M., Massaro D. W.: Modeling coarticulation in synthetic visual speech. Models and Techniques in Computer Animation, Springer Verlag (1993), 139--156.
• 11: Jin-xiang Chai , Jing Xiao , Jessica Hodgins, Vision-based control of 3D facial animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 12: Zhigang Deng , Pei-Ying Chiang , Pamela Fox , Ulrich Neumann, Animating blendshape faces by cross-mapping motion capture data, Proceedings of the 2006 symposium on Interactive 3D graphics and games, March 14-17, 2006, Redwood City, California [doi>10.1145/1111411.1111419]
• 13: {DLN05} Deng Z., Lewis J. P., Neumann U.: Synthesizing speech animation by learning compact speech co-articulation models. In Proc. of Computer Graphics International (2005), pp. 19--25.
• 14: Zhigang Deng , Ulrich Neumann , J. P. Lewis , Tae-Yong Kim , Murtaza Bulut , Shrikanth Narayanan, Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces, IEEE Transactions on Visualization and Computer Graphics, v.12 n.6, p.1523-1534, November 2006 [doi>10.1109/TVCG.2006.90]
• 15: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 16: {fes04} http://www.cstr.ed.ac.uk/projects/festival/, 2004.
• 17: Brian Guenter , Cindy Grimm , Daniel Wood , Henrique Malvar , Fredric Pighin, Making faces, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.55-66, July 1998 [doi>10.1145/280814.280822]
• 18: Pushkar Joshi , Wen C. Tien , Mathieu Desbrun , Frédéric Pighin, Learning controls for blend shape based realistic facial animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 19: Sumedha Kshirsagar , Stephane Garchery , Nadia Magnenat-Thalmann, Feature Point Based Mesh Deformation Applied to MPEG-4 Facial Animation, Proceedings of the IFIP TC5/WG5.10 DEFORM'2000 Workshop and AVATARS'2000 Workshop on Deformable Avatars, p.24-34, November 29-December 01, 2000
• 20: Kolja Kähler , Jörg Haber , Hans-Peter Seidel, Geometry-based muscle modeling for facial animation, No description on Graphics interface 2001, p.37-46, June 07-09, 2001, Ottawa, Ontario, Canada
• 21: Scott A. King , Richard E. Parent, Creating Speech-Synchronized Animation, IEEE Transactions on Visualization and Computer Graphics, v.11 n.3, p.341-352, May 2005 [doi>10.1109/TVCG.2005.43]
• 22: {KT03} Kshirsagar S., Thalmann N. M.: Visyllable based speech animation. Computer Graphics Forum 22, 3 (2003).
• 23: {Lew91} Lewis J. P.: Automated lip-sync: Background and techniques. Journal of Visualization and Computer Animation (1991), 118--122.
• 24: Yuencheng Lee , Demetri Terzopoulos , Keith Walters, Realistic modeling for facial animation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.55-62, September 1995 [doi>10.1145/218380.218407]
• 25: Jiyong Ma , Ron Cole , Bryan Pellom , Wayne Ward , Barbara Wise, Accurate Visible Speech Synthesis Based on Concatenating Variable Length Motion Capture Data, IEEE Transactions on Visualization and Computer Graphics, v.12 n.2, p.266-276, March 2006 [doi>10.1109/TVCG.2006.18]
• 26: Jun-yong Noh , Ulrich Neumann, Expression cloning, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.277-288, August 2001 [doi>10.1145/383259.383290]
• 27: Catherine Rose Emma Pelachaud, Communication and coarticulation in facial animation, University of Pennsylvania, Philadelphia, PA, 1992
• 28: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 29: Hyewon Pyun , Yejin Kim , Wonseok Chae , Hyung Woo Kang , Sung Yong Shin, An example-based approach for facial expression cloning, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 30: Frederic I. Parke , Keith Waters, Computer facial animation, A. K. Peters, Ltd., Natick, MA, 1996
• 31: Alla Safonova , Jessica K. Hodgins , Nancy S. Pollard, Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 32: Eftychios Sifakis , Igor Neverov , Ronald Fedkiw, Automatic determination of facial muscle activations from sparse motion capture marker data, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 33: {TSL00} Tenenbaum J., Silva V. D., Langford J.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 5500 (2000), 2319--2333.
• 34: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 35: {WF95} Waters K., Frisble J.: A coordinated muscle model for speech animation. Proc. of Graphics Interface '95 (1995), 163--170.
• 36: Lance Williams, Performance-driven facial animation, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.235-242, September 1990, Dallas, TX, USA
• 37: Qingshan Zhang , Zicheng Liu , Baining Guo , Harry Shum, Geometry-driven photorealistic facial expression synthesis, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 38: Li Zhang , Noah Snavely , Brian Curless , Steven M. Seitz, Spacetime faces: high resolution capture for modeling and animation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation