Dynamic, expressive speech animation from a single mesh

ABSTRACT

In this work we present a method for human face animation which allows us to generate animations for a novel person given just a single mesh of their face. These animations can be of arbitrary text and may include emotional expressions. We build a multilinear model from data which encapsulates the variation in dynamic face motions over changes in identity, expression, and over different texts. We then describe a synthesis method consisting of a phoneme planning and a blending stage which uses this model as a base and attempts to preserve both face shape and dynamics given a novel text and an emotion at each point in time.

REFERENCES

• 1: {BBVP03} Blanz V., Basso C., Vetter T., Poggio T.: Reanimating faces in images and video. In EUROGRAPHICS 2003 (EUROGRAPHICS-03): the European Association for Computer Graphics, 24th Annual Conference (Granada, Spain, 2003), Brunet P., Fellner D. W., (Eds.), vol. 22 of Computer Graphics Forum, The Eurographics Association, Blackwell, pp. 641--650.
• 2: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 3: Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
• 4: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 5: {Cam} Cambridge University Engineering Department: Hidden markov model toolkit. http://htk.eng.cam.ac.uk/.
• 6: Erika Chuang , Christoph Bregler, Mood swings: expressive speech animation, ACM Transactions on Graphics (TOG), v.24 n.2, p.331-347, April 2005 [doi>10.1145/1061347.1061355]
• 7: Yao-Jen Chang , Tony Ezzat, Transferable videorealistic speech animation, Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 29-31, 2005, Los Angeles, California [doi>10.1145/1073368.1073388]
• 8: Yong Cao , Wen C. Tien , Petros Faloutsos , Frédéric Pighin, Expressive speech-driven facial animation, ACM Transactions on Graphics (TOG), v.24 n.4, p.1283-1302, October 2005 [doi>10.1145/1095878.1095881]
• 9: Z. Deng , J. P. Lewis , U. Neumann, Synthesizing speech animation by learning compact speech co-articulation models, Proceedings of the Computer Graphics International 2005, p.19-25, June 02-04, 2005
• 10: {EF78} Ekman P., Friesen W.: The facial action coding system: A technique for the measurement of facial movement, 1978.
• 11: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 12: Michael J. Jones , Tomaso Poggio, Multidimensional Morphable Models: A Framework for Representing and Matching Object Classes, International Journal of Computer Vision, v.29 n.2, p.107-131, Aug. 1, 1998 [doi>10.1023/A:1008074226832]
• 13: Zhigang Deng , Ulrich Neumann , J. P. Lewis , Tae-Yong Kim , Murtaza Bulut , Shrikanth Narayanan, Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces, IEEE Transactions on Visualization and Computer Graphics, v.12 n.6, p.1523-1534, November 2006 [doi>10.1109/TVCG.2006.90]
• 14: Scott A. King , Richard E. Parent, Creating Speech-Synchronized Animation, IEEE Transactions on Visualization and Computer Graphics, v.11 n.3, p.341-352, May 2005 [doi>10.1109/TVCG.2005.43]
• 15: {Lat97} Lathauwer L. D.: Signal Processing based on Multilinear Algebra. PhD thesis, Faculteit der Toegepaste Wetenschappen. Katholieke Universiteit Leuven, 1997.
• 16: Yuencheng Lee , Demetri Terzopoulos , Keith Walters, Realistic modeling for facial animation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.55-62, September 1995 [doi>10.1145/218380.218407]
• 17: {LZPW03} Levin A., Zomet A., Peleg S., Weiss Y.: Seamless image stitching in the gradient domain, 2003.
• 18: {MKPG05} Mueller P., Kalberer G. A., Proesmans M., Gool L. V.: Realistic speech animation based on observed 3d face dynamics. IEE Proc. Vision, Image and Signal Processing 152 (August 2005), 491--500.
• 19: Wojciech Matusik , Matthias Zwicker , Frédo Durand, Texture design using a simplicial complex of morphable textures, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 20: F. I. Parke, Parameterized Models for Facial Animation, IEEE Computer Graphics and Applications, v.2 n.9, p.61-68, September 1982 [doi>10.1109/MCG.1982.1674492]
• 21: Frederic Pighin , J. P. Lewis, Facial motion retargeting, ACM SIGGRAPH 2006 Courses, July 30-August 03, 2006, Boston, Massachusetts [doi>10.1145/1185657.1185842]
• 22: Eftychios Sifakis , Andrew Selle , Avram Robinson-Mosher , Ronald Fedkiw, Simulating speech with a physics-based facial muscle model, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
• 23: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 24: M. A. O. Vasilescu , Demetri Terzopoulos, Multilinear Analysis of Image Ensembles: TensorFaces, Proceedings of the 7th European Conference on Computer Vision-Part I, p.447-460, May 28-31, 2002
• 25: M. Alex O. Vasilescu , Demetri Terzopoulos, TensorTextures: multilinear image-based rendering, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 26: Keith Waters, A muscle model for animation three-dimensional facial expression, Proceedings of the 14th annual conference on Computer graphics and interactive techniques, p.17-24, August 1987
• 27: Lance Williams, Performance-driven facial animation, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.235-242, September 1990, Dallas, TX, USA
• 28: Hongcheng Wang , Qing Wu , Lin Shi , Yizhou Yu , Narendra Ahuja, Out-of-core tensor approximation of multi-dimensional matrices of visual data, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 29: Jin-xiang Chai , Jing Xiao , Jessica Hodgins, Vision-based control of 3D facial animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 30: Li Zhang , Noah Snavely , Brian Curless , Steven M. Seitz, Spacetime faces: high resolution capture for modeling and animation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism