Transferable videorealistic speech animation

ABSTRACT

Image-based videorealistic speech animation achieves significant visual realism at the cost of the collection of a large 5- to 10-minute video corpus from the specific person to be animated. This requirement hinders its use in broad applications, since a large video corpus for a specific person under a controlled recording setup may not be easily obtained In this paper, we propose a model transfer and adaptation algorithm which allows for a novel person to be animated using only a small video corpus. The algorithm starts with a multidimensional morphable model (MMM) previously trained from a different speaker with a large corpus, and transfers it to the novel speaker with a much smaller corpus. The algorithm consists of 1) a novel matching-by-synthesis algorithm which semi-automatically selects new MMM prototype images from the new video corpus and 2) a novel gradient descent linear regression algorithm which adapts the MMM phoneme models to the data in the novel video corpus. Encouraging experimental results are presented in which a morphable model trained from a performer with a 10-minute corpus is transferred to a novel person using a 15-second movie clip of him as the adaptation video corpus.

REFERENCES

• 1: {BBPV03} Blanz V., Basso C., Poggio T., Vetter T.: Reanimating faces in images and video. In Proc. Eurographics '03 (2003), vol. 22.
• 2: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 3: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 4: D. Beymer , T. Poggio, Face recognition from one example view, Proceedings of the Fifth International Conference on Computer Vision, p.500, June 20-23, 1995
• 5: {CC02} Chang Y. J., Chen Y. C.: Facial model adaptation from a monocular image sequence using a textured polygonal model. Signal Processing: Image Communication 17, 5 (May 2002), 373--392.
• 6: Yong Cao , Petros Faloutsos , Eddie Kohler , Frédéric Pighin, Real-time speech motion synthesis from recorded motions, Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, August 27-29, 2004, Grenoble, France [doi>10.1145/1028523.1028570]
• 7: {CG00} Cosatto E., Graf H. P.: Photo-realistic talking-heads from image samples. IEEE Trans. on Multimedia 2, 3 (Sept. 2000), 152--163.
• 8: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 9: {Gal98} Gales M. J. F.: Cluster adaptive training for speech recognition. In Proc. the 5th International Conference on Spoken Language Processing (1998), pp. 1783--1786.
• 10: Volker Strom, Visual Prosody: Facial Movements Accompanying Speech, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, p.396, May 20-21, 2002
• 11: {GL94} Gauvain J. L., Lee C. H.: Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains. IEEE Trans. on Speech and Audio Processing 2, 2 (Apr. 1994), 291--298.
• 12: Michael Gleicher, Retargetting motion to new characters, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.33-42, July 1998 [doi>10.1145/280814.280820]
• 13: Xuedong Huang , Hsiao-Wuen Hon , Raj Reddy, Spoken Language Processing: A Guide to Theory, Algorithm, and System Development, Prentice Hall PTR, Upper Saddle River, NJ, 2001
• 14: Michael J. Jones , Tomaso Poggio, Multidimensional Morphable Models: A Framework for Representing and Matching Object Classes, International Journal of Computer Vision, v.29 n.2, p.107-131, Aug. 1, 1998 [doi>10.1023/A:1008074226832]
• 15: {KNJ*98} Kuhn R., Nguyen P., Junqua J. C., Goldwasser L., Niedzielski N., Fincke S., Field K., Contolini M.: Eigenvoices for speaker adaptation. In Proc. the 5th International Conference on Spoken Language Processing (1998), pp. 1771--1774.
• 16: Zicheng Liu , Ying Shan , Zhengyou Zhang, Expressive expression mapping with ratio images, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.271-276, August 2001 [doi>10.1145/383259.383289]
• 17: {LW95} Leggetter C. J., Woodland P. C.: Maximum likelihood linear regression for speaker adaptation of continuous density hidden markov models. Computer Speech and Language 9, 2 (1995), 171--185.
• 18: {NJ04} Na K., Jung M.: Hierarchical retargetting of fine facial motions. In Proc. Eurographics '04 (2004).
• 19: Jun-yong Noh , Ulrich Neumann, Expression cloning, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.277-288, August 2001 [doi>10.1145/383259.383290]
• 20: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 21: Arno Schödl , Richard Szeliski , David H. Salesin , Irfan Essa, Video textures, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.489-498, July 2000 [doi>10.1145/344779.345012]
• 22: {WHL*04} Wang Y., Huang X., Lee C. S., Zhang S., Li Z., Samaras D., Metaxas D., Elgammal A., Huang P.: High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. In Proc. Eurographics '04 (2004).
• 23: Qingshan Zhang , Zicheng Liu , Baining Guo , Harry Shum, Geometry-driven photorealistic facial expression synthesis, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism