Real-time expression cloning using appearance models

ABSTRACT

Active Appearance Models (AAMs) are generative parametric models commonly used to track, recognise and synthesise faces in images and video sequences. In this paper we describe a method for transferring dynamic facial gestures between subjects in real-time. The main advantages of our approach are that: 1) the mapping is computed automatically and does not require high-level semantic information describing facial expressions or visual speech gestures. 2) The mapping is simple and intuitive, allowing expressions to be transferred and rendered in real-time. 3) The mapped expression can be constrained to have the appearance of the target producing the expression, rather than the source expression imposed onto the target face. 4) Near-videorealistic talking faces for new subjects can be created without the cost of recording and processing a complete training corpus for each. Our system enables face-to-face interaction with an avatar driven by an AAM of an actual person in real-time and we show examples of arbitrary expressive speech frames cloned across different subjects.

REFERENCES

• 1: V. Blanz, C. Basso, T. Poggio, and T. Vetter. Reanimating faces in images and video. In Eurographics, pages 641--650, 2003.
• 2: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 3: Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
• 4: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 5: Yao-Jen Chang , Tony Ezzat, Transferable videorealistic speech animation, Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 29-31, 2005, Los Angeles, California [doi>10.1145/1073368.1073388]
• 6: B. Choe, H. Lee, and H. Ko. Performance-driven muscle-based facial animation. Journal of Visualization and Computer Animation, 11(2):67--79, May 2001.
• 7: E. Chuang and C. Bregler. Performance driven facial animation using blendshape interpolation. Technical Report CS--TR--2002--02, Stanford University, April 2002.
• 8: Timothy F. Cootes , Gareth J. Edwards , Christopher J. Taylor, Active Appearance Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.681-685, June 2001 [doi>10.1109/34.927467]
• 9: E. Cosatto , H. Graf, Sample-Based Synthesis of Photo-Realistic Talking Heads, Proceedings of the Computer Animation, p.103, June 08-10, 1998
• 10: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 11: T. Ezzat , T. Poggio, MikeTalk: A Talking Facial Display Based on Morphing Visemes, Proceedings of the Computer Animation, p.96, June 08-10, 1998
• 12: Brian Guenter , Cindy Grimm , Daniel Wood , Henrique Malvar , Fredric Pighin, Making faces, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.55-66, July 1998 [doi>10.1145/280814.280822]
• 13: T. Hawkins, A. Wenger, C. Tchou, A. Gardner, F. Goransson, and P. Debevec. Animatable facial reflectance fields. In Eurographics Symposium on Rendering, June 2004.
• 14: Yuencheng Lee , Demetri Terzopoulos , Keith Walters, Realistic modeling for facial animation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.55-62, September 1995 [doi>10.1145/218380.218407]
• 15: Zicheng Liu , Ying Shan , Zhengyou Zhang, Expressive expression mapping with ratio images, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.271-276, August 2001 [doi>10.1145/383259.383289]
• 16: I. Macedo, E. Vital Brazil, and L. Velho. Expression transfer between photographs through multilinear aam's. In Brazilian Symposium on Computer Graphics and Image Processing, pages 239--246, 2006.
• 17: D. Massaro. Perceiving Talking Faces. The MIT Press, 1998.
• 18: Jun-yong Noh , Ulrich Neumann, Expression cloning, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.277-288, August 2001 [doi>10.1145/383259.383290]
• 19: F. I. Parke, Parameterized Models for Facial Animation, IEEE Computer Graphics and Applications, v.2 n.9, p.61-68, September 1982 [doi>10.1109/MCG.1982.1674492]
• 20: Frederic I. Parke , Keith Waters, Computer facial animation, A. K. Peters, Ltd., Natick, MA, 1996
• 21: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 22: D. Terzopoulos and K. Waters. Physically-based facial modelling, analysis and animation. Journal of Visualization and Computer Animation, 1(2):73--80, 1990.
• 23: B. Theobald, J. Bangham, I. Matthews, and G. Cawley. Visual speech synthesis using statistical models of shape and appearance. In Proceedings of Auditory Visual Speech Processing, pages 78--83, 2001.
• 24: B. Theobald, J. Bangham, I. Matthews, and G. Cawley. Near-videorealistic synthetic talking faces: Implementation and evaluation. Speech Communication, 44:127--140, 2004.
• 25: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 26: Keith Waters, A muscle model for animation three-dimensional facial expression, Proceedings of the 14th annual conference on Computer graphics and interactive techniques, p.17-24, August 1987
• 27: Lance Williams, Performance-driven facial animation, ACM SIGGRAPH Computer Graphics, v.24 n.4, p.235-242, Aug. 1990
• 28: Qingshan Zhang , Zicheng Liu , Baining Guo , Demetri Terzopoulos , Heung-Yeung Shum, Geometry-Driven Photorealistic Facial Expression Synthesis, IEEE Transactions on Visualization and Computer Graphics, v.12 n.1, p.48-60, January 2006 [doi>10.1109/TVCG.2006.9]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality; Animation

General Terms:
Human Factors, Measurement, Performance

Keywords:
active appearance models, expression cloning, facial animation