Animating blendshape faces by cross-mapping motion capture data

ABSTRACT

Animating 3D faces to achieve compelling realism is a challenging task in the entertainment industry. Previously proposed face transfer approaches generally require a high-quality animated source face in order to transfer its motion to new 3D faces. In this work, we present a semi-automatic technique to directly animate popularized 3D blendshape face models by mapping facial motion capture data spaces to 3D blendshape face spaces. After sparse markers on the face of a human subject are captured by motion capture systems while a video camera is simultaneously used to record his/her front face, then we carefully select a few motion capture frames and accompanying video frames as reference mocap-video pairs. Users manually tune blendshape weights to perceptually match the animated blendshape face models with reference facial images (the reference mocap-video pairs) in order to create reference mocap-weight pairs. Finally, the Radial Basis Function (RBF) regression technique is used to map any new facial motion capture frame to blendshape weights based on the reference mocap-weight pairs. Our results demonstrate that this technique is efficient to animate blendshape face models, while offering its generality and flexiblity.

REFERENCES

• 1: Thaddeus Beier , Shawn Neely, Feature-based image metamorphosis, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.35-42, July 1992
• 2: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 3: Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. Computer Graphics Forum(Proc. of Eurographics 2003) 22, 3.
• 4: Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
• 5: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 6: Armin Bruderlin , Lance Williams, Motion signal processing, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.97-104, September 1995 [doi>10.1145/218380.218421]
• 7: Martin D. Buhmann , M. D. Buhmann, Radial Basis Functions, Cambridge University Press, New York, NY, 2003
• 8: Carlos Busso , Zhigang Deng , Ulrich Neumann , Shrikanth Narayanan, Natural head motion synthesis driven by acoustic prosodic features: Virtual Humans and Social Agents, Computer Animation and Virtual Worlds, v.16 n.3-4, p.283-290, July 2005 [doi>10.1002/cav.v16:3/4]
• 9: Yong Cao , Petros Faloutsos , Eddie Kohler , Frédéric Pighin, Real-time speech motion synthesis from recorded motions, Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, August 27-29, 2004, Grenoble, France [doi>10.1145/1028523.1028570]
• 10: Jin-xiang Chai , Jing Xiao , Jessica Hodgins, Vision-based control of 3D facial animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 11: Choe, B. W., and Ko, H. S. 2001. Analysis and synthesis of facial expressions with hand-generated muscle actuation basis. In IEEE Computer Animation Conference, 12--19.
• 12: Douglas DeCarlo , Dimitris Metaxas, The Integration of Optical Flow and Deformable Models with Applications to Human Face Shape and Motion Estimation, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.231, June 18-20, 1996
• 13: Deng, Z., Lewis, J. P., and Neumann, U. 2005. Synthesizing speech animation by learning compact speech co-articulation models. In Proc. of Computer Graphics International (CGI 2005), IEEE Computer Society Press, 19--25.
• 14: Irfan Essa , Sumit Basu , Trevor Darrell , Alex Pentland, Modeling, Tracking and Interactive Animation of Faces and Heads Using Input from Video, Proceedings of the Computer Animation, p.68, June 03-04, 1996
• 15: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 16: Festival. http://www.cstr.ed.ac.uk/projects/festival/.
• 17: Kshirsagar, S., and Thalmann, N. M. 2003. Visyllable based speech animation. Computer Graphics Forum (Proc. of Eurographics'03) 22, 3.
• 18: Seungyong Lee , George Wolberg , Sung Yong Shin, Scattered Data Interpolation with Multilevel B-Splines, IEEE Transactions on Visualization and Computer Graphics, v.3 n.3, p.228-244, July 1997 [doi>10.1109/2945.620490]
• 19: J. P. Lewis , Matt Cordner , Nickson Fong, Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.165-172, July 2000 [doi>10.1145/344779.344862]
• 20: J. P. Lewis , Jonathan Mooser , Zhigang Deng , Ulrich Neumann, Reducing blendshape interference by selected motion attenuation, Proceedings of the 2005 symposium on Interactive 3D graphics and games, April 03-06, 2005, Washington, District of Columbia [doi>10.1145/1053427.1053431]
• 21: H. Li , P. Roivainen , R. Forcheimer, 3-D Motion Estimation in Model-Based Facial Image Coding, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.6, p.545-555, June 1993 [doi>10.1109/34.216724]
• 22: Jiyong Ma , Ronald Cole , Bryan Pellom , Wayne Ward , Barbara Wise, Accurate automatic visible speech synthesis of arbitrary 3D models based on concatenation of diviseme motion capture data: Research Articles, Computer Animation and Virtual Worlds, v.15 n.5, p.485-500, December 2004 [doi>10.1002/cav.v15:5]
• 23: Gregory M. Nielson, Scattered Data Modeling, IEEE Computer Graphics and Applications, v.13 n.1, p.60-70, January 1993 [doi>10.1109/38.180119]
• 24: Jun-yong Noh , Ulrich Neumann, Expression cloning, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.277-288, August 2001 [doi>10.1145/383259.383290]
• 25: Noh, J. Y., Fidaleo, D., and Neumann, U. 2002. Gesture driven facial animation. USC CS Technical Report 02--761.
• 26: Frederic I. Parke , Keith Waters, Computer facial animation, A. K. Peters, Ltd., Natick, MA, 1996
• 27: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 28: Pighin, F., Szeliski, R., and Salesin. D. 1999. Resynthesizing facial animation through 3d model-based tracking. In IEEE International Conference on Computer Vision (ICCV), 143--150.
• 29: Hyewon Pyun , Yejin Kim , Wonseok Chae , Hyung Woo Kang , Sung Yong Shin, An example-based approach for facial expression cloning, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 30: Remington Scott, Sparking life: notes on the performance capture sessions for the Lord of the Rings: the Two Towers, ACM SIGGRAPH Computer Graphics, v.37 n.4, p.17-21, November 2003 [doi>10.1145/961261.961263]
• 31: Eftychios Sifakis , Igor Neverov , Ronald Fedkiw, Automatic determination of facial muscle activations from sparse motion capture marker data, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 32: Stegmann, M. B., and Gomez, D. D. 2002. A brief introduction to statistical shape analysis. In Informatics and Mathematical Modelling, Technical University of Denmark, DTU.
• 33: Robert W. Sumner , Jovan Popović, Deformation transfer for triangle meshes, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 34: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 35: Lance Williams, Performance-driven facial animation, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.235-242, September 1990, Dallas, TX, USA

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

Keywords:
blend shape models, cross-mapping, facial animation, motion capture data, radial basis functions