Interactive 3D facial expression posing through 2D portrait manipulation

ABSTRACT

Sculpting various 3D facial expressions from a static 3D face model is a process with intensive manual tuning efforts. In this paper, we present an interactive 3D facial expression posing system through 2D portrait manipulation, where a manipulated 2D portrait serves a metaphor for automatically inferring its corresponding 3D facial expression with fine details. Users either rapidly assemble a face portrait through a pre-designed portrait component library or intuitively modify an initial portrait. During the editing procedure, when the users move one or a group of 2D control points on the portrait, other portrait control points are adjusted in order to automatically maintain the faceness of the edited portrait if the automated propagation function (switch) is optionally turned on. Finally, the 2D portrait is used as a query input to search for and reconstruct its corresponding 3D facial expression from a pre-recorded facial motion capture database. We showed that this system is effective for rapid 3D facial expression sculpting through a comparative user study.

REFERENCES

• 1: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 2: Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
• 3: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 4: R. Brilliant. Portraiture. Reaktion Books, 2001.
• 5: Carlos Busso , Zhigang Deng , Ulrich Neumann , Shrikanth Narayanan, Natural head motion synthesis driven by acoustic prosodic features: Virtual Humans and Social Agents, Computer Animation and Virtual Worlds, v.16 n.3-4, p.283-290, July 2005 [doi>10.1002/cav.v16:3/4]
• 6: Yong Cao , Petros Faloutsos , Frédéric Pighin, Unsupervised learning for speech motion editing, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 7: Edwin Chang , Odest Chadwicke Jenkins, Sketching articulation and pose for facial animation, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
• 8: Hong Chen , Ziqiang Liu , Chuck Rose , Yingqing Xu , Heung-Yeung Shum , David Salesin, Example-based composite sketching of human portraits, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987673]
• 9: H. Chen, Y.-Q. Xu, H. Shum, S. Zhu, and N. Zheng. Example-based facial sketch generation with nonparametric sampling. In ICCV 2001, pages 433--438, 2001.
• 10: B. W. Choe and H. S. Ko. Analysis and synthesis of facial expressions with hand-generated muscle actuation basis. In Proc. of IEEE Computer Animation, pages 12--19, 2001.
• 11: Erika S. Chuang , Hrishikesh Deshpande , Chris Bregler, Facial Expression Space Learning, Proceedings of the 10th Pacific Conference on Computer Graphics and Applications, p.68, October 09-11, 2002
• 12: Sabine Coquillart, Extended free-form deformation: a sculpturing tool for 3D geometric modeling, ACM SIGGRAPH Computer Graphics, v.24 n.4, p.187-196, Aug. 1990
• 13: James Davis , Maneesh Agrawala , Erika Chuang , Zoran Popović , David Salesin, A sketching interface for articulated figure animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 14: Zhigang Deng , Pei-Ying Chiang , Pamela Fox , Ulrich Neumann, Animating blendshape faces by cross-mapping motion capture data, Proceedings of the 2006 symposium on Interactive 3D graphics and games, March 14-17, 2006, Redwood City, California [doi>10.1145/1111411.1111419]
• 15: Zhigang Deng , Ulrich Neumann, eFASE: expressive facial animation synthesis and editing with phoneme-isomap controls, Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, September 02-04, 2006, Vienna, Austria
• 16: Zhigang Deng , Ulrich Neumann, Data-Driven 3D Facial Animation, 2007
• 17: Zhigang Deng , Ulrich Neumann , J. P. Lewis , Tae-Yong Kim , Murtaza Bulut , Shrikanth Narayanan, Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces, IEEE Transactions on Visualization and Computer Graphics, v.12 n.6, p.1523-1534, November 2006 [doi>10.1109/TVCG.2006.90]
• 18: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 19: Bruce Gooch , Erik Reinhard , Amy Gooch, Human facial illustrations: Creation and psychophysical evaluation, ACM Transactions on Graphics (TOG), v.23 n.1, p.27-44, January 2004 [doi>10.1145/966131.966133]
• 20: T. Hastie, R. Ribshirani, and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer-Verlag, 2001.
• 21: Pushkar Joshi , Wen C. Tien , Mathieu Desbrun , Frédéric Pighin, Learning controls for blend shape based realistic facial animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 22: Robert D. Kalnins , Lee Markosian , Barbara J. Meier , Michael A. Kowalski , Joseph C. Lee , Philip L. Davidson , Matthew Webb , John F. Hughes , Adam Finkelstein, WYSIWYG NPR: drawing strokes directly on 3D models, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 23: P. Kalra, A. Mangili, N. Thalmann, and D. Thalmann. Simulation of facial muscle actions based on rational free from deformations. In Eurographics'92, volume 11, pages 59--69, 1992.
• 24: Olga A. Karpenko , John F. Hughes, SmoothSketch: 3D free-form shapes from complex sketches, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 25: Sumedha Kshirsagar , Stephane Garchery , Nadia Magnenat-Thalmann, Feature Point Based Mesh Deformation Applied to MPEG-4 Facial Animation, Proceedings of the IFIP TC5/WG5.10 DEFORM'2000 Workshop and AVATARS'2000 Workshop on Deformable Avatars, p.24-34, November 29-December 01, 2000
• 26: Manfred Lau , Jinxiang Chai , Ying-Qing Xu , Heung-Yeung Shum, Face poser: interactive modeling of 3D facial expressions using model priors, Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation, August 02-04, 2007, San Diego, California
• 27: K. R. Laughery and R. H. Fowler. Sketch artist and identi-kit procedure for recalling faces. Journal of Applied Psychology, 65(3):307--316, 1980.
• 28: Yuencheng Lee , Demetri Terzopoulos , Keith Walters, Realistic modeling for facial animation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.55-62, September 1995 [doi>10.1145/218380.218407]
• 29: J. P. Lewis , Jonathan Mooser , Zhigang Deng , Ulrich Neumann, Reducing blendshape interference by selected motion attenuation, Proceedings of the 2005 symposium on Interactive 3D graphics and games, April 03-06, 2005, Washington, District of Columbia [doi>10.1145/1053427.1053431]
• 30: J. P. Lewis , Matt Cordner , Nickson Fong, Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.165-172, July 2000 [doi>10.1145/344779.344862]
• 31: Q. Li and Z. Deng. Facial motion capture editing by automated orthogonal blendshape construction and weight propagation. IEEE Computer Graphics and Applications, 2008, (to appear).
• 32: Zhenyao Mo , J. P. Lewis , Ulrich Neumann, Improved automatic caricature by feature normalization and exaggeration, ACM SIGGRAPH 2004 Sketches, August 08-12, 2004, Los Angeles, California [doi>10.1145/1186223.1186294]
• 33: A. Moore. An intoductory tutorial on KD-trees. PhD Thesis: Efficient Memory based Learning for Robot Control, PhD Thesis Technical Report No. 209, 1990. University of Cambridge.
• 34: A. Moore and J. Ostlund. Simple kd-tree library. http://www.autonlab.org/, 2007.
• 35: Jun-yong Noh , Ulrich Neumann, Expression cloning, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.277-288, August 2001 [doi>10.1145/383259.383290]
• 36: Frederic I. Parke , Keith Waters, Computer facial animation, A. K. Peters, Ltd., Natick, MA, 1996
• 37: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 38: Hyewon Pyun , Yejin Kim , Wonseok Chae , Hyung Woo Kang , Sung Yong Shin, An example-based approach for facial expression cloning, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 39: T. Sederberg and S. Parry. Free-form deformation of solid geometry models. In Proc. of SIGGRAPH'96, volume 20, pages 151--160, 1996.
• 40: Eftychios Sifakis , Igor Neverov , Ronald Fedkiw, Automatic determination of facial muscle activations from sparse motion capture marker data, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 41: Karan Singh , Eugene Fiume, Wires: a geometric deformation technique, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.405-414, July 1998 [doi>10.1145/280814.280946]
• 42: M. Sousa and J. Buchannan. Computer-generated pencil drawing. In Proc. SKIGraph'99, 1999.
• 43: M. Viad and H. Yahia. Facial animation with wrinkles. In Proc. of Eurgraphics Workshop on Animation and Simulation, 1992.
• 44: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 45: C. Wang and D. R. Forsey. Langwidere: A new facial animation system. In Proc. of IEEE Computer Animation, pages 59--68, 1994.
• 46: Qingshan Zhang , Zicheng Liu , Baining Guo , Harry Shum, Geometry-driven photorealistic facial expression synthesis, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Keywords:
data-driven, face portrait, facial expression, motion capture