Video-based character animation

ABSTRACT

In this paper we introduce a video-based representation for free viewpoint visualization and motion control of 3D character models created from multiple view video sequences of real people. Previous approaches to video-based rendering provide no control of scene dynamics to manipulate, retarget, and create new 3D content from captured scenes. Here we contribute a new approach, combining image based reconstruction and video-based animation to allow controlled animation of people from captured multiple view video sequences. We represent a character as a motion graph of free viewpoint video motions for animation control. We introduce the use of geometry videos to represent reconstructed scenes of people for free viewpoint video rendering. We describe a novel spherical matching algorithm to derive global surface to surface correspondence in spherical geometry images for motion blending and the construction of seamless transitions between motion sequences. Finally, we demonstrate interactive video-based character animation with real-time rendering and free viewpoint visualization. This approach synthesizes highly realistic character animations with dynamic surface shape and appearance captured from multiple view video of people.

REFERENCES

• 1: {AFO03} Arikan O., Forsyth D. A., O'Brien J. F.: Motion synthesis from annotations. Proceedings ACM SIGGRAPH (2003), 402--408.
• 2: Peter J. Burt , Edward H. Adelson, A multiresolution spline with application to image mosaics, ACM Transactions on Graphics (TOG), v.2 n.4, p.217-236, October 1983 [doi>10.1145/245.247]
• 3: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 4: {Bou03} Bouguet J.-Y.: Camera Calibration Toolbox for Matlab: www.vision.caltech.edu/bouguetj/calib-doc. Tech. rep., MRL-INTEL, 2003.
• 5: Hector M. Briceño , Pedro V. Sander , Leonard McMillan , Steven Gortler , Hugues Hoppe, Geometry videos: a new representation for 3D animations, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 6: {CG00} Cosatto E., Graf P.: Photo-realistic talking heads from image samples. IEEE Transaction on Multimedia 2, 3 (2000), 152--163.
• 7: W. Bruce Culbertson , Thomas Malzbender , Gregory G. Slabaugh, Generalized Voxel Coloring, Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, p.100-115, September 21-22, 1999
• 8: Haili Chui , Anand Rangarajan, A new point matching algorithm for non-rigid registration, Computer Vision and Image Understanding, v.89 n.2-3, p.114-141, February 2003 [doi>10.1016/S1077-3142(03)00009-2]
• 9: Joel Carranza , Christian Theobalt , Marcus A. Magnor , Hans-Peter Seidel, Free-viewpoint video of human actors, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 10: {DYB98} Debevec P., Yu Y., Borshukov G.: Efficient view-dependent image-based rendering with projective texture-mapping. 9th Eurographics Rendering Workshop (1998), 105--116.
• 11: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 12: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 13: {GM04} Goldluecke B., Marcus M.: Space-time isosurface evolution for temporally coherent 3d reconstruction. IEEE International Conference on Computer Vision and Pattern Recognition I (2004), 350--355.
• 14: Hugues Hoppe, Progressive meshes, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.99-108, August 1996 [doi>10.1145/237170.237216]
• 15: Lucas Kovar , Michael Gleicher , Frédéric Pighin, Motion graphs, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 16: Takeo Kanade , Peter Rander , P. J. Narayanan, Virtualized Reality: Constructing Virtual Worlds from Real Scenes, IEEE MultiMedia, v.4 n.1, p.34-47, January 1997 [doi>10.1109/93.580394]
• 17: {KSE*03} Kwatra V., Schödl A., Essa I., Turk G., Bobick A.: Graphcut textures: Image and video synthesis using graph cuts. Proceedings ACM SIGGRAPH (2003), 277--286.
• 18: Yan Li , Tianshu Wang , Heung-Yeung Shum, Motion texture: a two-level statistical model for character motion synthesis, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 19: William E. Lorensen , Harvey E. Cline, Marching cubes: A high resolution 3D surface construction algorithm, ACM SIGGRAPH Computer Graphics, v.21 n.4, p.163-169, July 1987
• 20: Jehee Lee , Jinxiang Chai , Paul S. A. Reitsma , Jessica K. Hodgins , Nancy S. Pollard, Interactive control of avatars animated with human motion data, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 21: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 22: Wojciech Matusik , Chris Buehler , Ramesh Raskar , Steven J. Gortler , Leonard McMillan, Image-based visual hulls, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.369-374, July 2000 [doi>10.1145/344779.344951]
• 23: Saied Moezzi , Li-Cheng Tai , Philippe Gerard, Virtual View Generation for 3D Digital Video, IEEE MultiMedia, v.4 n.1, p.18-26, January 1997 [doi>10.1109/93.580392]
• 24: {PF01} Plankers R., Fua P.: Articulated soft objects for video-based body modeling. IEEE International Conference on Computer Vision (2001), 394--401.
• 25: {PH03} Praun E., Hoppe H.: Spherical parameterization and remeshing. Proceedings ACM SIGGRAPH (2003), 340--349.
• 26: Arno Schödl , Irfan A. Essa, Controlled animation of video sprites, Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 21-22, 2002, San Antonio, Texas [doi>10.1145/545261.545281]
• 27: Jonathan Starck , Adrian Hilton, Model-Based Multiple View Reconstruction of People, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.915, October 13-16, 2003
• 28: J. Starck , A. Hilton, Virtual view synthesis of people from multiple view video sequences, Graphical Models, v.67 n.6, p.600-620, November 2005 [doi>10.1016/j.gmod.2005.01.008]
• 29: Arno Schödl , Richard Szeliski , David H. Salesin , Irfan Essa, Video textures, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.489-498, July 2000 [doi>10.1145/344779.345012]
• 30: Sundar Vedula , Simon Baker , Takeo Kanade, Spatio-temporal view interpolation, Proceedings of the 13th Eurographics workshop on Rendering, June 26-28, 2002, Pisa, Italy
• 31: {VMR} www.ee.surrey.ac.uk/research/vssp/vmrg/vcphome.htm.
• 32: {ZBS04} Zhou K., Bao H., Shi J.: 3d surface filtering using spherical harmonics. Computer-Aided Design, Elsevier 36 (2004), 363--375.
• 33: C. Lawrence Zitnick , Sing Bing Kang , Matthew Uyttendaele , Simon Winder , Richard Szeliski, High-quality video view interpolation using a layered representation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism