Performance capture from sparse multi-view video

ABSTRACT

This paper proposes a new marker-less approach to capturing human performances from multi-view video. Our algorithm can jointly reconstruct spatio-temporally coherent geometry, motion and textural surface appearance of actors that perform complex and rapid moves. Furthermore, since our algorithm is purely meshbased and makes as few as possible prior assumptions about the type of subject being tracked, it can even capture performances of people wearing wide apparel, such as a dancer wearing a skirt. To serve this purpose our method efficiently and effectively combines the power of surface- and volume-based shape deformation techniques with a new mesh-based analysis-through-synthesis framework. This framework extracts motion constraints from video and makes the laser-scan of the tracked subject mimic the recorded performance. Also small-scale time-varying shape detail is recovered by applying model-guided multi-view stereo to refine the model surface. Our method delivers captured performance data at high level of detail, is highly versatile, and is applicable to many complex types of scenes that could not be handled by alternative marker-based or marker-free recording techniques.

REFERENCES

• 1: Brett Allen , Brian Curless , Zoran Popović, Articulated body deformation from range scan data, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 2: Balan, A. O., Sigal, L., Black, M. J., Davis, J. E., and Haussecker, H. W. 2007. Detailed human shape and pose from images. In Proc. CVPR.
• 3: Bernd Bickel , Mario Botsch , Roland Angst , Wojciech Matusik , Miguel Otaduy , Hanspeter Pfister , Markus Gross, Multi-scale capture of facial geometry and motion, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 4: Mario Botsch , Olga Sorkine, On Linear Variational Surface Deformation Methods, IEEE Transactions on Visualization and Computer Graphics, v.14 n.1, p.213-230, January 2008 [doi>10.1109/TVCG.2007.1054]
• 5: Botsch, M., Pauly, M., Wicke, M., and Gross, M. 2007. Adaptive space deformations based on rigid cells. Computer Graphics Forum 26, 3, 339--347.
• 6: Richard H. Byrd , Peihuang Lu , Jorge Nocedal , Ciyou Zhu, A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific Computing, v.16 n.5, p.1190-1208, Sept. 1995 [doi>10.1137/0916069]
• 7: Joel Carranza , Christian Theobalt , Marcus A. Magnor , Hans-Peter Seidel, Free-viewpoint video of human actors, ACM SIGGRAPH 2003 Papers, July 27-31, 2003, San Diego, California
• 8: de Aguiar, E., Theobalt, C., Stoll, C., and Seidel, H.-P. 2007. Marker-less deformable mesh tracking for human shape and motion capture. In Proc. CVPR, IEEE, 1--8.
• 9: de Aguiar, E., Theobalt, C., Stoll, C., and Seidel, H. 2007. Marker-less 3d feature tracking for mesh-based human motion capture. In Proc. ICCV HUMO07, 1--15.
• 10: de Aguiar, E., Theobalt, C., Thrun, S., and Seidel, H.-P. 2008. Automatic conversion of mesh animations into skeleton-based animations. Computer Graphics Forum (Proc. Eurographics EG'08) 27, 2 (4), 389--397.
• 11: Einarsson, P., Chabert, C.-F., Jones, A., Ma, W.-C., Lamond, B., im Hawkins, Bolas, M., Sylwan, S., and Debevec, P. 2006. Relighting human locomotion with flowed reflectance fields. In Proc. EGSR, 183--194.
• 12: Michael Goesele , Brian Curless , Steven M. Seitz, Multi-View Stereo Revisited, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2402-2409, June 17-22, 2006 [doi>10.1109/CVPR.2006.199]
• 13: Markus Gross , Stephan Würmlin , Martin Naef , Edouard Lamboray , Christian Spagno , Andreas Kunz , Esther Koller-Meier , Tomas Svoboda , Luc Van Gool , Silke Lang , Kai Strehlke , Andrew Vande Moere , Oliver Staadt, blue-c: a spatially immersive display and 3D video portal for telepresence, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 14: Takeo Kanade , Peter Rander , P. J. Narayanan, Virtualized Reality: Constructing Virtual Worlds from Real Scenes, IEEE MultiMedia, v.4 n.1, p.34-47, January 1997 [doi>10.1109/93.580394]
• 15: Michael Kazhdan , Matthew Bolitho , Hugues Hoppe, Poisson surface reconstruction, Proceedings of the fourth Eurographics symposium on Geometry processing, June 26-28, 2006, Cagliari, Sardinia, Italy
• 16: Marius Leordeanu , Martial Hebert, A Spectral Technique for Correspondence Problems Using Pairwise Constraints, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1482-1489, October 17-20, 2005 [doi>10.1109/ICCV.2005.20]
• 17: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 18: Wojciech Matusik , Chris Buehler , Ramesh Raskar , Steven J. Gortler , Leonard McMillan, Image-based visual hulls, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.369-374, July 2000 [doi>10.1145/344779.344951]
• 19: Alberto Menache, Understanding Motion Capture for Computer Animation and Video Games, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 20: Niloy J. Mitra , Simon Flöry , Maks Ovsjanikov , Natasha Gelfand , Leonidas Guibas , Helmut Pottmann, Dynamic geometry registration, Proceedings of the fifth Eurographics symposium on Geometry processing, July 04-06, 2007, Barcelona, Spain
• 21: Thomas B. Moeslund , Adrian Hilton , Volker Krüger, A survey of advances in vision-based human motion capture and analysis, Computer Vision and Image Understanding, v.104 n.2, p.90-126, November 2006 [doi>10.1016/j.cviu.2006.08.002]
• 22: Matthias Müller , Julie Dorsey , Leonard McMillan , Robert Jagnow , Barbara Cutler, Stable real-time deformations, Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 21-22, 2002, San Antonio, Texas [doi>10.1145/545261.545269]
• 23: Paramount, 2007. Beowulf movie page. http://www.beowulfmovie.com/.
• 24: Sang Il Park , Jessica K. Hodgins, Capturing and animating skin deformation in human motion, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 25: Ronald Poppe, Vision-based human motion analysis: An overview, Computer Vision and Image Understanding, v.108 n.1-2, p.4-18, October, 2007 [doi>10.1016/j.cviu.2006.10.016]
• 26: Rosenhahn, B., Kersting, U., Powel, K., and Seidel, H.-P. 2006. Cloth x-ray: Mocap of people wearing textiles. In LNCS 4174: Proc. DAGM, 495--504.
• 27: Peter Sand , Leonard McMillan , Jovan Popović, Continuous capture of skin deformation, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 28: Volker Scholz , Timo Stich , Marcus Magnor , Michael Keckeisen , Markus Wacker, Garment motion capture using color-coded patterns, ACM SIGGRAPH 2005 Sketches, July 31-August 04, 2005, Los Angeles, California [doi>10.1145/1187112.1187157]
• 29: Mikio Shinya, Unifying Measured Point Sequences of Deforming Objects, Proceedings of the 3D Data Processing, Visualization, and Transmission, 2nd International Symposium, p.904-911, September 06-09, 2004 [doi>10.1109/3DPVT.2004.135]
• 30: Olga Sorkine , Marc Alexa, As-rigid-as-possible surface modeling, Proceedings of the fifth Eurographics symposium on Geometry processing, July 04-06, 2007, Barcelona, Spain
• 31: Jonathan Starck , Adrian Hilton, Surface Capture for Performance-Based Animation, IEEE Computer Graphics and Applications, v.27 n.3, p.21-31, May 2007 [doi>10.1109/MCG.2007.68]
• 32: Stoll, C., Karni, Z., Rössl, C., Yamauchi, H., and Seidel, H.-P. 2006. Template deformation for point cloud fitting. In Proc. SGP, 27--35.
• 33: Robert W. Sumner , Jovan Popović, Deformation transfer for triangle meshes, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 34: Sundar Vedula , Simon Baker , Takeo Kanade, Image-based spatio-temporal modeling and view interpolation of dynamic events, ACM Transactions on Graphics (TOG), v.24 n.2, p.240-261, April 2005 [doi>10.1145/1061347.1061351]
• 35: Michael Wand , Philipp Jenke , Qixing Huang , Martin Bokeloh , Leonidas Guibas , Andreas Schilling, Reconstruction of deforming geometry from time-varying point clouds, Proceedings of the fifth Eurographics symposium on Geometry processing, July 04-06, 2007, Barcelona, Spain
• 36: Waschbüsch, M., Würmlin, S., Cotting, D., Sadlo, F., and Gross, M. 2005. Scalable 3D video of dynamic scenes. In Proc. Pacific Graphics, 629--638.
• 37: Ryan White , Keenan Crane , D. A. Forsyth, Capturing and animating occluded cloth, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 38: Bennett Wilburn , Neel Joshi , Vaibhav Vaish , Eino-Ville Talvala , Emilio Antunez , Adam Barth , Andrew Adams , Mark Horowitz , Marc Levoy, High performance imaging using large camera arrays, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 39: Weiwei Xu , Kun Zhou , Yizhou Yu , Qifeng Tan , Qunsheng Peng , Baining Guo, Gradient domain editing of deforming mesh sequences, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 40: Yamauchi, H., Gumhold, S., Zayer, R., and Seidel, H.-P. 2005. Mesh segmentation driven by gaussian curvature. Visual Computer 21, 8--10, 649--658.
• 41: C. Lawrence Zitnick , Sing Bing Kang , Matthew Uyttendaele , Simon Winder , Richard Szeliski, High-quality video view interpolation using a layered representation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

Keywords:
marker-less scene reconstruction, multi-view video analysis, performance capture