Performance animation from low-dimensional control signals

ABSTRACT

This paper introduces an approach to performance animation that employs video cameras and a small set of retro-reflective markers to create a low-cost, easy-to-use system that might someday be practical for home use. The low-dimensional control signals from the user's performance are supplemented by a database of pre-recorded human motion. At run time, the system automatically learns a series of local models from a set of motion capture examples that are a close match to the marker locations captured by the cameras. These local models are then used to reconstruct the motion of the user as a full-body animation. We demonstrate the power of this approach with real-time control of six different behaviors using two video cameras and a small set of retro-reflective markers. We compare the resulting animation to animation from commercial motion capture equipment with a full set of markers.

REFERENCES

• 1: David W. Aha, Editorial, Artificial Intelligence Review, v.11 n.1-5, p.7-10, Feb. 1997 [doi>10.1023/A:1006538427943]
• 2: Okan Arikan , D. A. Forsyth, Interactive motion generation from examples, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 3: Okan Arikan , David A. Forsyth , James F. O'Brien, Motion synthesis from annotations, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 4: Christopher G. Atkeson , Andrew W. Moore , Stefan Schaal, Locally Weighted Learning, Artificial Intelligence Review, v.11 n.1-5, p.11-73, Feb. 1997 [doi>10.1023/A:1006559212014]
• 5: Christopher G. Atkeson , Andrew W. Moore , Stefan Schaal, Locally Weighted Learning for Control, Artificial Intelligence Review, v.11 n.1-5, p.75-113, Feb. 1997 [doi>10.1023/A:1006511328852]
• 6: Badler, N. I., Hollick, M., and Granieri, J. 1993. Real-time control of a virtual human using minimal sensors. In Presence. 2(1):82--86.
• 7: Bazaraa, M. S., Sherali, H. D., and Shetty, C. 1993. Nonlinear Programming: Theory and Algorithms. John Wiley and Sons Ltd. 2nd Edition.
• 8: Christopher M. Bishop, Neural networks for pattern recognition, Oxford University Press, Oxford, 1996
• 9: Matthew Brand , Aaron Hertzmann, Style machines, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.183-192, July 2000 [doi>10.1145/344779.344865]
• 10: Matthew Brand, Shadow Puppetry, Proceedings of the International Conference on Computer Vision-Volume 2, p.1237, September 20-25, 1999
• 11: Bregler, C., and Omohundro, S. 1995. Nonlinear image interpolation using manifold learning. In Advances in Neural Information Processing Systems 7. 973--980.
• 12: Cheung, G., Baker, S., and Kanade, T. 2003. Shape-from-silhouette of articulated object and its use for human body kinematics estimation and motion capture. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 77--84.
• 13: Daelemans, W., and Van De Bosch. A. 2001. Treetalk: Memory-based word phonemisation. In Data-Driven Techniques in Speech Synthesis, Kluwer. 149--172.
• 14: Mira Dontcheva , Gary Yngve , Zoran Popović, Layered acting for character animation, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 15: William T. Freeman , David B. Anderson , Paul A. Beardsley , Chris N. Dodge , Michal Roth , Craig D. Weissman , William S. Yerazunis , Hiroshi Kage , Kazuo Kyuma , Yasunari Miyake , Ken-ichi Tanaka, Computer Vision for Interactive Computer Graphics, IEEE Computer Graphics and Applications, v.18 n.3, p.42-53, May 1998 [doi>10.1109/38.674971]
• 16: Fukunaga, K., and Olsen. D. 1971. An algorithm for finding intrinsic dimensionality of data. In IEEE Transactions on Computers. C-20:176--183.
• 17: Keith Grochow , Steven L. Martin , Aaron Hertzmann , Zoran Popović, Style-based inverse kinematics, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 18: Shang Guo , James Robergé, A high-level control mechanism for human locomotion based on parametric frame space interpolation, Proceedings of the Eurographics workshop on Computer animation and simulation '96, p.95-107, December 1996, Poitiers, France
• 19: Hinton. G., Revow, M., and Dayan, P. 1995. Recognizing handwritten digits using mixtures of linear models. In Advances in Neural Information Processing Systems 7. 1015--1022.
• 20: Howe. N., Leventon, M., and Freeman, W. 1999. Bayesian reconstruction of 3d human motion from single-camera video. In Advances in Neural Information Processing Systems 12. 820--826.
• 21: Konami Boxing and Police 911 GAME, 2001. http://www.konami.com.
• 22: Lucas Kovar , Michael Gleicher, Automated extraction and parameterization of motions in large data sets, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 23: Lucas Kovar , Michael Gleicher , Frédéric Pighin, Motion graphs, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 24: Lawrence, N. D. 2004. Gaussian process latent variable models for visualization of high dimensional data. In Advances in Neural Information Processing Systems 16. 329--336.
• 25: Jehee Lee , Jinxiang Chai , Paul S. A. Reitsma , Jessica K. Hodgins , Nancy S. Pollard, Interactive control of avatars animated with human motion data, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 26: Yan Li , Tianshu Wang , Heung-Yeung Shum, Motion texture: a two-level statistical model for character motion synthesis, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 27: Mardia, K., Kent, J., and Bibby, M. 1979. Multivariate Analysis. Academy Press.
• 28: MicroStrain 3DM-G, 2004. http://www.microstrain.com.
• 29: Oore, S., Terzopoulos, D., and Hinton, G. 2002. A desktop input device and interface for interactive 3d character. In Proceedings of Graphics Interface 2002. 133--140.
• 30: Katherine Pullen , Christoph Bregler, Motion capture assisted animation: texturing and synthesis, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 31: Ren, L., Shakhnarovich, G., Hodgins, J. K., Pfister, H., and Viola, P. A. 2004. Learning silhouette features for control of human motion. In Computer Science Technical Reports 2004, Carnegie Mellon University. CMU-CS-04-165.
• 32: Charles Rose , Michael F. Cohen , Bobby Bodenheimer, Verbs and Adverbs: Multidimensional Motion Interpolation, IEEE Computer Graphics and Applications, v.18 n.5, p.32-40, September 1998 [doi>10.1109/38.708559]
• 33: Roweis, S., and Saul, L. 2000. Nonlinear dimensionality reduction by locally linear embedding. In Science. 290(5500):2323--2326.
• 34: Alla Safonova , Jessica K. Hodgins , Nancy S. Pollard, Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 35: Bernhard Schölkopf , Alexander J. Smola , Klaus-Robert Müller, Kernel principal component analysis, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 36: Semwal, S., Hightower, R., and Stansfield, S. 1998. Mapping algorithms for real-time control of an avatar using eight sensors. In Presence. 7(1):1--21.
• 37: Hyun Joon Shin , Jehee Lee , Sung Yong Shin , Michael Gleicher, Computer puppetry: An importance-based approach, ACM Transactions on Graphics (TOG), v.20 n.2, p.67-94, April 2001 [doi>10.1145/502122.502123]
• 38: Hedvig Sidenbladh , Michael J. Black , Leonid Sigal, Implicit Probabilistic Models of Human Motion for Synthesis and Tracking, Proceedings of the 7th European Conference on Computer Vision-Part I, p.784-800, May 28-31, 2002
• 39: Sony Eye Toy Systems, 2003. http://www.eyetoy.com.
• 40: Stone, M. 1974. Cross-validatory choice and assessment of statistical predictions. In Journal of the Royal Statistical Society. 36:111--147.
• 41: Tenenbaum, J., Silva, V., and Langford, J. 2000. A global geometric framework for nonlinear dimensionality reduction. In Science. 290(5500):2319--2323.
• 42: Vicon Systems, 2004. http://www.vicon.com.
• 43: Douglas J. Wiley , James K. Hahn, Interpolation Synthesis of Articulated Figure Motion, IEEE Computer Graphics and Applications, v.17 n.6, p.39-45, November 1997 [doi>10.1109/38.626968]
• 44: Xsens Mt-9, 2004. http://www.xsens.com.
• 45: Gang Xu , Zhengyou Zhang, Epipolar Geometry in Stereo, Motion, and Object Recognition: A Unified Approach, Kluwer Academic Publishers, Norwell, MA, 1996
• 46: Yamane, K., and Nakamura, Y. 2003. Natural motion animation through constraining and deconstraining at will. In IEEE Transactions on Visualization and Computer Graphics. 9(3):352--360.
• 47: Katsu Yamane , James J. Kuffner , Jessica K. Hodgins, Synthesizing animations of human manipulation tasks, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 48: KangKang Yin , Dinesh K. Pai, FootSee: an interactive animation system, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 49: Zhang, Z. 1999. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the International Conference on Computer Vision. 666--673.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

Keywords:
dimensionality reduction, lazy learning, local modeling, motion capture data, online control of human motion, performance animation, vision-based interface