Controlled animation of video sprites

ABSTRACT

We introduce a new optimization algorithm for video sprites to animate realistic-looking characters. Video sprites are animations created by rearranging recorded video frames of a moving object. Our new technique to find good frame arrangements is based on repeated partial replacements of the sequence. It allows the user to specify animations using a flexible cost function. We also show a fast technique to compute video sprite transitions and a simple algorithm to correct for perspective effects of the input footage. We use our techniques to create character animations of animals, which are difficult both to train in the real world and to animate as 3D models.

REFERENCES

• 1: Okan Arikan , D. A. Forsyth, Interactive motion generation from examples, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 2: BARJOSEPH, Z. 1999. Statistical learning of multi-dimensional textures. Master's thesis, The Hebrew University of Jerusalem.
• 3: Jeremy S. De Bonet, Multiresolution sampling procedure for analysis and synthesis of texture images, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.361-368, August 1997 [doi>10.1145/258734.258882]
• 4: BRADKSI, G., AND PISAREVSKY, V. 2000. Intel's computer vision library: Applications in calibration, stereo, segmentation, tracking, gesture, face, and object recognition. In In Proceedings of IEEE Computer Vision and Pattern Recognition Conference 2000, vol. II, II:796-797. Demonstration Paper.
• 5: C. Bregler , J. Malik, Tracking People with Twists and Exponential Maps, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.8, June 23-25, 1998
• 6: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 7: Chris Buehler , Michael Bosse , Leonard McMillan , Steven Gortler , Michael Cohen, Unstructured lumigraph rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.425-432, August 2001 [doi>10.1145/383259.383309]
• 8: Shenchang Eric Chen, QuickTime VR: an image-based approach to virtual environment navigation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.29-38, September 1995 [doi>10.1145/218380.218395]
• 9: Michael F. Cohen, Interactive spacetime control for animation, ACM SIGGRAPH Computer Graphics, v.26 n.2, p.293-302, July 1992
• 10: E. Cosatto , H. Graf, Sample-Based Synthesis of Photo-Realistic Talking Heads, Proceedings of the Computer Animation, p.103, June 08-10, 1998
• 11: Paul Dana, Issues and techniques for keyframing transformations, Graphics Gems III, Academic Press Professional, Inc., San Diego, CA, 1992
• 12: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 13: Irfan Essa , Sumit Basu , Trevor Darrell , Alex Pentland, Modeling, Tracking and Interactive Animation of Faces and Heads Using Input from Video, Proceedings of the Computer Animation, p.68, June 03-04, 1996
• 14: FITZGIBBON, A. W. 2001. Stochastic rigidity: Image registration for nowhere-static scenes. In Proceeding of IEEE International Conference on Computer Vision 2001, I: 662-669.
• 15: Michael Gleicher, Retargetting motion to new characters, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.33-42, July 1998 [doi>10.1145/280814.280820]
• 16: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 17: David J. Heeger , James R. Bergen, Pyramid-based texture analysis/synthesis, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.229-238, September 1995 [doi>10.1145/218380.218446]
• 18: Jessica K. Hodgins , Wayne L. Wooten , David C. Brogan , James F. O'Brien, Animating human athletics, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.71-78, September 1995 [doi>10.1145/218380.218414]
• 19: HODGINS, J. K. 1998. Animating human motion. Scientific American 278, 3 (Mar.), 64-69 (Intl. ed. 46-51).
• 20: KAELBLING, L. P., LITTMAN, M. L., AND MOORE, A. P. 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237-285.
• 21: Lucas Kovar , Michael Gleicher , Frédéric Pighin, Motion graphs, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 22: John Lasseter, Principles of traditional animation applied to 3D computer animation, ACM SIGGRAPH Computer Graphics, v.21 n.4, p.35-44, July 1987
• 23: Joseph Laszlo , Michiel van de Panne , Eugene Fiume, Limit cycle control and its application to the animation of balancing and walking, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.155-162, August 1996 [doi>10.1145/237170.237231]
• 24: Jehee Lee , Jinxiang Chai , Paul S. A. Reitsma , Jessica K. Hodgins , Nancy S. Pollard, Interactive control of avatars animated with human motion data, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 25: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 26: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• 27: J. Thomas Ngo , Joe Marks, Spacetime constraints revisited, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.343-350, September 1993 [doi>10.1145/166117.166160]
• 28: Zoran Popović , Andrew Witkin, Physically based motion transformation, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.11-20, July 1999 [doi>10.1145/311535.311536]
• 29: Charles Rose , Michael F. Cohen , Bobby Bodenheimer, Verbs and Adverbs: Multidimensional Motion Interpolation, IEEE Computer Graphics and Applications, v.18 n.5, p.32-40, September 1998 [doi>10.1109/38.708559]
• 30: SCHÖDL, A., AND ESSA, I. A. 2001. Machine learning for video-based rendering. In Advances in Neural Information Processing Systems, MIT Press, USA, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds., vol. 13, 1002-1008.
• 31: Arno Schödl , Richard Szeliski , David H. Salesin , Irfan Essa, Video textures, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.489-498, July 2000 [doi>10.1145/344779.345012]
• 32: Steven M. Seitz , Charles R. Dyer, View morphing, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.21-30, August 1996 [doi>10.1145/237170.237196]
• 33: SOATTO, S., DORETTO, G., AND WU, Y. N. 2001. Dynamic textures. In Proceeding of IEEE International Conference on Computer Vision, II: 439-446.
• 34: Richard Szeliski , Heung-Yeung Shum, Creating full view panoramic image mosaics and environment maps, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.251-258, August 1997 [doi>10.1145/258734.258861]
• 35: SZUMMER, M., AND PICARD, R. W. 1996. Temporal texture modeling. In Proceeding of IEEE International Conference on Image Processing, vol. 3, 823-826.
• 36: D. Terzopoulos , K. Waters, Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.6, p.569-579, June 1993 [doi>10.1109/34.216726]
• 37: Li-Yi Wei , Marc Levoy, Fast texture synthesis using tree-structured vector quantization, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.479-488, July 2000 [doi>10.1145/344779.345009]
• 38: Lance Williams, Performance-driven facial animation, ACM SIGGRAPH Computer Graphics, v.24 n.4, p.235-242, Aug. 1990
• 39: Andrew Witkin , Michael Kass, Spacetime constraints, ACM SIGGRAPH Computer Graphics, v.22 n.4, p.159-168, Aug. 1988
• 40: ZHANG, Z. 1998. A flexibe new technique for camera calibration. Tech. Rep. 98-71, Microsoft Research. www.research.microsoft.com/~zhang/Calib/.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

Keywords:
character animation, optimization, video sprites, video textures