Video puppetry: a performative interface for cutout animation

ABSTRACT

We present a video-based interface that allows users of all skill levels to quickly create cutout-style animations by performing the character motions. The puppeteer first creates a cast of physical puppets using paper, markers and scissors. He then physically moves these puppets to tell a story. Using an inexpensive overhead camera our system tracks the motions of the puppets and renders them on a new background while removing the puppeteer's hands. Our system runs in real-time (at 30 fps) so that the puppeteer and the audience can immediately see the animation that is created. Our system also supports a variety of constraints and effects including articulated characters, multi-track animation, scene changes, camera controls, 2 1/2-D environments, shadows, and animation cycles. Users have evaluated our system both quantitatively and qualitatively: In tests of low-level dexterity, our system has similar accuracy to a mouse interface. For simple story telling, users prefer our system over either a mouse interface or traditional puppetry. We demonstrate that even first-time users, including an eleven-year-old, can use our system to quickly turn an original story idea into an animation.

REFERENCES

• 1: Aseem Agarwala , Aaron Hertzmann , David H. Salesin , Steven M. Seitz, Keyframe-based tracking for rotoscoping and animation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Aseem Agarwala, SnakeToonz: a semi-automatic approach to creating cel animation from video, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508554]
• 3: Alvarado, C., and Davis, R. 2001. Resolving Ambiguities to Create a Natural Sketch Based Interface. In Proc. IJCAI, 1365--1371.
• 4: Baecker, R. 1969. Picture-Driven Animation. In Proc. Spring Joint Computer Conference, 273--288.
• 5: Adrien Bousseau , Fabrice Neyret , Joëlle Thollot , David Salesin, Video watercolorization using bidirectional texture advection, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 6: Collomosse, J., and Hall, P. 2005. Video Paintbox: The Fine Art of Video Painting. Computers & Graphics, Vol. 29, No. 6, 862--870.
• 7: J. P. Collomosse , P. M. Hall, Video motion analysis for the synthesis of dynamic cues and Futurist art, Graphical Models, v.68 n.5, p.402-414, September 2006 [doi>10.1016/j.gmod.2006.05.003]
• 8: James Davis , Maneesh Agrawala , Erika Chuang , Zoran Popović , David Salesin, A sketching interface for articulated figure animation, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 9: Richard C. Davis , Brien Colwell , James A. Landay, K-sketch: a 'kinetic' sketch pad for novice animators, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357122]
• 10: Mira Dontcheva , Gary Yngve , Zoran Popović, Layered acting for character animation, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 11: Mark Fiala, ARTag, a Fiducial Marker System Using Digital Techniques, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.590-596, June 20-26, 2005 [doi>10.1109/CVPR.2005.74]
• 12: S. Gottschalk , M. C. Lin , D. Manocha, OBBTree: a hierarchical structure for rapid interference detection, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.171-180, August 1996 [doi>10.1145/237170.237244]
• 13: Horn, B. K. P. 1986. Closed-Form Solution of Absolute Orientation using Unit Quaternions. Journal of the Optical Society of America, Vol. 4, 629--642.
• 14: Takeo Igarashi , Tomer Moscovich , John F. Hughes, As-rigid-as-possible shape manipulation, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 15: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 16: Jiwon Kim , Steven M. Seitz , Maneesh Agrawala, Video-based document tracking: unifying your physical and electronic desktops, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029650]
• 17: Brian Knep , Craig Hayes , Rick Sayre , Tom Williams, Dinosaur input device, Proceedings of the SIGCHI conference on Human factors in computing systems, p.304-309, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223943]
• 18: Joseph J. LaViola, Jr. , Robert C. Zeleznik, MathPad²: a system for the creation and exploration of mathematical sketches, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 19: Gun A. Lee , Gerard J. Kim , Mark Billinghurst, Immersive authoring: What You eXperience Is What You Get (WYXIWYG), Communications of the ACM, v.48 n.7, p.76-81, July 2005 [doi>10.1145/1070838.1070840]
• 20: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 21: Milgram, P., and Kishino, F. 1994. A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems, Vol. 77, No. 12, 1321--1329.
• 22: Moscovich, T., and Hughes, J. 2001. Animation Sketching: An Approach to Accessible Animation. Tech. Rep. CS04-03, Brown University CS Department.
• 23: Oore, S., Terzopoulos, D., and Hinton, G. 2002. A Desktop Input Device and Interface for Interactive 3D Character Animation. Graphics Interface, Vol. 2, 133--140.
• 24: Jovan Popović , Steven M. Seitz , Michael Erdmann, Motion sketching for control of rigid-body simulations, ACM Transactions on Graphics (TOG), v.22 n.4, p.1034-1054, October 2003 [doi>10.1145/944020.944025]
• 25: Jun Rekimoto , Yuji Ayatsuka, CyberCode: designing augmented reality environments with visual tags, Proceedings of DARE 2000 on Designing augmented reality environments, p.1-10, April 2000, Elsinore, Denmark [doi>10.1145/354666.354667]
• 26: Daniela Rus , P. deSantis, The Self-Organizing Desk, Dartmouth College, Hanover, NH, 1997
• 27: David J. Sturman, Computer Puppetry, IEEE Computer Graphics and Applications, v.18 n.1, p.38-45, January 1998 [doi>10.1109/38.637269]
• 28: Thomas, F., and Johnston, O. 1981. Disney Animation: The Illusion of Life. Walt Disney Productions, New York.
• 29: Matthew Thorne , David Burke , Michiel van de Panne, Motion doodles: an interface for sketching character motion, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 30: Tomasi, C., and Kanade, T. 1991. Detection and Tracking of Point Features. Tech. Rep. CMU-CS-91-132, Carnegie Mellon University, Pittsburgh, PA, USA.
• 31: Trucco, E., and Plakas, K. 2006. Video Tracking: A Concise Survey. IEEE Journal of Oceanic Engineering, Vol. 31, No. 2, 520--529.
• 32: Jue Wang , Yingqing Xu , Heung-Yeung Shum , Michael F. Cohen, Video tooning, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 33: Jue Wang , Pravin Bhat , R. Alex Colburn , Maneesh Agrawala , Michael F. Cohen, Interactive video cutout, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 34: Jue Wang , Steven M. Drucker , Maneesh Agrawala , Michael F. Cohen, The cartoon animation filter, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 35: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 36: Andrew D. Wilson, PlayAnywhere: a compact interactive tabletop projection-vision system, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095047]
• 37: Holger Winnemöller , Sven C. Olsen , Bruce Gooch, Real-time video abstraction, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 38: Wisniewski, D., and Wisniewski, D. 1996. Worlds of Shadow: Teaching with Shadow Puppetry. Teacher Ideas Press.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Keywords:
animation, real-time, tangible user interface, vision