Video object annotation, navigation, and composition

ABSTRACT

We explore the use of tracked 2D object motion to enable novel approaches to interacting with video. These include moving annotations, video navigation by direct manipulation of objects, and creating an image composite from multiple video frames. Features in the video are automatically tracked and grouped in an off-line preprocess that enables later interactive manipulation. Examples of annotations include speech and thought balloons, video graffiti, path arrows, video hyperlinks, and schematic storyboards. We also demonstrate a direct-manipulation interface for random frame access using spatial constraints, and a drag-and-drop interface for assembling still images from videos. Taken together, our tools can be employed in a variety of applications including film and video editing, visual tagging, and authoring rich media such as hyperlinked video.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Aseem Agarwala , Aaron Hertzmann , David H. Salesin , Steven M. Seitz, Keyframe-based tracking for rotoscoping and animation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 3: ASTAR Learning Systems. http://www.astarls.com, 2006. {Online; accessed 5-January-2008}.
• 4: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 5: Yung-Yu Chuang , Aseem Agarwala , Brian Curless , David H. Salesin , Richard Szeliski, Video matting of complex scenes, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 6: J. Dakss, S. Agamanolis, E. Chalom, and V. M. Bove Jr. Hyperlinked video. In Proc. SPIE, volume 3528, pages 2--10, 1999.
• 7: Pierre Dragicevic , Gonzalo Ramos , Jacobo Bibliowitcz , Derek Nowrouzezahrai , Ravin Balakrishnan , Karan Singh, Video browsing by direct manipulation, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357096]
• 8: Dan B Goldman , Brian Curless , David Salesin , Steven M. Seitz, Schematic storyboarding for video visualization and editing, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 9: J. Goldman. Kind of a Blur. http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewMovie?id=197994758, 2005. {Short film available online; accessed 5-January-2008}.
• 10: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer vision, Cambridge University Press, New York, NY, 2000
• 11: Thorsten Karrer , Malte Weiss , Eric Lee , Jan Borchers, DRAGON: a direct manipulation interface for frame-accurate in-scene video navigation, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357097]
• 12: D. Kimber, T. Dunnigan, A. Girgensohn, F. Shipman, T. Turner, and T. Yang. Trailblazing: Video playback control by direct object manipulation. In ICME, pages 1015--1018, 2007.
• 13: David Kurlander , Tim Skelly , David Salesin, Comic Chat, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.225-236, August 1996 [doi>10.1145/237170.237260]
• 14: Yin Li , Jian Sun , Heung-Yeung Shum, Video object cut and paste, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 15: Chip Morningstar , F. Randall Farmer, The lessons of Lucasfilm's habitat, Cyberspace: first steps, MIT Press, Cambridge, MA, 1991
• 16: Y. Pritch, A. Rav-Acha, A. Gutman, and S. Peleg. Webcam synopsis: Peeking around the world. In Proc. ICCV, pages 1--8, 2007.
• 17: Yael Pritch , Alex Rav-Acha , Shmuel Peleg, Nonchronological Video Synopsis and Indexing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.11, p.1971-1984, November 2008 [doi>10.1109/TPAMI.2008.29]
• 18: Gonzalo Ramos , Ravin Balakrishnan, Fluid interaction techniques for the control and annotation of digital video, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.105-114, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964708]
• 19: Alex Rav-Acha , Yael Pritch , Dani Lischinski , Shmuel Peleg, Dynamosaics: Video Mosaics with Non-Chronological Time, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.58-65, June 20-26, 2005 [doi>10.1109/CVPR.2005.137]
• 20: Alex Rav-Acha , Yael Pritch , Shmuel Peleg, Making a Long Video Short: Dynamic Video Synopsis, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.435-441, June 17-22, 2006 [doi>10.1109/CVPR.2006.179]
• 21: E. Rosten, G. Reitmayr, and T. Drummond. Real-time video annotations for augmented reality. In Proc. Intl. Symp. on Visual Computing, 2005.
• 22: Peter Sand , Seth Teller, Long-range video motion estimation using point trajectories, Massachusetts Institute of Technology, Cambridge, MA, 2006
• 23: Peter Sand , Seth Teller, Particle Video: Long-Range Motion Estimation using Point Trajectories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2195-2202, June 17-22, 2006 [doi>10.1109/CVPR.2006.219]
• 24: Arno Schödl , Irfan A. Essa, Controlled animation of video sprites, Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 21-22, 2002, San Antonio, Texas [doi>10.1145/545261.545281]
• 25: Arno Schödl , Richard Szeliski , David H. Salesin , Irfan Essa, Video textures, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.489-498, July 2000 [doi>10.1145/344779.345012]
• 26: Josef Sivic , Frederik Schaffalitzky , Andrew Zisserman, Object Level Grouping for Video Shots, International Journal of Computer Vision, v.67 n.2, p.189-210, April 2006 [doi>10.1007/s11263-005-4264-y]
• 27: Jason McC. Smith , David Stotts , Sang-Uok Kum, An orthogonal taxonomy for hyperlink anchor generation in video streams using OvalTine, Proceedings of the eleventh ACM on Hypertext and hypermedia, p.11-18, May 30-June 03, 2000, San Antonio, Texas, United States [doi>10.1145/336296.336306]
• 28: Sportvision. Changing The Game. http://www.sportvision.com, 2006. {Online; accessed 5-January-2008}.
• 29: Jue Wang , Pravin Bhat , R. Alex Colburn , Maneesh Agrawala , Michael F. Cohen, Interactive video cutout, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 30: Wikipedia. Telestrator. http://en.wikipedia.org/w/index.php?title=Telestrator&oldid=180785495, 2006. {Online; accessed 5-January-2008}.
• 31: Alper Yilmaz , Omar Javed , Mubarak Shah, Object tracking: A survey, ACM Computing Surveys (CSUR), v.38 n.4, p.13-es, 2006 [doi>10.1145/1177352.1177355]
• 32: Li Zhang , Noah Snavely , Brian Curless , Steven M. Seitz, Spacetime faces: high resolution capture for modeling and animation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Algorithms, Design, Human Factors

Keywords:
direct manipulation, video annotation, video interaction, video navigation