Video textures

ABSTRACT

This paper introduces a new type of medium, called a video texture, which has qualities somewhere between those of a photograph and a video. A video texture provides a continuous infinitely varying stream of images. While the individual frames of a video texture may be repeated from time to time, the video sequence as a whole is never repeated exactly. Video textures can be used in place of digital photos to infuse a static image with dynamic qualities and explicit actions. We present techniques for analyzing a video clip to extract its structure, and for synthesizing a new, similar looking video of arbitrary length. We combine video textures with view morphing techniques to obtain 3D video textures. We also introduce video-based animation, in which the synthesis of video textures can be guided by a user through high-level interactive controls. Applications of video textures and their extensions include the display of dynamic scenes on web pages, the creation of dynamic backdrops for special effects and games, and the interactive control of video-based animation.

REFERENCES

• 1: Z. Bar-Joseph. Statistical learning of multi-dimensional textures. Master's thesis, The Hebrew University of Jerusalem, June 1999.
• 2: David Beymer, Feature Correspondence by Interleaving Shape and Texture Computations, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.921, June 18-20, 1996
• 3: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 4: Peter J. Burt , Edward H. Adelson, A multiresolution spline with application to image mosaics, ACM Transactions on Graphics (TOG), v.2 n.4, p.217-236, October 1983 [doi>10.1145/245.247]
• 5: Shenchang Eric Chen, QuickTime VR: an image-based approach to virtual environment navigation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.29-38, September 1995 [doi>10.1145/218380.218395]
• 6: Jeremy S. De Bonet, Multiresolution sampling procedure for analysis and synthesis of texture images, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.361-368, August 1997 [doi>10.1145/258734.258882]
• 7: Paul Debevec, Image-based modeling, rendering, and lighting in Fiat Lux, ACM SIGGRAPH 99 Conference abstracts and applications, p.249, August 08-13, 1999, Los Angeles, California, United States [doi>10.1145/311625.312148]
• 8: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 9: Alexei A. Efros , Thomas K. Leung, Texture Synthesis by Non-Parametric Sampling, Proceedings of the International Conference on Computer Vision-Volume 2, p.1033, September 20-25, 1999
• 10: Adam Finkelstein , Charles E. Jacobs , David H. Salesin, Multiresolution video, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.281-290, August 1996 [doi>10.1145/237170.237266]
• 11: William T. Freeman , Edward H. Adelson , David J. Heeger, Motion without movement, Proceedings of the 18th annual conference on Computer graphics and interactive techniques, p.27-30, July 1991
• 12: David J. Heeger , James R. Bergen, Pyramid-based texture analysis/synthesis, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.229-238, September 1995 [doi>10.1145/218380.218446]
• 13: Workshop on Image-Based Modeling and Rendering, Stanford University, March 1998. http://graphics.stanford.edu/workshops/ibr98/.
• 14: Charles E. Jacobs , Adam Finkelstein , David H. Salesin, Fast multiresolution image querying, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.277-286, September 1995 [doi>10.1145/218380.218454]
• 15: L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 1996.
• 16: Takeo Kanade , Peter Rander , P. J. Narayanan, Virtualized Reality: Constructing Virtual Worlds from Real Scenes, IEEE MultiMedia, v.4 n.1, p.34-47, January 1997 [doi>10.1109/93.580394]
• 17: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 18: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• 19: Saied Moezzi , Arun Katkere , Don Y. Kuramura , Ramesh Jain, Reality Modeling and Visualization from Multiple Video Sequences, IEEE Computer Graphics and Applications, v.16 n.6, p.58-63, November 1996 [doi>10.1109/38.544073]
• 20: S. A. Niyogi and E. H. Adelson. Analyzing and recognizing walking figures in xyt. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'94), pages 469-474, Seattle, Washington, June 1994.
• 21: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 22: Ramprasad Polana , Randal C. Nelson, Detection and Recognition of Periodic, Nonrigid Motion, International Journal of Computer Vision, v.23 n.3, p.261-282, June/July 1997 [doi>10.1023/A:1007975200487]
• 23: Stephen Pollard , Maurizio Pilu , Sean Hayes , Adele Lorusso, View Synthesis by Trinocular Edge Matching and Transfer, Proceedings of the 4th IEEE Workshop on Applications of Computer Vision (WACV'98), p.168, October 19-21, 1998
• 24: Steven M. Seitz , Charles R. Dyer, View morphing, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.21-30, August 1996 [doi>10.1145/237170.237196]
• 25: Steven M. Seitz , Charles R. Dyer, View-Invariant Analysis of Cyclic Motion, International Journal of Computer Vision, v.25 n.3, p.231-251, Dec. 1997 [doi>10.1023/A:1007928103394]
• 26: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• 27: Jonathan Shade , Dani Lischinski , David H. Salesin , Tony DeRose , John Snyder, Hierarchical image caching for accelerated walkthroughs of complex environments, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.75-82, August 1996 [doi>10.1145/237170.237209]
• 28: B. Shaw. Light of other days. In Other Days, Other Eyes. Ace Books, NewYork, 1972. (also published in Analog and various sci-fi anthologies).
• 29: Heung-Yeung Shum , Richard Szeliski, Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment, International Journal of Computer Vision, v.36 n.2, p.101-130, Feb. 2000 [doi>10.1023/A:1008195814169]
• 30: Alvy Ray Smith , James F. Blinn, Blue screen matting, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.259-268, August 1996 [doi>10.1145/237170.237263]
• 31: Richard Szeliski , Heung-Yeung Shum, Creating full view panoramic image mosaics and environment maps, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.251-258, August 1997 [doi>10.1145/258734.258861]
• 32: Jay Torborg , James T. Kajiya, Talisman: commodity realtime 3D graphics for the PC, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.353-363, August 1996 [doi>10.1145/237170.237274]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Animations

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Texture
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation
          Subjects: Display algorithms
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation; Color, shading, shadowing, and texture
  I.4 IMAGE PROCESSING AND COMPUTER VISION
        I.4.10 Image Representation
            Subjects: Morphological
      I.4.7 Feature Measurement
          Subjects: Texture

General Terms:
Algorithms, Design, Performance, Theory

Keywords:
animation, image-based rendering, morphing, multimedia, natural phenomena, texture synthesis, video sprites, video-based animation, video-based rendering, view morphing