Real-time video abstraction

ABSTRACT

We present an automatic, real-time video and image abstraction framework that abstracts imagery by modifying the contrast of visually important features, namely luminance and color opponency. We reduce contrast in low-contrast regions using an approximation to anisotropic diffusion, and artificially increase contrast in higher contrast regions with difference-of-Gaussian edges. The abstraction step is extensible and allows for artistic or data-driven control. Abstracted images can optionally be stylized using soft color quantization to create cartoon-like effects with good temporal coherence. Our framework design is highly parallel, allowing for a GPU-based, real-time implementation. We evaluate the effectiveness of our abstraction framework with a user-study and find that participants are faster at naming abstracted faces of known persons compared to photographs. Participants are also better at remembering abstracted images of arbitrary scenes in a memory task.

REFERENCES

• 1: Arad, N., and Gotsman, C. 1999. Enhancement by image-dependent warping. IEEE Trans. on Image Processing 8, 9, 1063--1074.
• 2: Barash, D., and Comaniciu, D. 2004. A common framework for non-linear diffusion, adaptive smoothing, bilateral filtering and mean shift. Image and Video Computing 22, 1, 73--81.
• 3: Rein van den Boomgaard , Joost van de Weijer, On the Equivalence of Local-Mode Finding, Robust Estimation and Mean-Shift Analysis as Used in Early Vision Tasks, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, p.30927, August 11-15, 2002
• 4: J Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.8 n.6, p.679-698, Nov. 1986 [doi>10.1109/TPAMI.1986.4767851]
• 5: John P. Collomosse , David Rowntree , Peter M. Hall, Stroke Surfaces: Temporally Coherent Artistic Animations from Video, IEEE Transactions on Visualization and Computer Graphics, v.11 n.5, p.540-549, September 2005 [doi>10.1109/TVCG.2005.85]
• 6: Doug DeCarlo , Anthony Santella, Stylization and abstraction of photographs, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 7: James H. Elder, Are Edges Incomplete?, International Journal of Computer Vision, v.34 n.2-3, p.97-122, Nov. 1999 [doi>10.1023/A:1008183703117]
• 8: Jan Fischer , Dirk Bartz, Stylized Augmented Reality for Improved Immersion, Proceedings of the 2005 IEEE Conference 2005 on Virtual Reality, p.195-202, 325, March 12-16, 2005 [doi>10.1109/VR.2005.71]
• 9: Bruce Gooch , Erik Reinhard , Amy Gooch, Human facial illustrations: Creation and psychophysical evaluation, ACM Transactions on Graphics (TOG), v.23 n.1, p.27-44, January 2004 [doi>10.1145/966131.966133]
• 10: Aaron Hertzmann, Paint By Relaxation, Computer Graphics International 2001, p.47-54, July 03-06, 2001
• 11: Itti, L., and Koch, C. 2001. Computational modeling of visual attention. Nature Reviews Neuroscience 2, 3, 194--203.
• 12: Loviscach, J. 1999. Scharfzeichner: Klare bilddetails durch verformung. Computer Technik 22, 236ff.
• 13: Marr, D., and Hildreth, E. C. 1980. Theory of edge detection. Proc. Royal Soc. London, Bio. Sci. 207, 187--217.
• 14: Palmer, S. E. 1999. Vision Science: Photons to Phenomenology. The MIT Press.
• 15: P. Perona , J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.7, p.629-639, July 1990 [doi>10.1109/34.56205]
• 16: Pham, T. Q., and Vliet, L. J. V. 2005. Separable bilateral filtering for fast video preprocessing. In IEEE Internat. Conf. on Multimedia & Expo, CD1-4.
• 17: Claudio M. Privitera , Lawrence W. Stark, Algorithms for Defining Visual Regions-of-Interest: Comparison with Eye Fixations, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.9, p.970-982, September 2000 [doi>10.1109/34.877520]
• 18: Ramesh Raskar , Kar-Han Tan , Rogerio Feris , Jingyi Yu , Matthew Turk, Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 19: Takafumi Saito , Tokiichiro Takahashi, Comprehensible rendering of 3-D shapes, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.197-206, September 1990, Dallas, TX, USA
• 20: Anthony Santella , Doug DeCarlo, Visual interest and NPR: an evaluation and manifesto, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987669]
• 21: Stevenage, S. V. 1995. Can caricatures really produce distinctiveness effects? British Journal of Psychology 86, 127--146.
• 22: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 23: Jue Wang , Yingqing Xu , Heung-Yeung Shum , Michael F. Cohen, Video tooning, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 24: Georges Winkenbach , David H. Salesin, Computer-generated pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.91-100, July 1994 [doi>10.1145/192161.192184]
• 25: Wyszecki, G., and Styles, W. 1982. Color Science: Concepts and Methods, Quantitative Data and Formulae. Wiley, New York, NY.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Keywords:
image abstraction, non-photorealistic rendering, visual communication, visual perception