Video SnapCut: robust video object cutout using localized classifiers

ABSTRACT

Although tremendous success has been achieved for interactive object cutout in still images, accurately extracting dynamic objects in video remains a very challenging problem. Previous video cutout systems present two major limitations: (1) reliance on global statistics, thus lacking the ability to deal with complex and diverse scenes; and (2) treating segmentation as a global optimization, thus lacking a practical workflow that can guarantee the convergence of the systems to the desired results.

We present Video SnapCut, a robust video object cutout system that significantly advances the state-of-the-art. In our system segmentation is achieved by the collaboration of a set of local classifiers, each adaptively integrating multiple local image features. We show how this segmentation paradigm naturally supports local user editing and propagates them across time. The object cutout system is completed with a novel coherent video matting technique. A comprehensive evaluation and comparison is presented, demonstrating the effectiveness of the proposed system at achieving high quality results, as well as the robustness of the system against various types of inputs.

REFERENCES

• 1: Adobe Creative Team, Adobe Photoshop CS4 Classroom in a Book, Adobe Press, 2008
• 2: Aseem Agarwala , Aaron Hertzmann , David H. Salesin , Steven M. Seitz, Keyframe-based tracking for rotoscoping and animation, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 3: Christopher J. Armstrong , Brian L. Price , William A. Barrett, Interactive segmentation of image volumes with Live Surface, Computers and Graphics, v.31 n.2, p.212-229, April, 2007 [doi>10.1016/j.cag.2006.11.015]
• 4: Bai, X., and Sapiro, G. 2007. A geodesic framework for fast interactive image and video segmentation and matting. In Proc. of IEEE ICCV.
• 5: Blake, A., and Isard, M. 1998. Active Contours. Springer-Verlag.
• 6: Yuri Boykov , Olga Veksler , Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.11, p.1222-1239, November 2001 [doi>10.1109/34.969114]
• 7: Hamilton Y. Chong , Steven J. Gortler , Todd Zickler, A perception-based color space for illumination-invariant image processing, ACM SIGGRAPH 2008 papers, August 11-15, 2008, Los Angeles, California
• 8: Yung-Yu Chuang , Aseem Agarwala , Brian Curless , David H. Salesin , Richard Szeliski, Video matting of complex scenes, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 9: Kohli, P., Kumar, M. P., and Torr, P. H. S. 2007. P3 & beyond: solving energies with higher order cliques. In Proc. of IEEE CVPR.
• 10: Oleg Komogortsev , Javed Khan, Predictive perceptual compression for real time video communication, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027577]
• 11: Anat Levin , Dani Lischinski , Yair Weiss, A Closed-Form Solution to Natural Image Matting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.2, p.228-242, February 2008 [doi>10.1109/TPAMI.2007.1177]
• 12: Yin Li , Jian Sun , Chi-Keung Tang , Heung-Yeung Shum, Lazy snapping, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 13: Yin Li , Jian Sun , Heung-Yeung Shum, Video object cut and paste, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 14: Li, Y., Adelson, E., and Agarwala, A. 2008. Scribbleboost: Adding classification to edge-aware interpolation of local image and video adjustments. In Proc. of EGSR, 1255--1264.
• 15: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 16: Eric N. Mortensen , William A. Barrett, Intelligent scissors for image composition, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.191-198, September 1995 [doi>10.1145/218380.218442]
• 17: Protiere, A., and Sapiro, G. 2007. Interactive image segmentation via adaptive weighted distances. IEEE Trans. Image Processing 16, 1046--1057.
• 18: Carsten Rother , Vladimir Kolmogorov , Andrew Blake, "GrabCut": interactive foreground extraction using iterated graph cuts, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 19: Stewart, S., 2003. Confessions of a roto artist: Three rules for better mattes. http://www.pinnaclesys.com/SupportFiles/Rotoscoping.pdf
• 20: Wandell, B. 1995. Foundations of Vision. Sinauer Associates.
• 21: Jue Wang , Michael F. Cohen, Image and video matting: a survey, Foundations and Trends® in Computer Graphics and Vision, v.3 n.2, p.97-175, January 2007 [doi>10.1561/0600000019]
• 22: Wang, J., and Cohen, M. 2007. Optimized color sampling for robust matting. In Proc. of IEEE CVPR.
• 23: Jue Wang , Yingqing Xu , Heung-Yeung Shum , Michael F. Cohen, Video tooning, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 24: Jue Wang , Pravin Bhat , R. Alex Colburn , Maneesh Agrawala , Michael F. Cohen, Interactive video cutout, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 25: Jue Wang , Maneesh Agrawala , Michael F. Cohen, Soft scissors: an interactive tool for realtime high quality matting, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 26: Michael Waschbüsch , Stephan Würmlin , Markus Gross, Interactive 3D video editing, The Visual Computer: International Journal of Computer Graphics, v.22 n.9, p.631-641, September 2006 [doi>10.1007/s00371-006-0053-z]
• 27: Ting Yu , Cha Zhang , Michael Cohen , Yong Rui , Ying Wu, Monocular Video Foreground/Background Segmentation by Tracking Spatial-Color Gaussian Mixture Models, Proceedings of the IEEE Workshop on Motion and Video Computing, p.5, February 23-24, 2007 [doi>10.1109/WMVC.2007.27]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.6 Segmentation
          Subjects: Edge and feature detection
      I.4.7 Feature Measurement
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection

General Terms:
Algorithms, Design, Measurement