Video completion via motion guided spatial-temporal global optimization

ABSTRACT

In this paper, a novel global optimization based approach is proposed for video completion whose target is to restore the spatial-temporal missing regions of a video in a visually plausible way. Our algorithm consists of two stages: motion field completion and color completion via global optimization. First, local motions within the missing parts are completed patch-by-patch greedily using pre-computed available motions in the video. Then the missing regions are filled by sampling patches from available parts of the video. We formulate the video completion as a global energy minimization problem by Markov random fields (MRFs). Based on the completed motion field of the video, a well-defined energy function involving both spatial and temporal coherence relationship is constructed. A coarse-to-fine Belief Propagation (BP) is proposed to solve the optimization problem. Experimental results have demonstrated the good performance of our algorithm.

REFERENCES

• 1: M. Bertalm1o, A. Bertozzi, and G. Sapiro. Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting. CVPR, pages 355--362, 2001.
• 2: M. Bertalm1o, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. SIGGRAPH, pages 417--424, 2000.
• 3: A. Criminisi, P. Perez, and K. Toyama. Region filling and object removal by exemplar--based image inpainting. IEEE Trans. on Image Processing, pages 1200--1212, 2004.
• 4: A. Efros and T. Leung. Texture synthesis by non-parametric sampling. ICCV, pages 1033--1038, 1999.
• 5: P. Felzenszwalb and D. Huttenlocher. Efficient Belief Propagation for Early Vision. IJCV, pages 41--54, 2006.
• 6: T. Huang, S. Chen, J. Liu, and X. Tang. Image inpainting by global structure and texture propagation. ACM MM, 2007.
• 7: J. Jia, T. Wu, Y. Tai, and C. Tang. Video repairing: Inference of foreground and background under severe occlusion. CVPR, pages 364--371, 2004.
• 8: N. Komodakis and G. Tziritas. Image completion using global optimization. CVPR, pages 442--452, 2006.
• 9: K. Patwardhan, G. Sapiro, and M. Bertalmio. Video Inpainting Under Constrained Camera Motion. IEEE Trans. on Image Processing, pages 545--553, 2007.
• 10: Y. Shen, F. Lu, X. Cao, and H. Foroosh. Video Completion for Perspective Camera Under Constrained Motion. ICPR, pages 63--66, 2006.
• 11: J. Shi and C. Tomasi. Good features to track. CVPR, 1994.
• 12: T. Shiratori, Y. Matsushita, S. Kang, and X. Tang. Video Completion by Motion Field Transfer. CVPR, 2006.
• 13: J. Sun, L. Yuan, J. Jia, and H. Shum. Image completion with structure propagation. SIGGRAPH, pages 861--868, 2005.
• 14: Y. Wexler, E. Shechtman, and M. Irani. Space-time video completion. CVPR, 2004.
• 15: Y. Zhang, J. Xiao, and M. Shah. Motion Layer Based Object Removal in Videos. WACV/MOTIONS, 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Algorithms, Experimentation

Keywords:
belief propagation, motion, video completion