Invertible motion blur in video

ABSTRACT

We show that motion blur in successive video frames is invertible even if the point-spread function (PSF) due to motion smear in a single photo is non-invertible. Blurred photos exhibit nulls (zeros) in the frequency transform of the PSF, leading to an ill-posed deconvolution. Hardware solutions to avoid this require specialized devices such as the coded exposure camera or accelerating sensor motion. We employ ordinary video cameras and introduce the notion of null-filling along with joint-invertibility of multiple blur-functions. The key idea is to record the same object with varying PSFs, so that the nulls in the frequency component of one frame can be filled by other frames. The combined frequency transform becomes null-free, making deblurring well-posed. We achieve jointly-invertible blur simply by changing the exposure time of successive frames. We address the problem of automatic deblurring of objects moving with constant velocity by solving the four critical components: preservation of all spatial frequencies, segmentation of moving parts, motion estimation of moving parts, and non-degradation of the static parts of the scene. We demonstrate several challenging cases of object motion blur including textured backgrounds and partial occluders.

REFERENCES

• 1: Agrawal, A., and Raskar, R. 2007. Resolving Objects at Higher Resolution from a Single Motion-blurred Image. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
• 2: B. Bascle , Andrew Blake , Andrew Zisserman, Motion Deblurring and Super-resolution from an Image Sequence, Proceedings of the 4th European Conference on Computer Vision-Volume II, p.573-582, April 15-18, 1996
• 3: Moshe Ben-Ezra , Shree K. Nayar, Motion-Based Motion Deblurring, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.6, p.689-698, June 2004 [doi>10.1109/TPAMI.2004.1]
• 4: Berenstein, C., and Patrick, E. 1990. Exact Deconvolution for Multiple Convolution Operators-an Overview, plus Performance Characterizations for Imaging Sensors. Proc. of the IEEE 78 (Apr.), 723--734.
• 5: Wei-Ge Chen , N. Nandhakumar , Worthy N. Martin, Image Motion Estimation From Motion Smear-A New Computational Model, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.4, p.412-425, April 1996 [doi>10.1109/34.491622]
• 6: Chen, J., Yuan, L., Tang, C.-K., and Quan, L. 2008. Robust Dual Motion Deblurring. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
• 7: Cho, S., Matsushita, Y., and Lee, S. 2007. Removing Non-Uniform Motion Blur from Images. In Proc. Int'l Conf. Computer Vision, 1--8.
• 8: Dai, S., and Wu, Y. 2008. Motion from Blur. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
• 9: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 10: Dowski, E. R., and Cathey, W. 1995. Extended Depth of Field through Wavefront Coding. Appl. Optics 34, 11 (Apr.), 1859--1866.
• 11: Rob Fergus , Barun Singh , Aaron Hertzmann , Sam T. Roweis , William T. Freeman, Removing camera shake from a single photograph, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 12: Grossberg, M., and Nayar, S. 2003. High Dynamic Range from Multiple Images: Which Exposures to Combine? In ICCV Workshop on Color and Photometric Methods in Computer Vision (CPMCV).
• 13: Peter A. Jansson, Deconvolution of images and spectra (2nd ed.), Academic Press, Inc., Orlando, FL, 1996
• 14: Jia, J. 2007. Single Image Motion Deblurring using Transparency. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
• 15: Joshi, N., Szeliski, R., and Kriegman, D. 2008. PSF Estimation using Sharp Edge Prediction. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
• 16: Anat Levin , Rob Fergus , Frédo Durand , William T. Freeman, Image and depth from a conventional camera with a coded aperture, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 17: Anat Levin , Dani Lischinski , Yair Weiss, A Closed-Form Solution to Natural Image Matting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.2, p.228-242, February 2008 [doi>10.1109/TPAMI.2007.1177]
• 18: Anat Levin , Peter Sand , Taeg Sang Cho , Frédo Durand , William T. Freeman, Motion-invariant photography, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 19: Lucy, L. 1974. An iterative technique for the rectification of observed distributions. J. Astronomy 79, 745--754.
• 20: Mann, S., and Picard, R. W. 1995. Being Undigital with Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures. In Proc. of IS&T 48th Annual Conference, 422--428.
• 21: Hajime Nagahara , Sujit Kuthirummal , Changyin Zhou , Shree K. Nayar, Flexible Depth of Field Photography, Proceedings of the 10th European Conference on Computer Vision: Part IV, October 12-18, 2008, Marseille, France [doi>10.1007/978-3-540-88693-8_5]
• 22: Piccardi, M. 2004. Background Subtraction Techniques: a Review. In Proc. IEEE SMC Intl. Conf. Systems, Man and Cybernetics.
• 23: Ramesh Raskar , Amit Agrawal , Jack Tumblin, Coded exposure photography: motion deblurring using fluttered shutter, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 24: Alex Rav-Acha , Shmuel Peleg, Two motion-blurred images are better than one, Pattern Recognition Letters, v.26 n.3, p.311-317, February 2005 [doi>10.1016/j.patrec.2004.10.017]
• 25: Richardson, W. 1972. Bayesian-based iterative method of image restoration. J. Opt. Soc. of America 62, 1, 55--59.
• 26: Schultz, R. R., and Stevenson, R. L. 1996. Extraction of High-Resolution Frames from Video Sequences. IEEE Trans. Image Processing 5 (jun), 996--1011.
• 27: Sellent, A., Eisemann, M., and Magnor, M. 2008. Calculating Motion Fields from Images with Two Different Exposure Times. Tech. rep., Computer Graphics Lab, Technical University of Braunschweig, 5.
• 28: Qi Shan , Jiaya Jia , Aseem Agarwala, High-quality motion deblurring from a single image, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 29: E. Shechtman , Yaron Caspi , Michal Irani, Increasing Space-Time Resolution in Video, Proceedings of the 7th European Conference on Computer Vision-Part I, p.753-768, May 28-31, 2002
• 30: Tai., W. Y., Hao, D., Brown, M. S., and Lin, S. 2008. Image/Video Deblurring using a Hybrid Camera. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
• 31: Telleen, J., Sullivan, A., Yee, J., Gunawardane, P., Wang, O., Collins, I., and Davis, J. 2007. Synthetic Shutter Speed Imaging. In Proc. Eurographics, 591--598.
• 32: Bennett Wilburn , Neel Joshi , Vaibhav Vaish , Eino-Ville Talvala , Emilio Antunez , Adam Barth , Andrew Adams , Mark Horowitz , Marc Levoy, High performance imaging using large camera arrays, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 33: Lu Yuan , Jian Sun , Long Quan , Heung-Yeung Shum, Image deblurring with blurred/noisy image pairs, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 34: Lu Yuan , Jian Sun , Long Quan , Heung-Yeung Shum, Progressive inter-scale and intra-scale non-blind image deconvolution, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Motion; Video analysis
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.6 Segmentation
          Subjects: Edge and feature detection
      I.4.7 Feature Measurement
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Measurement, Performance

Keywords:
PSF, PSF estimation, computational photography, invertibility, motion deblurring