Image deblurring with blurred/noisy image pairs

ABSTRACT

Taking satisfactory photos under dim lighting conditions using a hand-held camera is challenging. If the camera is set to a long exposure time, the image is blurred due to camera shake. On the other hand, the image is dark and noisy if it is taken with a short exposure time but with a high camera gain. By combining information extracted from both blurred and noisy images, however, we show in this paper how to produce a high quality image that cannot be obtained by simply denoising the noisy image, or deblurring the blurred image alone.

Our approach is image deblurring with the help of the noisy image. First, both images are used to estimate an accurate blur kernel, which otherwise is difficult to obtain from a single blurred image. Second, and again using both images, a residual deconvolution is proposed to significantly reduce ringing artifacts inherent to image deconvolution. Third, the remaining ringing artifacts in smooth image regions are further suppressed by a gain-controlled deconvolution process. We demonstrate the effectiveness of our approach using a number of indoor and outdoor images taken by off-the-shelf hand-held cameras in poor lighting environments.

REFERENCES

• 1: Bardsley, J., Jefferies, S., Nagy, J., and Plemmons, R. 2006. Blind iterative restoration of images with spatially-varying blur. In Optics Express, 1767--1782.
• 2: B. Bascle , Andrew Blake , Andrew Zisserman, Motion Deblurring and Super-resolution from an Image Sequence, Proceedings of the 4th European Conference on Computer Vision-Volume II, p.573-582, April 15-18, 1996
• 3: Ben-Ezra, M., and Nayar, S. K. 2003. Motion deblurring using hybrid imaging. In Processings of CVPR, vol. I, 657--664.
• 4: Eric P. Bennett , Leonard McMillan, Video enhancement using per-pixel virtual exposures, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 5: Antoni Buades , Bartomeu Coll , Jean-Michel Morel, A Non-Local Algorithm for Image Denoising, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.60-65, June 20-26, 2005 [doi>10.1109/CVPR.2005.38]
• 6: J Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.8 n.6, p.679-698, Nov. 1986 [doi>10.1109/TPAMI.1986.4767851]
• 7: Caron, J. N., M., N. N., and J., R. C. 2002. Noniterative blind data restoration by use of an extracted filter function. Applied optics (Appl. opt.) 41, 32, 68--84.
• 8: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 9: Frédo Durand , Julie Dorsey, Fast bilateral filtering for the display of high-dynamic-range images, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 10: Elmar Eisemann , Frédo Durand, Flash photography enhancement via intrinsic relighting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 11: Engl, H. W., Hanke, M., and Neubauer, A. 2000. Regularization of Inverse Problems. Kluwer Academic.
• 12: Raanan Fattal , Dani Lischinski , Michael Werman, Gradient domain high dynamic range compression, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 13: Rob Fergus , Barun Singh , Aaron Hertzmann , Sam T. Roweis , William T. Freeman, Removing camera shake from a single photograph, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 14: Donald Geman , George Reynolds, Constrained Restoration and the Recovery of Discontinuities, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.14 n.3, p.367-383, March 1992 [doi>10.1109/34.120331]
• 15: H. Richardson, W. 1972. Bayesian-based iterative method of image restoration. JOSA, A 62, 1, 55--59.
• 16: Jalobeanu, A., Blanc-Feraud, L., and Zerubia, J. 2002. Estimation of blur and noise parameters in remote sensing. In Proceedings of ICASSP, 249--256.
• 17: Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Bayesian correction of image intensity with spatial consideration. In Proceedings of ECCV, 342--354.
• 18: Kundur, D., and Hatzinakos, D. 1996. Blind image deconvolution. IEEE Signal Processing Magazine. 13, 3, 43--64.
• 19: Levin, A. 2006. Blind motion deblurring using image statistics. In Advances in Neural Information Processing Systems (NIPS).
• 20: Yuanzhen Li , Lavanya Sharan , Edward H. Adelson, Compressing and companding high dynamic range images with subband architectures, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 21: Lim, S. H., and Silverstein, D. A. 2006. Method for deblurring an image. US Patent Application, Pub. No. US2006/0187308 A1, Aug 24, 2006.
• 22: Xinqiao Liu , A. El Gamal, Simultaneous image formation and motion blur restoration via multiple capture, Proceedings of the Acoustics, Speech, and Signal Processing, 2001. on IEEE International Conference, p.1841-1844, May 07-11, 2001 [doi>10.1109/ICASSP.2001.941301]
• 23: Ce Liu , William T. Freeman , Richard Szeliski , Sing Bing Kang, Noise Estimation from a Single Image, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.901-908, June 17-22, 2006 [doi>10.1109/CVPR.2006.207]
• 24: Neelamani, R., Choi, H., and Baraniuk, R. 2004. ForWaRd: Fourier-wavelet regularized deconvolution for ill-conditioned systems. IEEE Trans. on Signal Processing 52, 2, 418--433.
• 25: Nikon. 2005. http://www.nikon.co.jp/main/eng/portfolio/about/technology/nikon_technology/vr_e/index.htm.
• 26: P. Perona , J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.7, p.629-639, July 1990 [doi>10.1109/34.56205]
• 27: Georg Petschnigg , Richard Szeliski , Maneesh Agrawala , Michael Cohen , Hugues Hoppe , Kentaro Toyama, Digital photography with flash and no-flash image pairs, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 28: Portilla, J., Strela, V., Wainwright, M., and Simoncelli., E. P. 2003. Image denoising using scale mixtures of gaussians in the wavelet domain. IEEE Trans. on Image Processing 12, 11, 1338--1351.
• 29: Ramesh Raskar , Amit Agrawal , Jack Tumblin, Coded exposure photography: motion deblurring using fluttered shutter, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 30: Rav-Acha, A., and Peleg, S. 2000. Restoration of multiple images with motion blur in different directions. IEEE Workshop on Applications of Computer Vision.
• 31: Alex Rav-Acha , Shmuel Peleg, Two motion-blurred images are better than one, Pattern Recognition Letters, v.26 n.3, p.311-317, February 2005 [doi>10.1016/j.patrec.2004.10.017]
• 32: Reeves, S. J., and Mersereau, R. M. 1992. Blur identification by the method of generalized cross-validation. IEEE Trans. on Image Processing. 1, 3, 301--311.
• 33: Stefan Roth , Michael J. Black, Fields of Experts: A Framework for Learning Image Priors, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.860-867, June 20-26, 2005 [doi>10.1109/CVPR.2005.160]
• 34: Leonid I. Rudin , Stanley Osher , Emad Fatemi, Nonlinear total variation based noise removal algorithms, Physica D, v.60 n.1-4, p.259-268, Nov. 1, 1992
• 35: Simoncelli, E. P., and Adelson, E. H. 1996. Noise removal via bayesian wavelet coring. In Proceedings of ICIP, vol. I, 379--382.
• 36: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 37: Y. Yitzhaky, I. Mor, A. L., and Kopeika., N. 1998. Direct method for restoration of motion blurred images. J. Opt. Soc. Am., A 15, 6, 1512--1519.
• 38: Zarowin, C. B. 1994. Robust, noniterative, and computationally efficient modification of vab cittert deconvolution optical figuring. JOSA, A 11, 10, 2571--2583.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.6 Segmentation
          Subjects: Region growing, partitioning
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Pattern analysis

General Terms:
Algorithms, Design