Extracting depth and matte using a color-filtered aperture

ABSTRACT

This paper presents a method for automatically extracting a scene depth map and the alpha matte of a foreground object by capturing a scene through RGB color filters placed in the camera lens aperture. By dividing the aperture into three regions through which only light in one of the RGB color bands can pass, we can acquir three shifted views of a scene in the RGB planes of an image in a single exposure. In other words, a captured image has depth-dependent color misalignment. We develop a color alignment measure to estimate disparities between the RGB planes for depth reconstruction. We also exploit color misalignment cues in our matting algorithm in order to disambiguate between the foreground and background regions even where their colors are similar. Based on the extracted depth and matte, the color misalignment in the captured image can be canceled, and various image editing operations can be applied to the reconstructed image, including novel view synthesis, postexposure refocusing, and composition over different backgrounds.

REFERENCES

• 1: Edward H. Adelson , John Y. A. Wang, Single Lens Stereo with a Plenoptic Camera, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.14 n.2, p.99-106, February 1992 [doi>10.1109/34.121783]
• 2: Amari, Y., and Adelson, E. H. 1992. Single-eye range estimation by using displaced apertures with color filters. In Proc. Int. Conf. Industrial Electronics, Control, Instrumentation, and Automation, vol. 3, 1588--1592.
• 3: Yosuke Bando , Tomoyuki Nishita, Towards Digital Refocusing from a Single Photograph, Proceedings of the 15th Pacific Conference on Computer Graphics and Applications, p.363-372, October 29-November 02, 2007 [doi>10.1109/PG.2007.64]
• 4: Yuri Boykov , Olga Veksler , Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.11, p.1222-1239, November 2001 [doi>10.1109/34.969114]
• 5: Chang, I.-C., Huang, C.-L., Hsueh, W.-J., Lin, H.-C., Chen, C.-C., and Yeh, Y.-H. 2002. A novel 3-D hand-held camera based on tri-aperture lens. In Proc. SPIE 4925, 655--662.
• 6: Hang Chang , Qing Yang , Chunhong Pan, An Iterative Bayesian Approach for Digital Matting, Proceedings of the 18th International Conference on Pattern Recognition, p.122-125, August 20-24, 2006 [doi>10.1109/ICPR.2006.259]
• 7: Yung-Yu Chuang , Brian Curless , David H. Salesin, New models and methods for matting and compositing, University of Washington, 2004
• 8: Georgeiv, T., Zheng, K. C., Curless, B., Salesin, D., Nayar, S., and Intwala, C. 2006. Spatio-angular resolution tradeoff in integral photography. In Proc. EGSR, 263--272.
• 9: Gradshteyn, I. S., and Ryzhik, I. M. 2000. Table of integrals, series, and products (sixth edition). Academic Press.
• 10: Leo Grady, Random Walks for Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.11, p.1768-1783, November 2006 [doi>10.1109/TPAMI.2006.233]
• 11: Paul Green , Wenyang Sun , Wojciech Matusik , Frédo Durand, Multi-aperture photography, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 12: Neel Joshi , Wojciech Matusik , Shai Avidan, Natural video matting using camera arrays, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 13: Junhwan Kim , Vladimir Kolmogorov , Ramin Zabih, Visual Correspondence Using Energy Minimization and Mutual Information, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1033, October 13-16, 2003
• 14: Anat Levin , Rob Fergus , Frédo Durand , William T. Freeman, Image and depth from a conventional camera with a coded aperture, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 15: Anat Levin , Dani Lischinski , Yair Weiss, A Closed-Form Solution to Natural Image Matting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.2, p.228-242, February 2008 [doi>10.1109/TPAMI.2007.1177]
• 16: Lewis, J. P. 1995. Fast template matching. In Proc. Vision Interface, 120--123.
• 17: Chia-Kai Liang , Tai-Hsu Lin , Bing-Yi Wong , Chi Liu , Homer H. Chen, Programmable aperture photography: multiplexed light field acquisition, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 18: Morgan McGuire , Wojciech Matusik , Hanspeter Pfister , John F. Hughes , Frédo Durand, Defocus video matting, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 19: McGuire, M., Matusik, W., and Yerazunis, W. 2006. Practical, real-time studio matting using dual imagers. In Proc. EGSR, 235--244.
• 20: Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P., 2005. Light field photography with a hand-held plenoptic camera. Tech. Rep. CSTR 2005-02, Stanford Computer Science, Apr.
• 21: Ren Ng, Fourier slice photography, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 22: Omer, I., and Werman, M. 2004. Color lines: image specific color representation. In Proc. CVPR, vol. 2, 946--953.
• 23: Raskar, R., Tumblin, J., Mohan, A., Agrawal, A., and Li, Y. 2006. Computational photography. In Proc. Eurographics STAR.
• 24: Alvy Ray Smith , James F. Blinn, Blue screen matting, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.259-268, August 1996 [doi>10.1145/237170.237263]
• 25: Ashok Veeraraghavan , Ramesh Raskar , Amit Agrawal , Ankit Mohan , Jack Tumblin, Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 26: Vlahos, P., 1971. Electronic composite photography. U. S. Patent 3,595,987.
• 27: Wang, J., and Cohen, M. F. 2007. Optimized color sampling for robust matting. In Proc. CVPR.
• 28: Yonatan Wexler , Andrew W. Fitzgibbon , Andrew Zisserman, Bayesian Estimation of Layers from Multiple Images, Proceedings of the 7th European Conference on Computer Vision-Part III, p.487-501, May 28-31, 2002
• 29: Xiong, W., and Jia, J. 2007. Stereo matching on objects with fractional boundary. In Proc. CVPR.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Depth cues

General Terms:
Design, Performance

Keywords:
alpha matting, color correlation, color filters, computational camera, computational photography, depth estimation