Projection defocus analysis for scene capture and image display

ABSTRACT

In order to produce bright images, projectors have large apertures and hence narrow depths of field. In this paper, we present methods for robust scene capture and enhanced image display based on projection defocus analysis. We model a projector's defocus using a linear system. This model is used to develop a novel temporal defocus analysis method to recover depth at each camera pixel by estimating the parameters of its projection defocus kemel in frequency domain. Compared to most depth recovery methods, our approach is more accurate near depth discontinuities. Furthermore, by using a coaxial projector-camera system, we ensure that depth is computed at all camera pixels, without any missing parts. We show that the recovered scene geometry can be used for refocus synthesis and for depth-based image composition. Using the same projector defocus model and estimation technique, we also propose a defocus compensation method that filters a projection image in a spatially-varying, depth-dependent manner to minimize its defocus blur after it is projected onto the scene. This method effectively increases the depth of field of a projector without modifying its optics. Finally, we present an algorithm that exploits projector defocus to reduce the strong pixelation artifacts produced by digital projectors, while preserving the quality of the projected image. We have experimentally verified each of our methods using real scenes.

REFERENCES

• 1: Bimber, O., and Emmerling, A. 2006. Multi-focal projection. IEEE Trans. on Visualization and Computer Graphics to appear.
• 2: Oliver Bimber , Gordon Wetzstein , Andreas Emmerling , Christian Nitschke, Enabling View-Dependent Stereoscopic Projection in Real Environments, Proceedings of the Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality, p.14-23, October 05-08, 2005 [doi>10.1109/ISMAR.2005.27]
• 3: B. Curless , M. Levoy, Better optical triangulation through spacetime analysis, Proceedings of the Fifth International Conference on Computer Vision, p.987, June 20-23, 1995
• 4: James Davis , Diego Nehab , Ravi Ramamoorthi , Szymon Rusinkiewicz, Spacetime Stereo: A Unifying Framework for Depth from Triangulation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.2, p.296-302, February 2005 [doi>10.1109/TPAMI.2005.37]
• 5: Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, Cambridge, MA, 1993
• 6: Paolo Favaro , Stefano Soatto, A Geometric Approach to Shape from Defocus, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.3, p.406-417, March 2005 [doi>10.1109/TPAMI.2005.43]
• 7: A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.814-821, June 20-26, 2005 [doi>10.1109/CVPR.2005.41]
• 8: Girod, B., and Scherock, S. 1989. Depth from defocus of structured light. In Proc. SPIE Conf. on Optics, Illumination, and Image Sensing for Machine Vision.
• 9: Gonzales-Banos, H., and Davis, J. 2004. A method for computing depth under ambient illumination using multi-shuttered light. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 234--241.
• 10: Grossberg, M., Peri, H., Nayar, S., and Belhumeur, P. 2004. Making One Object Look Like Another: Controlling Appearance using a Projector-Camera System. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. I, 452--459.
• 11: Berthold K. P. Horn , Michael J. Brooks, Shape from shading, MIT Press, Cambridge, MA, 1989
• 12: Huang, P. S., Zhang, C. P., and Chiang, F. P. 2003. High speed 3-d shape measurement based on digital fringe projection. Optical Engineering 42, 1, 163--168.
• 13: Anil K. Jain, Fundamentals of digital image processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1989
• 14: Hailin Jin , Paolo Favaro, A Variational Approach to Shape from Defocus, Proceedings of the 7th European Conference on Computer Vision-Part II, p.18-30, May 28-31, 2002
• 15: Kanade, T., Gruss, A., and Carley, L. 1991. A very fast vlsi rangefinder. In Proc. Int. Conf. on Robotics and Automation, vol. 39, 1322--1329.
• 16: Thomas P. Koninckx , Pieter Peers , Philip Dutre , Luc Van Gool, Scene-Adapted Structured Light, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.611-618, June 20-26, 2005 [doi>10.1109/CVPR.2005.311]
• 17: Levoy, M., Chen, B., Vaish, V., Horowitz, M., Mcdowall, I., and Bolas, M. 2004. Synthetic aperture confocal imaging. In SIGGRAPH Conference Proceedings, 825--834.
• 18: Lennart Ljung, System identification: theory for the user, Prentice-Hall, Inc., Upper Saddle River, NJ, 1986
• 19: Majumder, A., and Welch, G. 2001. Computer graphics optique: Optical superposition of projected computer graphics. In Proc. Eurographics Workshop on Virtual Enviroment/Immersive Projection Technology.
• 20: Mcguire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. In SIGGRAPH Conference Proceedings, 567--576.
• 21: S. K. Nayar , Y. Nakagawa, Shape from Focus, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.16 n.8, p.824-831, August 1994 [doi>10.1109/34.308479]
• 22: Shree K. Nayar , Masahiro Watanabe , Minori Noguchi, Real-Time Focus Range Sensor, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.12, p.1186-1198, December 1996 [doi>10.1109/34.546256]
• 23: Nocedal, J., and Wright, S. J. 1999. Numerical Optimization. Springer.
• 24: Alan V. Oppenheim , Alan S. Willsky , S. Hamid Nawab, Signals & systems (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1996
• 25: A. P. Pentland, A new sense for depth of field, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.9 n.4, p.523-531, July 1987 [doi>10.1109/TPAMI.1987.4767940]
• 26: Ashish Raj , Ramin Zabih, A Graph Cut Algorithm for Generalized Image Deconvolution, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1048-1054, October 17-20, 2005 [doi>10.1109/ICCV.2005.8]
• 27: A. N. Rajagopalan , S. Chaudhuri, A Variational Approach to Recovering Depth From Defocused Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.10, p.1158-1164, October 1997 [doi>10.1109/34.625126]
• 28: Ramesh Raskar , Greg Welch , Matt Cutts , Adam Lake , Lev Stesin , Henry Fuchs, The office of the future: a unified approach to image-based modeling and spatially immersive displays, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.179-188, July 1998 [doi>10.1145/280814.280861]
• 29: Raskar, R., Welch, G., Low, K., and Bandyopadhyay, D. 2001. Shader lamps. In Proc. Eurographics Workshop on Rendering.
• 30: Raskar, R., Van Baar, J., Beardsley, P., Willwacher, T., Rao, S., and Forlines, C. 2003. ilamps: geometrically aware and self-configuring projectors. In SIGGRAPH Conference Proceedings, 809--818.
• 31: Raskar, R., Han Tan, K., Feris, R., Yu, J., and Turk, M. 2004. Non-photorealistic camera: Depth edge detection and stylized rendering using multi-flash imaging. In SIGGRAPH Conference Proceedings, 679--688.
• 32: Scharstein, D., and Szeliski, R. 2003. High-accuracy stereo depth maps using structured light. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 195--202.
• 33: Yoav Y. Schechner , Nahum Kiryati , Ronen Basri, Separation of Transparent Layers using Focus, International Journal of Computer Vision, v.39 n.1, p.25-39, Aug. 2000 [doi>10.1023/A:1008166017466]
• 34: Sen, P., Chen, B., Garg, G., Marschner, S. R., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. In SIGGRAPH Conference Proceedings, 745--755.
• 35: Tappen, M. F., Russell, B. C., and Freeman, W. T. 2004. Efficient graphical models for processing images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, vol. 2, 673--680.
• 36: Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In ACM Annual Conference on Computer Graphics, 548--558.
• 37: Zhengyou Zhang, A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.11, p.1330-1334, November 2000 [doi>10.1109/34.888718]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

Keywords:
depth recovery, image composition, multi-focal projection, projector defocus, projector depixelation, refocus synthesis, temporal defocus analysis