Light field transfer: global illumination between real and synthetic objects

ABSTRACT

We present a novel image-based method for compositing real and synthetic objects in the same scene with a high degree of visual realism. Ours is the first technique to allow global illumination and near-field lighting effects between both real and synthetic objects at interactive rates, without needing a geometric and material model of the real scene. We achieve this by using a light field interface between real and synthetic components---thus, indirect illumination can be simulated using only two 4D light fields, one captured from and one projected onto the real scene. Multiple bounces of interreflections are obtained simply by iterating this approach. The interactivity of our technique enables its use with time-varying scenes, including dynamic objects. This is in sharp contrast to the alternative approach of using 6D or 8D light transport functions of real objects, which are very expensive in terms of acquisition and storage and hence not suitable for real-time applications. In our method, 4D radiance fields are simultaneously captured and projected by using a lens array, video camera, and digital projector. The method supports full global illumination with restricted object placement, and accommodates moderately specular materials. We implement a complete system and show several example scene compositions that demonstrate global illumination effects between dynamic real and synthetic objects. Our implementation requires a single point light source and dark background.

REFERENCES

• 1: Arnaldi, B., Pueyo, X., and Vilaplana, J. 1991. On the division of environments by virtual walls for radiosity computation. In EGWR '91, 198--205.
• 2: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 3: Fournier, A., Gunawan, A., and Romanzin, C. 1993. Common illumination between real and computer generated scenes. In Proc. of Graph. Interface, 254--262.
• 4: A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.814-821, June 20-26, 2005 [doi>10.1109/CVPR.2005.41]
• 5: Garg, G., Talvala, E., Levoy, M., and Lensch, H. 2006. Symmetric photography: Exploiting data-sparseness in reflectance fields. In EGSR '06, 251--262.
• 6: Simon Gibson , Alan Murta, Interactive Rendering with Real-World Illumination, Proceedings of the Eurographics Workshop on Rendering Techniques 2000, p.365-376, June 26-28, 2000
• 7: Grossberg, M., Peri, H., Nayar, S., and Belhumeur, P. 2004. Making one object look like another: controlling appearance using a projector-camera system. In CVPR '04, 452--459.
• 8: Jacobs, K., and Loscos, C. 2006. Classification of illumination methods for mixed reality. Comp. Graph. Forum 25, 1, 29--51.
• 9: Vincent Masselus , Pieter Peers , Philip Dutré , Yves D. Willems, Relighting with 4D incident light fields, ACM SIGGRAPH 2003 Papers, July 27-31, 2003, San Diego, California
• 10: Wojciech Matusik , Hanspeter Pfister, 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 11: Shree K. Nayar , Gurunandan Krishnan , Michael D. Grossberg , Ramesh Raskar, Fast separation of direct and global components of a scene using high frequency illumination, ACM SIGGRAPH 2006 Papers, July 30-August 03, 2006, Boston, Massachusetts
• 12: Ravi Ramamoorthi , Pat Hanrahan, A signal-processing framework for inverse rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.117-128, August 2001 [doi>10.1145/383259.383271]
• 13: Ramesh Raskar , Greg Welch , Kok-Lim Low , Deepak Bandyopadhyay, Shader Lamps: Animating Real Objects With Image-Based Illumination, Proceedings of the 12th Eurographics Workshop on Rendering Techniques, p.89-102, June 25-27, 2001
• 14: Imari Sato , Yoichi Sato , Katsushi Ikeuchi, Acquiring a Radiance Distribution to Superimpose Virtual Objects onto a Real Scene, IEEE Transactions on Visualization and Computer Graphics, v.5 n.1, p.1-12, January 1999 [doi>10.1109/2945.764865]
• 15: Pradeep Sen , Billy Chen , Gaurav Garg , Stephen R. Marschner , Mark Horowitz , Marc Levoy , Hendrik P. A. Lensch, Dual photography, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 16: J. Unger , A. Wenger , T. Hawkins , A. Gardner , P. Debevec, Capturing and rendering with incident light fields, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 17: Ruigang Yang , Xinyu Huang , Sifang Li , Christopher Jaynes, Toward the Light Field Display: Autostereoscopic Rendering via a Cluster of Projectors, IEEE Transactions on Visualization and Computer Graphics, v.14 n.1, p.84-96, January 2008 [doi>10.1109/TVCG.2007.70410]
• 18: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]
• 19: Zhengyou Zhang, A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.11, p.1330-1334, November 2000 [doi>10.1109/34.888718]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Raytracing

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality

Keywords:
augmented reality, global illumination, image-based relighting, light field