Consistent interactive augmentation of live camera images with correct near-field illumination

ABSTRACT

Inserting virtual objects in real camera images with correct lighting is an active area of research. Current methods use a high dynamic range camera with a fish-eye lens to capture the incoming illumination. The main problem with this approach is the limitation to distant illumination. Therefore, the focus of our work is a real-time description of both near - and far-field illumination for interactive movement of virtual objects in the camera image of a real room. The daylight, which is coming in through the windows, produces a spatially varying distribution of indirect light in the room; therefore a near-field description of incoming light is necessary. Our approach is to measure the daylight from outside and to simulate the resulting indirect light in the room. To accomplish this, we develop a special dynamic form of the irradiance volume for real-time updates of indirect light in the room and combine this with importance sampling and shadow maps for light from outside. This separation allows object movements with interactive frame rates (10--17 fps). To verify the correctness of our approach, we compare images of synthetic objects with real objects.

REFERENCES

• 1: Kusuma Agusanto , Li Li , Zhu Chuangui , Ng Wan Sing, Photorealistic rendering for augmented reality using environment illumination, Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.208, October 07-10, 2003
• 2: Annen, T., Kautz, J., Durand, F., and Seidel, H.-P. 2004. Spherical Harmonic Gradients for Mid-Range Illumination. In Proceedings of the Eurographics Symposium on Rendering, 331--336.
• 3: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 4: Kirill Dmitriev , Thomas Annen , Grzegorz Krawczyk , Karol Myszkowski , Hans-Peter Seidel, A CAVE system for interactive modeling of global illumination in car interior, Proceedings of the ACM symposium on Virtual reality software and technology, November 10-12, 2004, Hong Kong [doi>10.1145/1077534.1077560]
• 5: George Drettakis , Luc Robert , Sylvain Bougnoux, Interactive Common Illumination for Computer Augmented Reality, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.45-56, June 16-18, 1997
• 6: Fournier, A., Gunavan, A., and Romanzin, C. 1993. Common illumination between real and computer generated scenes. In Proc. of Graphics Interface 1993, 254--262.
• 7: Simon Gibson , Toby Howard, Interactive reconstruction of virtual environments from photographs, with application to scene-of-crime analysis, Proceedings of the ACM symposium on Virtual reality software and technology, October 22-25, 2000, Seoul, Korea [doi>10.1145/502390.502399]
• 8: Simon Gibson , Alan Murta, Interactive Rendering with Real-World Illumination, Proceedings of the Eurographics Workshop on Rendering Techniques 2000, p.365-376, June 26-28, 2000
• 9: Simon Gibson , Jon Cook , Toby Howard , Roger Hubbold, Rapid shadow generation in real-world lighting environments, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 10: Michael Goesele , Xavier Granier , Wolfgang Heidrich , Hans-Peter Seidel, Accurate light source acquisition and rendering, ACM SIGGRAPH 2003 Papers, July 27-31, 2003, San Diego, California
• 11: Gene Greger , Peter Shirley , Philip M. Hubbard , Donald P. Greenberg, The Irradiance Volume, IEEE Computer Graphics and Applications, v.18 n.2, p.32-43, March 1998 [doi>10.1109/38.656788]
• 12: Grosch, T., Mueller, S., and Kresse, W. 2003. Goniometric light reconstruction for augmented reality image synthesis. In Graphiktag im Rahmen der GI Jahrestagung, Frankfurt am Main,.
• 13: Grosch, T. 2005. Differential photon mapping: Consistent augmentation of photographs with correction of all light paths. In Eurographics 2005, Trinity College, Dublin, Ireland.
• 14: Grosch, T. 2005. PanoAR: Interactive augmentation of omnidirectional images with consistent lighting. In Mirage 2005, Computer Vision / Computer Graphics Collaboration Techniques and Applications, 25--34.
• 15: Michael Haller , Stephan Drab , Werner Hartmann, A real-time shadow approach for an augmented reality application using shadow volumes, Proceedings of the ACM symposium on Virtual reality software and technology, October 01-03, 2003, Osaka, Japan [doi>10.1145/1008653.1008665]
• 16: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer visiond, Cambridge University Press, New York, NY, 2001
• 17: Miloš Hašan , Fabio Pellacini , Kavita Bala, Direct-to-indirect transfer for cinematic relighting, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 18: Vlastimil Havran , Miloslaw Smyk , Grzegorz Krawczyk , Karol Myszkowski , Hans-Peter Seidel, Importance sampling for video environment maps, ACM SIGGRAPH 2005 Sketches, July 31-August 04, 2005, Los Angeles, California [doi>10.1145/1187112.1187243]
• 19: B. Hoefflinger, High-Dynamic-Range (HDR) Vision: Microelectronics, Image Processing, Computer Graphics (Springer Series in Advanced Microelectronics), Springer-Verlag New York, Inc., Secaucus, NJ, 2007
• 20: Masayuki Kanbara , Naokazu Yokoya, Geometric and Photometric Registration for Real-Time Augmented Reality, Proceedings of the 1st International Symposium on Mixed and Augmented Reality, p.279, September 30-October 01, 2002
• 21: Masayuki Kanbara , Naokazu Yokoya, Real-time Estimation of Light Source Environment for Photorealistic Augmented Reality, Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 2, p.911-914, August 23-26, 2004 [doi>10.1109/ICPR.2004.735]
• 22: Hirokazu Kato , Mark Billinghurst, Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System, Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, p.85, October 20-21, 1999
• 23: Kontkanen, J., and Laine, S. 2006. Sampling Precomputed volumetric lighting. Journal of Graphics Tools 11, 3.
• 24: Kontkanen, J., Turquin, E., Holzschuch, N., and Sillion, F. X. 2006. Wavelet Radiance Transport for Interactive Indirect Lighting. In 17th Eurographics Symposium on Rendering, 161--172.
• 25: Korn, M., Stange, M., Von Arb, A., Blum, L., Kreil, M., Kunze, K., Anhenn, J., Wallrath, T., and Grosch, T. 2006. Interactive augmentation of live images using a HDR stereo camera. In Dritter Workshop Virtuelle und Erweiterte Realitt der GI-Fachgruppe VR/AR, vol. 3, 107--118.
• 26: Krawczyk, G., Goesele, M., and Seidel, H.-P. 2005. Photometric calibration of high dynamic range cameras. Research Report MPI-I-2005-4-005, Max-Planck-Institut für Informatik, Stuhlsatzenhausweg 85, 66123 Saarbrücken, Germany, April.
• 27: Anders Wang Kristensen , Tomas Akenine-Möller , Henrik Wann Jensen, Precomputed local radiance transfer for real-time lighting design, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 28: Loscos, C., Frasson, M., Drettakis, G., Walter, B., Granier, X., and Poulin, P. 1999. Interactive virtual relighting and remodeling of real scenes. In Tenth Eurographics Workshop on Rendering, 329--340.
• 29: Mantiuk, R., Pattanaik, S., and Myszkowski, K. 2002. Cube-map data structure for interactive global illumination computation in dynamic diffuse environments. In ICCVG.
• 30: Eihachiro Nakamae , Koichi Harada , Takao Ishizaki , Tomoyuki Nishita, A montage method: the overlaying of the computer generated images onto a background photograph, Proceedings of the 13th annual conference on Computer graphics and interactive techniques, p.207-214, August 1986
• 31: Ren Ng , Ravi Ramamoorthi , Pat Hanrahan, All-frequency shadows using non-linear wavelet lighting approximation, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 32: Nijasure, M., Pattanaik, S. N., and Goel, V. 2005. Real-time global illumination on GPUs. journal of graphics tools 10, 2, 55--71.
• 33: Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994. Efficient Re-rendering of Naturally Illuminated Environments. In Fifth Eurographics Workshop on Rendering, Springer-Verlag, Darmstadt, Germany, 359--373.
• 34: Perez, R., Seals, R., and Michalsky, J. 1993. All-weather model for sky luminance distribution-preliminary configuration and validation. In Solar Energy 50, 3, 235--245.
• 35: Matt Pharr , Greg Humphreys, Physically Based Rendering: From Theory to Implementation, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2004
• 36: Ravi Ramamoorthi , Pat Hanrahan, An efficient representation for irradiance environment maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.497-500, August 2001 [doi>10.1145/383259.383317]
• 37: Erik Reinhard , Greg Ward , Sumanta Pattanaik , Paul Debevec, High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 38: Ritschel, T., and Grosch, T. 2006. On-line estimation of diffuse materials. In Dritter Workshop Virtuelle und Erweiterte Realitt der GI-Fachgruppe VR/AR, vol. 3, 95--106.
• 39: Imari Sato , Yoichi Sato , Katsushi Ikeuchi, Acquiring a Radiance Distribution to Superimpose Virtual Objects onto a Real Scene, IEEE Transactions on Visualization and Computer Graphics, v.5 n.1, p.1-12, January 1999 [doi>10.1109/2945.764865]
• 40: Francois X. Sillion , Claude Puech, Radiosity and Global Illumination, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1994
• 41: Peter-Pike Sloan , Jan Kautz , John Snyder, Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 42: Lance Williams, Casting curved shadows on curved surfaces, Proceedings of the 5th annual conference on Computer graphics and interactive techniques, p.270-274, August 23-25, 1978

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture; Radiosity

Keywords:
augmented image synthesis, global illumination