A signal-processing framework for inverse rendering

ABSTRACT

Realism in computer-generated images requires accurate input models for lighting, textures and BRDFs. One of the best ways of obtaining high-quality data is through measurements of scene attributes from real photographs by inverse rendering. However, inverse rendering methods have been largely limited to settings with highly controlled lighting. One of the reasons for this is the lack of a coherent mathematical framework for inverse rendering under general illumination conditions. Our main contribution is the introduction of a signal-processing framework which describes the reflected light field as a convolution of the lighting and BRDF, and expresses it mathematically as a product of spherical harmonic coefficients of the BRDF and the lighting. Inverse rendering can then be viewed as deconvolution. We apply this theory to a variety of problems in inverse rendering, explaining a number of previous empirical results. We will show why certain problems are ill-posed or numerically ill-conditioned, and why other problems are more amenable to solution. The theory developed here also leads to new practical representations and algorithms. For instance, we present a method to factor the lighting and BRDF from a small number of views, i.e. to estimate both simultaneously when neither is known.

REFERENCES

• 1: R. Basri and D. Jacobs. Lambertian reflectance and linear subspaces. In International Conference on Computer Vision, 2001.
• 2: Rui Bastos , Kenneth Hoff , William Wynn , Anselmo Lastra, Increased photorealism for interactive architectural walkthroughs, Proceedings of the 1999 symposium on Interactive 3D graphics, p.183-190, April 26-29, 1999, Atlanta, Georgia, United States [doi>10.1145/300523.300551]
• 3: Brian Cabral , Nelson Max , Rebecca Springmeyer, Bidirectional reflection functions from surface bump maps, Proceedings of the 14th annual conference on Computer graphics and interactive techniques, p.273-281, August 1987
• 4: Brian Curless , Marc Levoy, A volumetric method for building complex models from range images, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.303-312, August 1996 [doi>10.1145/237170.237269]
• 5: Kristin J. Dana , Bram van Ginneken , Shree K. Nayar , Jan J. Koenderink, Reflectance and texture of real-world surfaces, ACM Transactions on Graphics (TOG), v.18 n.1, p.1-34, Jan. 1999 [doi>10.1145/300776.300778]
• 6: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 7: R. Dror, E. Adelson, and A. Willsky. Estimating surface reflectance properties from images under unknown illumination. In SPIE Photonics West: Human Vision and Electronic Imaging VI, 2001.
• 8: M. D'Zmura. Computational Models of Visual Processing, chapter Shading Ambiguity: Reflectance and Illumination, pages 187-207. MIT Press, 1991.
• 9: Numerical Algorithms Group. NAG C Library Manual, Mark 5. 1999.
• 10: Katsushi Ikeuchi , Kosuke Sato, Determining Reflectance Properties of an Object Using Range and Brightness Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13 n.11, p.1139-1153, November 1991 [doi>10.1109/34.103274]
• 11: T. Inui, Y. Tanabe, and Y. Onodera. Group theory and its applications in physics. Springer Verlag, 1990.
• 12: K. F. Karner, H. Mayer, and M. Gervautz. An image based measurement system for anisotropic reflection. Computer Graphics Forum, 15(3):119-128, 1996.
• 13: G.J. Klinker, S.A. Shafer, and T. Kanade. The measurement of highlights in color images. IJCV, 2(1):7-32, 1988.
• 14: J. J. Koenderink and A. J. van Doorn. Phenomenological description of bidirectional surface reflection. JOSA A, 15(11):2903-2912, 1998.
• 15: E. Land and J. McCann. Lightness and retinex theory. Journal of the Optical Society of America, 61(1):1-11, 1971.
• 16: Marc Levoy , Kari Pulli , Brian Curless , Szymon Rusinkiewicz , David Koller , Lucas Pereira , Matt Ginzton , Sean Anderson , James Davis , Jeremy Ginsberg , Jonathan Shade , Duane Fulk, The digital Michelangelo project: 3D scanning of large statues, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.131-144, July 2000 [doi>10.1145/344779.344849]
• 17: R.C. Love. Surface Reflection Model Estimation from Naturally Illuminated Image Sequences. PhD thesis, Leeds, 1997.
• 18: R. Lu, J. Koenderink, and A. Kappers. Optical properties (bidirectional reflection distribution functions) of velvet. Applied Optics, 37(25):5974-5984, 1998.
• 19: T.M. MacRobert. Spherical harmonics; an elementary treatise on harmonic functions, with applications. Dover Publications, 1948.
• 20: S.R. Marschner and D.P. Greenberg. Inverse lighting for photography. In Fifth Color Imaging Conference, pages 262-265, 1997.
• 21: S.R. Marschner, S.H. Westin, E.P.F. Lafortune, and K.E. Torrance. Image-Based BRDF measurement. Applied Optics, 39(16):2592-2600, 2000.
• 22: N. McCormick. Inverse radiative transfer problems: a review. Nuclear Science and Engineering, 112:185-198, 1992.
• 23: G. Miller and C. Hoffman. Illumination and reflection maps: Simulated objects in simulated and real environments. SIGGRAPH 84 Advanced Computer Graphics Animation seminar notes, 1984.
• 24: F. E. Nicodemus, J. C. Richmond, J. J. Hsia, I. W. Ginsberg, and T. Limperis. Geometric Considerations and Nomenclature for Reflectance. National Bureau of Standards (US), 1977.
• 25: J. Nimeroff, E. Simoncelli, and J. Dorsey. Efficient re-rendering of naturally illuminated environments. In EGWR 94, pages 359-373.
• 26: R.W. Preisendorfer. Hydrologic Optics. US Dept Commerce, 1976.
• 27: R. Ramamoorthi and P. Hanrahan. Analysis of planar light fields from homogeneous convex curved surfaces under distant illumination. In SPIE Photonics West: Human Vision and Electronic Imaging VI, 2001.
• 28: Ravi Ramamoorthi , Pat Hanrahan, An efficient representation for irradiance environment maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.497-500, August 2001 [doi>10.1145/383259.383317]
• 29: R. Ramamoorthi and P. Hanrahan. On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object. To appear, Journal of the Optical Society of America A, 2001.
• 30: I. Sato, Y. Sato, and K. Ikeuchi. Illumination distribution from brightness in shadows: adaptive estimation of illumination distribution with unknown reflectance properties in shadow regions. In ICCV 99, pages 875 - 882, 1999.
• 31: Yoichi Sato , Katsushi Ikeuchi, Reflectance Analysis Under Solar Illumination, Carnegie Mellon University, Pittsburgh, PA, 1994
• 32: Yoichi Sato , Mark D. Wheeler , Katsushi Ikeuchi, Object shape and reflectance modeling from observation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.379-387, August 1997 [doi>10.1145/258734.258885]
• 33: Françis X. Sillion , James R. Arvo , Stephen H. Westin , Donald P. Greenberg, A global illumination solution for general reflectance distributions, Proceedings of the 18th annual conference on Computer graphics and interactive techniques, p.187-196, July 1991
• 34: Stephen Robert Marschner , Donald P. Greenberg, Inverse rendering for computer graphics, 1998
• 35: Shoji Tominaga , Norihiro Tanaka, Estimating Reflection Parameters from a Single Color Image, IEEE Computer Graphics and Applications, v.20 n.5, p.58-66, September 2000 [doi>10.1109/38.865881]
• 36: K. E. Torrance and E. M. Sparrow. Theory for off-specular reflection from roughened surfaces. JOSA, 57(9):1105-1114, 1967.
• 37: Gregory J. Ward, Measuring and modeling anisotropic reflection, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.265-272, July 1992
• 38: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]
• 39: Yizhou Yu , Jitendra Malik, Recovering photometric properties of architectural scenes from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.207-217, July 1998 [doi>10.1145/280814.280874]
• 40: Zhengyou Zhang, A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.11, p.1330-1334, November 2000 [doi>10.1109/34.888718]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture
      I.4.8 Scene Analysis
          Subjects: Photometry

General Terms:
Algorithms, Design, Theory

Keywords:
BRDF, illumination, inverse rendering, irradiance, light field, radiance, signal processing, spherical harmonics