Image-based rendering of diffuse, specular and glossy surfaces from a single image

ABSTRACT

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The result is a full model of the reflectance properties of all surfaces, which can be rendered under novel illumination conditions with, for example, viewpoint modification and the addition of new synthetic objects. Our technique produces a reflectance model using a small number of parameters. These parameters nevertheless approximate the BRDF and allow the recovery of the photometric properties of diffuse, specular, isotropic or anisotropic textured objects. The input data are a geometric model of the scene including the light source positions and the camera properties, and a single image captured using this camera. Our algorithm generates a new synthetic image using classic rendering techniques, and a lambertian hypothesis about the reflectance model of the surfaces. Then, it iteratively compares the original image to the new one, and chooses a more complex reflectance model if the difference between the two images is greater than a user-defined threshold.

We present several synthetic images that are compared to the original ones, and some possible applications in augmented reality.

REFERENCES

• 1: Arthur Appel. Some techniques for shading machine renderings of solids. AFIPS 1968 Spring Joint Computer Conf., 32:37-45, 1968.
• 2: Réjean Baribeau , Marc Rioux , Guy Godin, Color Reflectance Modeling Using a Polychromatic Laser Range Sensor, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.14 n.2, p.263-269, February 1992 [doi>10.1109/34.121793]
• 3: Loren Carpenter, The A -buffer, an antialiased hidden surface method, Proceedings of the 11th annual conference on Computer graphics and interactive techniques, p.103-108, January 1984
• 4: Michael F. Cohen , Shenchang Eric Chen , John R. Wallace , Donald P. Greenberg, A progressive refinement approach to fast radiosity image generation, Proceedings of the 15th annual conference on Computer graphics and interactive techniques, p.75-84, June 1988
• 5: Michael F. Cohen , Donald P. Greenberg, The hemi-cube: a radiosity solution for complex environments, Proceedings of the 12th annual conference on Computer graphics and interactive techniques, p.31-40, July 1985
• 6: Kristin J. Dana , Bram van Ginneken , Shree K. Nayar , Jan J. Koenderink, Reflectance and texture of real-world surfaces, ACM Transactions on Graphics (TOG), v.18 n.1, p.1-34, Jan. 1999 [doi>10.1145/300776.300778]
• 7: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 8: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 9: Paul Ernest Debevec. Modeling and Rendering Architecture from Photographs. PhD thesis, University of California, Berkeley, 1996.
• 10: Daniel DeMenthon , Larry S. Davis, Model-Based Object Pose in 25 Lines of Code, Proceedings of the Second European Conference on Computer Vision, p.335-343, May 19-22, 1992
• 11: George Drettakis , Luc Robert , Sylvain Bougnoux, Interactive Common Illumination for Computer Augmented Reality, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.45-56, June 16-18, 1997
• 12: Fredo Durand. Visibilite tridimensionnelle: etude analytique et applications. Ph.D. thesis, Universite Joseph Fourier, Grenoble, France, July 1999.
• 13: Eugene Fiume , Alain Fournier , Larry Rudolph, A parallel scan conversion algorithm with anti-aliasing for a general-purpose ultracomputer, Proceedings of the 10th annual conference on Computer graphics and interactive techniques, p.141-150, July 25-29, 1983, Detroit, Michigan, United States
• 14: Alain Fournier, Atjeng S. Gunawan, and Chris Romanzin. Common illumination between real and computer generated scenes. In Graphics Interface '93, pages 254-262. Canadian Information Processing Society, May 1993. Held in Toronto, Ontario, Canada.
• 15: Andrew S. Glassner, An introduction to ray tracing, Academic Press Ltd., London, UK, 1989
• 16: Cindy M. Goral , Kenneth E. Torrance , Donald P. Greenberg , Bennett Battaile, Modeling the interaction of light between diffuse surfaces, Proceedings of the 11th annual conference on Computer graphics and interactive techniques, p.213-222, January 1984
• 17: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, New York, NY, 1992
• 18: Pat Hanrahan , David Salzman , Larry Aupperle, A rapid hierarchical radiosity algorithm, Proceedings of the 18th annual conference on Computer graphics and interactive techniques, p.197-206, July 1991
• 19: Nelder J.A. and Mead R. A simplex method for function minimization. Computer Journal, 7:308-313, 1965.
• 20: Konrad F. Karner, Heinz Mayer, and Michael Gervautz. An image based measurement system for anisotropic reflection. Computer Graphics Forum, 15(3):119-128, August 1996.
• 21: Greg Kay , Terry Caelli, Inverting an illumination model from range and intensity maps, CVGIP: Image Understanding, v.59 n.2, p.183-201, March 1994 [doi>10.1006/ciun.1994.1012]
• 22: Eric P. F. Lafortune , Sing-Choong Foo , Kenneth E. Torrance , Donald P. Greenberg, Non-linear approximation of reflectance functions, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.117-126, August 1997 [doi>10.1145/258734.258801]
• 23: C. Loscos, M. C. Frasson, G. Drettakis, B. Walter, X. Grainer, and P. Poulin. Interactive virtual relighting and remodeling of real scenes. Available from www.imagis.imag.fr/Publications RT-0230, Institut National de Recherche en Informatique en Automatique (INRIA), Grenoble, France, April 1999.
• 24: Céline Loscos , George Drettakis , Luc Robert, Interactive Virtual Relighting of Real Scenes, IEEE Transactions on Visualization and Computer Graphics, v.6 n.4, p.289-305, October 2000 [doi>10.1109/2945.895874]
• 25: Jiping Lu , J. Little, Reflectance function estimation and shape recovery from image sequence of a rotating object, Proceedings of the Fifth International Conference on Computer Vision, p.80, June 20-23, 1995
• 26: Stephen R. Marschner and Donald P. Greenberg. Inverse lighting for photography. In Proceedings of the Fifth Color Imaging Conference. Society for Imaging Science and Technology, November 1997.
• 27: Stephen R. Marschner, Stephen H. Westin, Eric P. F. Lafortune, Kenneth E. Torrance, and Donald P.Greenberg. Image-based brdf measurement including human skin. In Dani Lischinski and Greg Ward Larson, editors, Eurographics Rendering Workshop 1999. Eurographics, June 1999.
• 28: A. Rosenblum. Data Visualization, chapter Modeling Complex indoor scenes using an analysis/synthesis framework (Andre Gagalowicz). Academic Press, 1994.
• 29: Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. Illumination distribution from brightness in shadows: Adaptive extimation of illumination distribution with unknown reflectance properties in shadow regions. In Proceedings of IEEE ICCC'99, pages 875-882, September 1999.
• 30: Katsushi Ikeuchi , Kosuke Sato, Determining Reflectance Properties of an Object Using Range and Brightness Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13 n.11, p.1139-1153, November 1991 [doi>10.1109/34.103274]
• 31: Yoichi Sato and Katsushi Ikeuchi. Temporal-color space analysis of reflection. Journal of Optical Society of America, 11(11):2990-3002, November 1994.
• 32: Yoichi Sato , Katsushi Ikeuchi, Reflectance analysis for 3D computer graphics model generation, Graphical Models and Image Processing, v.58 n.5, p.437-451, Sept. 1996 [doi>10.1006/gmip.1996.0036]
• 33: Yoichi Sato , Mark D. Wheeler , Katsushi Ikeuchi, Object shape and reflectance modeling from observation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.379-387, August 1997 [doi>10.1145/258734.258885]
• 34: Tien-Tsin Wong , Pheng-Ann Heng , Siu-Hang Or , Wai-Yin Ng, Image-based Rendering with Controllable Illumination, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.13-22, June 16-18, 1997
• 35: K.E. Torrance, E.M. Sparrow, and R.C. Birkebak. Polarization, directional distribution, and off-specular peak phenomena in light reflected from roughened surfaces. Journal of Optical Society of America, 56(7):916-925, 1966.
• 36: Jack Tumblin , Holly Rushmeier, Tone Reproduction for Realistic Images, IEEE Computer Graphics and Applications, v.13 n.6, p.42-48, November 1993 [doi>10.1109/38.252554]
• 37: Gregory J. Ward, Measuring and modeling anisotropic reflection, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.265-272, July 1992
• 38: Gregory J. Ward, The RADIANCE lighting simulation and rendering system, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.459-472, July 1994 [doi>10.1145/192161.192286]
• 39: Hank Weghorst , Gary Hooper , Donald P. Greenberg, Improved Computational Methods for Ray Tracing, ACM Transactions on Graphics (TOG), v.3 n.1, p.52-69, Jan. 1984 [doi>10.1145/357332.357335]
• 40: George Wolberg, Digital Image Warping, IEEE Computer Society Press, Los Alamitos, CA, 1994
• 41: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Modeling and recovery of physical attributes

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation
          Subjects: Display algorithms
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Radiosity
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Shading

General Terms:
Algorithms, Design, Theory

Keywords:
BRDF models, global illumination, image-based rendering, inverse rendering, radiance, radiosity, reflectance recovery, rendering, rerendering