Image-based reconstruction of spatial appearance and geometric detail

ABSTRACT

Real-world objects are usually composed of a number of different materials that often show subtle changes even within a single material. Photorealistic rendering of such objects requires accurate measurements of the reflection properties of each material, as well as the spatially varying effects. We present an image-based measuring method that robustly detects the different materials of real objects and fits an average bidirectional reflectance distribution function (BRDF) to each of them. In order to model local changes as well, we project the measured data for each surface point into a basis formed by the recovered BRDFs leading to a truly spatially varying BRDF representation. Real-world objects often also have fine geometric detail that is not represented in an acquired mesh. To increase the detail, we derive normal maps even for non-Lambertian surfaces using our measured BRDFs. A high quality model of a real object can be generated with relatively little input data. The generated model allows for rendering under arbitrary viewing and lighting conditions and realistically reproduces the appearance of the original object.

REFERENCES

• 1: Fausto Bernardini , Ioana M. Martin , Holly Rushmeier, High-Quality Texture Reconstruction from Multiple Scans, IEEE Transactions on Visualization and Computer Graphics, v.7 n.4, p.318-332, October 2001 [doi>10.1109/2945.965346]
• 2: Samuel Boivin , Andre Gagalowicz, Image-based rendering of diffuse, specular and glossy surfaces from a single image, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.107-116, August 2001 [doi>10.1145/383259.383270]
• 3: Kristin J. Dana , Bram van Ginneken , Shree K. Nayar , Jan J. Koenderink, Reflectance and texture of real-world surfaces, ACM Transactions on Graphics (TOG), v.18 n.1, p.1-34, Jan. 1999 [doi>10.1145/300776.300778]
• 4: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 5: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 6: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 7: Forsyth, D. A. and Zisserman, A. 1989. Mutual Illumination. In Proceedings of Computer Vision and Pattern Recognition (CVPR '89). IEEE Computer Society Press, 466--473.
• 8: R. Furukawa , H. Kawasaki , K. Ikeuchi , M. Sakauchi, Appearance based object modeling using texture database: acquisition, compression and rendering, Proceedings of the 13th Eurographics workshop on Rendering, June 26-28, 2002, Pisa, Italy
• 9: Michael Garland , Paul S. Heckbert, Surface simplification using quadric error metrics, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.209-216, August 1997 [doi>10.1145/258734.258849]
• 10: Allen Gersho , Robert M. Gray, Vector quantization and signal compression, Kluwer Academic Publishers, Norwell, MA, 1991
• 11: Gibson, S., Howard, T., and Hubbold, R. 2001. Flexible image-based photometric reconstruction using virtual light sources. Comput. Graph. Forum 20, 3. ISSN 1067--7055.
• 12: Goesele, M., Heidrich, W., Lensch, H. P., and Seidel, H.-P. 2000. Building a photo studio for measurement purposes. In Proceedings of Vision, Modeling, and Visualization. 231--238.
• 13: Goesele, M., Heidrich, W., and Seidel, H.-P. 2001. Entropy based dark frame subtraction. In Image Processing, Image Quality, Image Capture Systems Conference (PICS). 293--298.
• 14: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 15: Jan Kautz , Hans-Peter Seidel, Towards interactive bump mapping with anisotropic shift-variant BRDFs, Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on Graphics hardware, p.51-58, August 21-22, 2000, Interlaken, Switzerland [doi>10.1145/346876.348214]
• 16: Kobbelt, L. 1996. Discrete fairing. In Proceedings of the 7th IMA Conference on the Mathematics of Surfaces. 101--131.
• 17: Jan J. Koenderink , Andrea J. van Doorn , Marigo Stavridi, Bidirectional Reflection Distribution Function Expressed in Terms of Surface Scattering Modes, Proceedings of the 4th European Conference on Computer Vision-Volume II, p.28-39, April 15-18, 1996
• 18: Eric P. F. Lafortune , Sing-Choong Foo , Kenneth E. Torrance , Donald P. Greenberg, Non-linear approximation of reflectance functions, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.117-126, August 1997 [doi>10.1145/258734.258801]
• 19: Hendrik P. A. Lensch , Wolfgang Heidrich , Hans-Peter Seidel, Automated Texture Registration and Stitching for Real World Models, Proceedings of the 8th Pacific Conference on Computer Graphics and Applications, p.317, October 03-05, 2000
• 20: Hendrik P.A. Lensch , Wolfgang Heidrich , Hans-Peter Seidel, A silhouette-based algorithm for texture registration and stitching, Graphical Models, v.63 n.4, p.245-262, July 2001 [doi>10.1006/gmod.2001.0554]
• 21: Hendrik P. A. Lensch , Michael Goesele , Jan Kautz , Wolfgang Heidrich , Hans-Peter Seidel, Image-Based Reconstruction of Spatially Varying Materials, Proceedings of the 12th Eurographics Workshop on Rendering Techniques, p.103-114, June 25-27, 2001
• 22: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 23: Lewis, R. 1993. Making shaders more physically plausible. In 4th Eurographics Workshop on Rendering. 47--62.
• 24: Xinguo Liu , Yizhou Yu , Heung-Yeung Shum, Synthesizing bidirectional texture functions for real-world surfaces, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.97-106, August 2001 [doi>10.1145/383259.383269]
• 25: Lloyd, S. 1982. Least squares quantization in PCM. IEEE Trans. Inform. Theory IT-28, 129--137.
• 26: Lu, R., Koenderink, J., and Kappers, A. 1998. Optical Properties (bidirectional reflectance distribution functions) of velvet. Applied Optics 37, 25 (Sept.), 5974--5984.
• 27: MacQueen, J. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability. Vol. 1.
• 28: Tom Malzbender , Dan Gelb , Hans Wolters, Polynomial texture maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.519-528, August 2001 [doi>10.1145/383259.383320]
• 29: Stephen Robert Marschner , Donald P. Greenberg, Inverse rendering for computer graphics, 1998
• 30: Marschner, S., Westin, S., Lafortune, E., Torrance, K., and Greenberg, D. 1999. Image-based BRDF Measurement Including Human Skin. In 10th Eurographics Workshop on Rendering. 131--144.
• 31: Wojciech Matusik , Hanspeter Pfister , Addy Ngan , Paul Beardsley , Remo Ziegler , Leonard McMillan, Image-based 3D photography using opacity hulls, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 32: Wojciech Matusik , Hanspeter Pfister , Remo Ziegler , Addy Ngan , Leonard McMillan, Acquisition and rendering of transparent and refractive objects, Proceedings of the 13th Eurographics workshop on Rendering, June 26-28, 2002, Pisa, Italy
• 33: McAllister, D. 2002. A Generalized Representation of Surface Appearance. Ph.D. thesis, University of North Carolina.
• 34: David K. McAllister , Anselmo Lastra , Wolfgang Heidrich, Efficient rendering of spatial bi-directional reflectance distribution functions, Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware, September 01-02, 2002, Saarbrucken, Germany
• 35: Miller, G., Rubin, S., and Ponceleon, D. 1998. Lazy decompression of surface light fields for precomputed global illumination. In 9th Eurographics Workshop on Rendering. 281--292.
• 36: Shree K. Nayar , Katsushi Ikeuchi , Takeo Kanade, Surface Reflection: Physical and Geometrical Perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13 n.7, p.611-634, July 1991 [doi>10.1109/34.85654]
• 37: Nayar, S. K., Ikeuchi, K., and Kanade, T. 1990. Shape from Interreflections. In International Conference on Computer Vision (ICCV '90). Osaka, Japan, 2--11.
• 38: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical recipes in C (2nd ed.): the art of scientific computing, Cambridge University Press, New York, NY, 1992
• 39: Ravi Ramamoorthi , Pat Hanrahan, A signal-processing framework for inverse rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.117-128, August 2001 [doi>10.1145/383259.383271]
• 40: Robertson, M. A., Borman, S., and Stevenson, R. L. 1999. Dynamic Range Improvement Through Multiple Exposures. In Proceedings of the International Conference on Image Processing (ICIP'99). IEEE, 159--163.
• 41: Rushmeier, H., Bernardini, F., Mittleman, J., and Taubin, G. 1998. Acquiring input for rendering at appropriate levels of detail: Digitizing a piet&abrave;. Eurographics Rendering Workshop 1998, 81--92. ISBN 3-211-83213-0. Held in Vienna, Austria.
• 42: Holly E. Rushmeier , Gabriel Taubin , André Guéziec, Appying Shape from Lighting Variation to Bump Map Capture, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.35-44, June 16-18, 1997
• 43: Yoichi Sato , Mark D. Wheeler , Katsushi Ikeuchi, Object shape and reflectance modeling from observation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.379-387, August 1997 [doi>10.1145/258734.258885]
• 44: Schirmacher, H., Heidrich, W., Rubick, M., Schiron, D., and Seidel, H.-P. 1999. Image-Based BRDF Reconstruction. In Proceedings of the 4th VMV Conference. 285--292.
• 45: Torrance, K. and Sparrow, E. 1967. Theory for off-specular reflection from roughened surfaces. J. Optic. Soc. Amer. 57, 9.
• 46: Gregory J. Ward, Measuring and modeling anisotropic reflection, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.265-272, July 1992
• 47: Stephen H. Westin , James R. Arvo , Kenneth E. Torrance, Predicting reflectance functions from complex surfaces, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.255-264, July 1992
• 48: Daniel N. Wood , Daniel I. Azuma , Ken Aldinger , Brian Curless , Tom Duchamp , David H. Salesin , Werner Stuetzle, Surface light fields for 3D photography, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.287-296, July 2000 [doi>10.1145/344779.344925]
• 49: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]
• 50: Yizhou Yu , Jitendra Malik, Recovering photometric properties of architectural scenes from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.207-217, July 1998 [doi>10.1145/280814.280874]
• 51: Ruo Zhang , Ping-Sing Tsai , James Edwin Cryer , Mubarak Shah, Shape from Shading: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.21 n.8, p.690-706, August 1999 [doi>10.1109/34.784284]
• 52: Zhang, Z. 1999. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. In Proceedings of the International Conference on Computer Vision. 666--673.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture

General Terms:
Measurement

Keywords:
BRDF measurement, normal map acquisition, photometric stereo, shape from shading, spatially varying BRDFs