Acquiring the reflectance field of a human face

ABSTRACT

We present a method to acquire the reflectance field of a human face and use these measurements to render the face under arbitrary changes in lighting and viewpoint. We first acquire images of the face from a small set of viewpoints under a dense sampling of incident illumination directions using a light stage. We then construct a reflectance function image for each observed image pixel from its values over the space of illumination directions. From the reflectance functions, we can directly generate images of the face from the original viewpoints in any form of sampled or computed illumination. To change the viewpoint, we use a model of skin reflectance to estimate the appearance of the reflectance functions for novel viewpoints. We demonstrate the technique with synthetic renderings of a person's face under novel illumination and viewpoints.

REFERENCES

• 1: ADELSON, E. H., AND BERGEN, J. R. Computational Models of Visual Processing. MIT Press, Cambridge, Mass., 1991, ch. 1. The Plenoptic Function and the Elements of Early Vision.
• 2: Thaddeus Beier , Shawn Neely, Feature-based image metamorphosis, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.35-42, July 1992
• 3: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 4: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 5: BUSBRIDGE, I. W. The Mathematics of Radiative Transfer. Cambridge University Press, Bristol, UK, 1960.
• 6: Robert L. Cook , Kenneth E. Torrance, A reflectance model for computer graphics, Proceedings of the 8th annual conference on Computer graphics and interactive techniques, p.307-316, August 03-07, 1981, Dallas, Texas, United States
• 7: Kristin J. Dana , Shree K. Nayar , Bram Van Ginneken , Jan J. Koenderink, Reflectance and Texture of Real-World Surfaces Authors, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.151, June 17-19, 1997
• 8: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 9: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 10: DEBEVEC, P. E., YU, Y., AND BORSHUKOV, G. D. Efficient viewdependent image-based rendering with projective texture-mapping. In 9th Eurographics workshop on Rendering (June 1998), pp. 105-116.
• 11: Pascal Fua , C. Miccio, From Regular Images to Animated Heads: A Least Squares Approach, Proceedings of the 5th European Conference on Computer Vision-Volume I, p.188-202, June 02-06, 1998
• 12: GERSHUN, A. Svetovoe Pole (the Light Field, in English). Journal of Mathematics and Physics XVIII (1939), 51--151.
• 13: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 14: Brian Guenter , Cindy Grimm , Daniel Wood , Henrique Malvar , Fredric Pighin, Making faces, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.55-66, July 1998 [doi>10.1145/280814.280822]
• 15: HAEBERLI, P. Synthetic lighting for photography. Available at http://www.sgi.com/grafica/synth/index.html, January 1992.
• 16: Pat Hanrahan , Wolfgang Krueger, Reflection from layered surfaces due to subsurface scattering, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.165-174, September 1993 [doi>10.1145/166117.166139]
• 17: KARNER, K. F., MAYER, H., AND GERVAUTZ, M. An image based measurement system for anisotropic reflection. In EUROGRAPHICS Annual Conference Proceedings (1996).
• 18: KAUTZ, J., AND McCOOL, M. D. Interactive rendering with arbitrary BRDFs using separable approximations. Eurographics Rendering Workshop 1999 (June 1999).
• 19: Eric P. F. Lafortune , Sing-Choong Foo , Kenneth E. Torrance , Donald P. Greenberg, Non-linear approximation of reflectance functions, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.117-126, August 1997 [doi>10.1145/258734.258801]
• 20: Yuencheng Lee , Demetri Terzopoulos , Keith Walters, Realistic modeling for facial animation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.55-62, September 1995 [doi>10.1145/218380.218407]
• 21: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 22: MARSCHNER, S. R., WESTIN, S. H., LAFORTUNE, E. P. F., TOR- RANCE, K. E., AND GREENBERG, D. P. Image-based BRDF measurement including human skin. Eurographics Rendering Workshop 1999 (June 1999).
• 23: MILLER, G. S. P., RUBIN, S., AND PONCELEON, D. Lazy decompression of surface light fields for precomputed global illumination. Eurographics Rendering Workshop 1998 (June 1998), 281-292.
• 24: Shree K. Nayar , Xi-Sheng Fang , Terrance Boult, Separation of Reflection Components Using Color and Polarization, International Journal of Computer Vision, v.21 n.3, p.163-186, Feb. 1997 [doi>10.1023/A:1007937815113]
• 25: NICODEMUS, F. E., RICHMOND, J. C., HSIA, J. J., GINSBERG, I. W., AND LIMPERIS, T. Geometric considerations and nomenclature for reflectance.
• 26: NIMEROFF, J. S., SIMONCELLI, E., AND DORSEY, J. Efficient rerendering of naturally illuminated environments. Fifth Eurographics Workshop on Rendering (June 1994), 359-373.
• 27: Michael Oren , Shree K. Nayar, Generalization of Lambert's reflectance model, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.239-246, July 1994 [doi>10.1145/192161.192213]
• 28: Frederick I. Parke, Computer generated animation of faces, Proceedings of the ACM annual conference, August 01-01, 1972, Boston, Massachusetts, United States [doi>10.1145/800193.569955]
• 29: Frédéric Pighin , Jamie Hecker , Dani Lischinski , Richard Szeliski , David H. Salesin, Synthesizing realistic facial expressions from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.75-84, July 1998 [doi>10.1145/280814.280825]
• 30: Mark A. Sagar , David Bullivant , Gordon D. Mallinson , Peter J. Hunter, A virtual environment and model of the eye for surgical simulation, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.205-212, July 1994 [doi>10.1145/192161.192200]
• 31: SATO, Y., AND IKEUCHI, K. Temporal-color space analysis of reflection. JOSA-A 11, 11 (November 1994), 2990-3002.
• 32: Yoichi Sato , Mark D. Wheeler , Katsushi Ikeuchi, Object shape and reflectance modeling from observation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.379-387, August 1997 [doi>10.1145/258734.258885]
• 33: SMITH, B., AND ROWE, L. Compressed domain processing of JPEG- encoded images. Real-Time Imaging 2, 2 (1996), 3-17.
• 34: TERZOPOULOS, D., AND WATERS, K. Physically-based facial modelling, analysis, and animation. Journal of Visualization and Computer Animation 1, 2 (August 1990), 73-80.
• 35: TORRANCE, K. E., AND SPARROW, g. M. Theory for off-specular reflection from roughened surfaces. Journal of Optical Society of America 57, 9 (1967).
• 36: VAN GEMERT, M. F. C., JACQUES, S. L., STERENBERG, H. J. C. M., AND STAR, W.M. Skin optics. IEEE Transactions on Biomedical Engineering 36, 12 (December 1989), 1146-1154.
• 37: Gregory J. Ward, Measuring and modeling anisotropic reflection, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.265-272, July 1992
• 38: Lance Williams, Performance-driven facial animation, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.235-242, September 1990, Dallas, TX, USA
• 39: Tien-Tsin Wong , Pheng-Ann Heng , Siu-Hang Or , Wai-Yin Ng, Image-based Rendering with Controllable Illumination, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.13-22, June 16-18, 1997
• 40: Wu, Y., THALMANN, N. M., AND THALMANN, D. A dynamic wrinkle model in facial animation and skin aging. Journal of Visualization and Computer Animation 6, 4 (October 1995), 195-206.
• 41: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]
• 42: Douglas E. Zongker , Dawn M. Werner , Brian Curless , David H. Salesin, Environment matting and compositing, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.205-214, July 1999 [doi>10.1145/311535.311558]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Intensity, color, photometry, and thresholding

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Radiosity; Color, shading, shadowing, and texture
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture
          Subjects: Radiometry
      I.4.8 Scene Analysis
          Subjects: Photometry

General Terms:
Design, Human Factors, Measurement, Performance, Theory

Keywords:
facial animation, image-based modeling, rendering and lighting