A lighting reproduction approach to live-action compositing

ABSTRACT

We describe a process for compositing a live performance of an actor into a virtual set wherein the actor is consistently illuminated by the virtual environment. The Light Stage used in this work is a two-meter sphere of inward-pointing RGB light emitting diodes focused on the actor, where each light can be set to an arbitrary color and intensity to replicate a real-world or virtual lighting environment. We implement a digital two-camera infrared matting system to composite the actor into the background plate of the environment without affecting the visible-spectrum illumination on the actor. The color reponse of the system is calibrated to produce correct color renditions of the actor as illuminated by the environment. We demonstrate moving-camera composites of actors into real-world environments and virtual sets such that the actor is properly illuminated by the environment into which they are composited.

REFERENCES

• 1: ALEXANDER, D. K., JONES, P. J., AND JENKINS, H. The artificial sky and heliodon facility at Cardiff University. In Proc. International Building Performance Simulation Association (IBPSA) (September 1999). (Abstract).
• 2: BEN-EZRA, M. Segmentation with invisible keying signal. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog. (June 2000), pp. 568-575.
• 3: Carlos F. Borges, Trichromatic approximation for computer graphics illumination models, ACM SIGGRAPH Computer Graphics, v.25 n.4, p.101-104, July 1991
• 4: Ron Brinkmann, The art and science of digital compositing, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 5: Yung-Yu Chuang , Douglas E. Zongker , Joel Hindorff , Brian Curless , David H. Salesin , Richard Szeliski, Environment matting extensions: towards higher accuracy and real-time capture, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.121-130, July 2000 [doi>10.1145/344779.344844]
• 6: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 7: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 8: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 9: Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, Cambridge, MA, 1993
• 10: FIELDING, R. The Technique of Special Effects Cinematography, 4th ed. Hastings House, New York, 1985, ch. 11, pp. 290-321.
• 11: GAT, N. Real-time multi- and hyper-spectral imaging for remote sensing and machine vision: an overview. In Proc. 1998 ASAE Annual International Mtg. (Orlando, Florida, July 1998).
• 12: James T. Kajiya, The rendering equation, ACM SIGGRAPH Computer Graphics, v.20 n.4, p.143-150, Aug. 1986
• 13: KOUDELKA, M., MAGDA, S., BELHUMEUR, P., AND KRIEGMAN, D. Image-based modeling and rendering of surfaces with arbitrary brdfs. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog. (2001), pp. 568-575.
• 14: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 15: Tom Malzbender , Dan Gelb , Hans Wolters, Polynomial texture maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.519-528, August 2001 [doi>10.1145/383259.383320]
• 16: Wojciech Matusik , Chris Buehler , Ramesh Raskar , Steven J. Gortler , Leonard McMillan, Image-based visual hulls, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.369-374, July 2000 [doi>10.1145/344779.344951]
• 17: MILLER, G. S., AND HOFFMAN, C. R. Illumination and reflection maps: Simulated objects in simulated and real environments. In SIGGRAPH 84 Course Notes for Advanced Computer Graphics Animation (July 1984).
• 18: NIMEROFF, J. S., SIMONCELLI, E., AND DORSEY, J. Efficient re-rendering of naturally illuminated environments. Fifth Eurographics Workshop on Rendering (June 1994), 359-373.
• 19: Thomas Porter , Tom Duff, Compositing digital images, Proceedings of the 11th annual conference on Computer graphics and interactive techniques, p.253-259, January 1984
• 20: Ravi Ramamoorthi , Pat Hanrahan, A signal-processing framework for inverse rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.117-128, August 2001 [doi>10.1145/383259.383271]
• 21: Alvy Ray Smith , James F. Blinn, Blue screen matting, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.259-268, August 1996 [doi>10.1145/237170.237263]
• 22: TCHOU, C., AND DEBEVEC, P. HDR Shop. Available at http://www.debevec.org/HDRShop, 2001.
• 23: TSAI, R. A versatile camera calibration technique for high accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation 3, 4 (August 1987), 323-344.
• 24: VIDOR, Z. An infrared self-matting process. Society of Motion Picture and Television Engineers 69 (June 1960), 425-427.
• 25: WINDOWS AND DAYLIGHTING GROUP. The sky simulator for daylighting studies. Tech. Rep. DA 188 LBL, Lawrence Berkeley Laboratories, January 1985.
• 26: WYSZECKI, G., AND STILES, W. S. Color Science: Concepts and Methods, Quantitative and Formulae, 2nd ed. Wiley, New York, 1982.
• 27: Douglas E. Zongker , Dawn M. Werner , Brian Curless , David H. Salesin, Environment matting and compositing, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.205-214, July 1999 [doi>10.1145/311535.311558]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Radiosity

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture

General Terms:
Algorithms

Keywords:
global illumination, image-based lighting, matting and compositing, radiosity, reflectance and shading