Lighting sensitive display

ABSTRACT

Although display devices have been used for decades, they have functioned without taking into account the illumination of their environment. We present the concept of a lighting sensitive display (LSD)---a display that measures the incident illumination and modifies its content accordingly. An ideal LSD would be able to measure the 4D illumination field incident upon it and generate a 4D light field in response to the illumination. However, current sensing and display technologies do not allow for such an ideal implementation. Our initial LSD prototype uses a 2D measurement of the illumination field and produces a 2D image in response to it. In particular, it renders a 3D scene such that it always appears to be lit by the real environment that the display resides in. The current system is designed to perform best when the light sources in the environment are distant from the display, and a single user in a known location views the display.

The displayed scene is represented by compressing a very large set of images (acquired or rendered) of the scene that correspond to different lighting conditions. The compression algorithm is a lossy one that exploits not only image correlations over the illumination dimensions but also coherences over the spatial dimensions of the image. This results in a highly compressed representation of the original image set. This representation enables us to achieve high quality relighting of the scene in real time. Our prototype LSD can render 640 × 480 images of scenes under complex and varying illuminations at 15 frames per second using a 2 GHz processor. We conclude with a discussion on the limitations of the current implementation and potential areas for future research.

REFERENCES

• 1: Antwerp, J. C. V. 1985. Automatic Brightness Control Apparatus. United States Patent 4,514,727 (April).
• 2: Shai Avidan, EigenSegments: A Spatio-Temporal Decomposition of an Ensemble of Images, Proceedings of the 7th European Conference on Computer Vision-Part III, p.747-758, May 28-31, 2002
• 3: Barbier, B., Ediar, S., and Brun, J. 1991. System for the Display of Luminous Data with Improved Readability. United States Patent 5,057,744 (October).
• 4: Basri, R. and Jacobs, D. 2001. Lambertian reflectance and linear subspaces. In International Conference on Computer Vision. 383--390.
• 5: P. N. Belhumeur , D. J. Kriegman, What is the set of images of an object under all possible lighting conditions?, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.270, June 18-20, 1996
• 6: Biggs, A. J. 1965. Compensator for Color-Television Receivers for Chromacity Variations in Ambient Light. United States Patent 3,200,193 (August).
• 7: Constable, D. W. 1978. Ambient Light Contrast and Color Control Circuit. United States Patent 4,090,216 (May).
• 8: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 9: Julie O'B. Dorsey , Françis X. Sillion , Donald P. Greenberg, Design and simulation of opera lighting and projection effects, Proceedings of the 18th annual conference on Computer graphics and interactive techniques, p.41-50, July 1991
• 10: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 11: Frédo Durand , Julie Dorsey, Interactive Tone Mapping, Proceedings of the Eurographics Workshop on Rendering Techniques 2000, p.219-230, June 26-28, 2000
• 12: Epstein, R., Hallinan, P., and Yuille, A. 1995. 5 plus or minus 2 eigenimages suffice: An empirical investigation of low-dimensional lighting models. In IEEE Workshop on Physics-Based Modeling in Computer Vision. 108--116.
• 13: Fitzgibbon, J. J. 1982. Dimmer and Dimmer Override Control for a Display Device. United States Patent 4,355,334 (October).
• 14: A. S. Georghiades , D. J. Kriegman , P. N. Belhumeur, Illumination Cones for Recognition under Variable Lighting: Faces, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.52, June 23-25, 1998
• 15: Gibson, V. R. 1964. Automatic Brightness-Contrast Control Using Photoresistive Element to Control Brightness and AGC Voltages in Response to Ambient Light. United States Patent 3,147,341 (September).
• 16: Hallinan, P. 1994. A low-dimensional representation of human faces for arbitrary lighting conditions. In IEEE Conference on Computer Vision and Pattern Recognition. 995--999.
• 17: Heijligers, H. 1962. Circuit Arrangement for Automatically Adjusting the Brightness and the Contrast in a Television Receiver. United States Patent 3,027,421 (March).
• 18: Henrik Wann Jensen, Realistic image synthesis using photon mapping, A. K. Peters, Ltd., Natick, MA, 2001
• 19: Kochanski, G. P. 1998. Retro-Focus Type Lens and Projection-Type Display Apparatus. United States Patent 5,854,661 (December).
• 20: Korda, P. 1965. Automatic Contrast and Brilliance Control System for Television Receivers. United States Patent 3,165,582 (January).
• 21: Koudelka, M., Belhumeur, P., Magda, S., and Kriegman, D. 2001. Image-based modeling and rendering of surfaces with arbitrary BRDFs. In IEEE Conference on Computer Vision and Pattern Recognition. 568--575.
• 22: Marc Levoy , Kari Pulli , Brian Curless , Szymon Rusinkiewicz , David Koller , Lucas Pereira , Matt Ginzton , Sean Anderson , James Davis , Jeremy Ginsberg , Jonathan Shade , Duane Fulk, The digital Michelangelo project: 3D scanning of large statues, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.131-144, July 2000 [doi>10.1145/344779.344849]
• 23: Zhouchen Lin , Tien-Tsin Wong , Heung-Yeung Shum, Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction, International Journal of Computer Vision, v.49 n.2-3, p.229-246, September-October 2002 [doi>10.1023/A:1020153824351]
• 24: Tom Malzbender , Dan Gelb , Hans Wolters, Polynomial texture maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.519-528, August 2001 [doi>10.1145/383259.383320]
• 25: Wojciech Matusik , Hanspeter Pfister , Addy Ngan , Paul Beardsley , Remo Ziegler , Leonard McMillan, Image-based 3D photography using opacity hulls, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 26: Miller, G. 1995. Volumetric hyper-reality, a computer graphics holy grail for the 21st century? In Graphics Interface '95. Quebec, Canada, 56--64.
• 27: Nayar, S. and Murase, H. 1994. The dimensionality of illumination manifolds in eigenspace. Tech. Rep. CUCS021-94, Columbia University, New York. August.
• 28: Nayar, S. K., Belhumeur, P. N., and Boult, T. E. 2001. Lighting sensitive display. In Technical Sketch, SIGGRAPH 01. Los Angeles.
• 29: Shree K. Nayar , Vlad Branzoi, Adaptive Dynamic Range Imaging: Optical Control of Pixel Exposures Over Space and Time, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1168, October 13-16, 2003
• 30: Newman, A. J. 1972. Method and Means for Providing a Lighted Display Having a Constant Contrast Ratio. United States Patent 3,649,755 (March).
• 31: Nimeroff, J., Simoncelli, E., and Dorsey, J. 1994. Efficient re-rendering of naturally illuminated environments. In Fifth Eurographics Workshop on Rendering. 359--373.
• 32: Nishino, K., Sato, Y., and Ikeuchi, K. 1999. Eigen-texture method: Appearance compression based on 3D model. In IEEE Conference on Computer Vision and Pattern Recognition. 618--624.
• 33: Ottenstein, J. 1993. Arrangement for Automatically Controlling Brightness of Cockpit Displays. United States Patent 5,270,818 (December).
• 34: Sumanta N. Pattanaik , James A. Ferwerda , Mark D. Fairchild , Donald P. Greenberg, A multiscale model of adaptation and spatial vision for realistic image display, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.287-298, July 1998 [doi>10.1145/280814.280922]
• 35: Prince, J. S., Herz, H. L., and Coleman, E. P. 1986. Brightness Control System for CRT Video Display. United States Patent 4,589,022 (May).
• 36: Ravi Ramamoorthi , Pat Hanrahan, A signal-processing framework for inverse rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.117-128, August 2001 [doi>10.1145/383259.383271]
• 37: Erik Reinhard , Michael Stark , Peter Shirley , James Ferwerda, Photographic tone reproduction for digital images, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 38: Imari Sato , Yoichi Sato , Katsushi Ikeuchi, Acquiring a Radiance Distribution to Superimpose Virtual Objects onto a Real Scene, IEEE Transactions on Visualization and Computer Graphics, v.5 n.1, p.1-12, January 1999 [doi>10.1109/2945.764865]
• 39: Amnon Shashua, On Photometric Issues in 3D Visual Recognition from aSingle 2D Image, International Journal of Computer Vision, v.21 n.1-2, p.99-122, January. 1997 [doi>10.1023/A:1007975506780]
• 40: Peter-Pike Sloan , Jan Kautz , John Snyder, Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 41: Smith, B. and Rowe, L. 1996. Compressed domain processing of JPEG encoded images. Real-Time Imaging 2 2, 2, 3--17.
• 42: Sunnybrook Technologies. 2003. High dynamic range displays. http://www.sunnybrook.com.
• 43: Szermy, N. 1968. Automatic Contrast and Brightness Control for Television Receiver Utilizing a Light Dependent Resistor. United States Patent 3,404,226 (October).
• 44: Patrick C. Teo , Eero P. Simoncelli , David J. Heeger, Efficient Linear Re-rendering for Interactive Lighting Design, Stanford University, Stanford, CA, 1997
• 45: Thomas, L. P. 1963. Television Receiver Circuits. United States Patent 3,096,399 (July).
• 46: Jack Tumblin , Holly Rushmeier, Tone Reproduction for Realistic Images, IEEE Computer Graphics and Applications, v.13 n.6, p.42-48, November 1993 [doi>10.1109/38.252554]
• 47: Greg Ward, A contrast-based scalefactor for luminance display, Graphics gems IV, Academic Press Professional, Inc., San Diego, CA, 1994
• 48: Daniel N. Wood , Daniel I. Azuma , Ken Aldinger , Brian Curless , Tom Duchamp , David H. Salesin , Werner Stuetzle, Surface light fields for 3D photography, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.287-296, July 2000 [doi>10.1145/344779.344925]
• 49: Yabuuchi, S. 1990. Display System with Luminance Calculation. United States Patent 4,952,917 (August).

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
        I.4.10 Image Representation
      I.4.2 Compression (Coding)

General Terms:
Algorithms, Design, Measurement

Keywords:
Color, display algorithms, illumination field, image-based rendering, light field, lighting sensitive display, principal component analysis, reactive display, real-time rendering, relighting, shading, shadowing, singular value decomposition; and texture, three-dimensional displays, virtual reality