Eyes for relighting

ABSTRACT

The combination of the cornea of an eye and a camera viewing the eye form a catadioptric (mirror + lens) imaging system with a very wide field of view. We present a detailed analysis of the characteristics of this corneal imaging system. Anatomical studies have shown that the shape of a normal cornea (without major defects) can be approximated with an ellipsoid of fixed eccentricity and size. Using this shape model, we can determine the geometric parameters of the corneal imaging system from the image. Then, an environment map of the scene with a large field of view can be computed from the image. The environment map represents the illumination of the scene with respect to the eye. This use of an eye as a natural light probe is advantageous in many relighting scenarios. For instance, it enables us to insert virtual objects into an image such that they appear consistent with the illumination of the scene. The eye is a particularly useful probe when relighting faces. It allows us to reconstruct the geometry of a face by simply waving a light source in front of the face. Finally, in the case of an already captured image, eyes could be the only direct means for obtaining illumination information. We show how illumination computed from eyes can be used to replace a face in an image with another one. We believe that the eye not only serves as a useful tool for relighting but also makes relighting possible in situations where current approaches are hard to use.

REFERENCES

• 1: AGARWAL, S., RAMAMOORTHI, R., BELONGIE, S., AND JENSEN, H. 2003. Structured Importance Sampling of Environment Maps. In Proc. of ACM SIGGRAPH 2003, 605--612.
• 2: Simon Baker , Shree K. Nayar, A Theory of Single-Viewpoint Catadioptric Image Formation, International Journal of Computer Vision, v.35 n.2, p.175-196, Nov.-Dec. 1999 [doi>10.1023/A:1008128724364]
• 3: BAKER, T. 1943. Ray tracing through non-spherical surfaces. Proc. of The Royal Society of London 55, 361--364.
• 4: BARSKY, B., BARGTEIL, A., GARCIA, D., AND KLEIN, S. 2002. Introducing Vision-Realistic Rendering. In Proc. of EGWR 2002, 1--7.
• 5: BARSKY, B. 2003. Geometry for Analysis of Corneal Shape. In Computational Geometry: Lectures at Morningside Center of Mathematics. Amer Mathematical Society, 33--56.
• 6: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 7: Samuel Boivin , Andre Gagalowicz, Image-based rendering of diffuse, specular and glossy surfaces from a single image, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.107-116, August 2001 [doi>10.1145/383259.383270]
• 8: Richard A. Bolt, Eyes at the interface, Proceedings of the 1982 conference on Human factors in computing systems, p.360-362, March 15-17, 1982, Gaithersburg, Maryland, United States [doi>10.1145/800049.801811]
• 9: BURKHARD, D., AND SHEALY, D. 1973. Flux Density for Ray Propoagation in Gemoetrical Optics. JOSA 63, 3 (Mar.), 299--304.
• 10: CORNBLEET, S. 1984. Microwave and Optical Ray Geometry. John Wiley and Sons.
• 11: J. G. Daugman, High Confidence Visual Recognition of Persons by a Test of Statistical Independence, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.11, p.1148-1161, November 1993 [doi>10.1109/34.244676]
• 12: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 13: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 14: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 15: FLOM, L., AND SAFIR, A., 1987. Iris Recognition System. US patent 4,641,349.
• 16: Robert T. Frankot , Rama Chellappa, A Method for Enforcing Integrability in Shape from Shading Algorithms, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.10 n.4, p.439-451, July 1988 [doi>10.1109/34.3909]
• 17: Athinodoros S. Georghiades, Recovering 3-D shape and reflectance from a small number of photographs, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 18: Mark A. Halstead , Brain A. Barsky , Stanley A. Klein , Robert B. Mandell, Reconstructing curved surfaces from specular reflection patterns using spline surface fitting of normals, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.335-342, August 1996 [doi>10.1145/237170.237272]
• 19: HUTCHINSON, T., WHITE, K., REICHERT, K., AND FREY, L. 1989. Human-computer Interaction using Eye-gaze Input. IEEE TSMC 19 (Nov./Dec.), 1527--1533.
• 20: Robert J. K. Jacob, What you look at is what you get: eye movement-based interaction techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.11-18, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97246]
• 21: KAUFMAN, P., AND ALM, A., Eds. 2003. Adler's Physiology of the Eye: Clinical Application, 10th ed. Mosby.
• 22: Aaron Lefohn , Brian Budge , Peter Shirley , Richard Caruso , Erik Reinhard, An Ocularist's Approach to Human Iris Synthesis, IEEE Computer Graphics and Applications, v.23 n.6, p.70-75, November 2003 [doi>10.1109/MCG.2003.1242384]
• 23: Hendrik P. A. Lensch , Michael Goesele , Jan Kautz , Wolfgang Heidrich , Hans-Peter Seidel, Image-Based Reconstruction of Spatially Varying Materials, Proceedings of the 12th Eurographics Workshop on Rendering Techniques, p.103-114, June 25-27, 2001
• 24: LIN, Z., WONG, T.-T., AND SHUM, H.-Y. 2001. Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction. In Proc. of IEEE CVPR 2001, vol. 1, 561--567.
• 25: MAGDA, S., KRIEGMAN, D., ZICKLER, T., AND BELHUMEUR, P. 2001. Beyond Lambert: Reconstructing Surfaces with Arbitrary BRDFs. In Proc. of IEEE ICCV 01, vol. II, 391--398.
• 26: MARSCHNER, S., AND GREENBERG, D. 1997. Inverse Lighting for Photography. In Proc. of IS&T/SID CIC, 262--265.
• 27: Wojciech Matusik , Hanspeter Pfister , Addy Ngan , Paul Beardsley , Remo Ziegler , Leonard McMillan, Image-based 3D photography using opacity hulls, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 28: MITSUNAGA, T., AND NAYAR, S. 1999. Radiometric Self Calibration. In Proc. of IEEE CVPR 99, vol. 1, 1374--1380.
• 29: Sina Mostafawy , Omid Kermani , Holger Lubatschowski, Virtual Eye: Retinal Image Visualization of the Human Eye, IEEE Computer Graphics and Applications, v.17 n.1, p.8-12, January 1997 [doi>10.1109/38.576849]
• 30: NAYAR, S., AND MITSUNAGA, T. 2000. High Dynamic Range Imaging: Spatially Varying Pixel Exposures. In Proc. of IEEE CVPR 00, 1472--1479.
• 31: NISHINO, K., AND NAYAR, S. 2004. Corneal imaging system: Environment from eyes. Tech. rep., Dept. of Computer Science, Columbia University. In preparation.
• 32: Ravi Ramamoorthi , Pat Hanrahan, A signal-processing framework for inverse rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.117-128, August 2001 [doi>10.1145/383259.383271]
• 33: Mark A. Sagar , David Bullivant , Gordon D. Mallinson , Peter J. Hunter, A virtual environment and model of the eye for surgical simulation, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.205-212, July 1994 [doi>10.1145/192161.192200]
• 34: Yoichi Sato , Mark D. Wheeler , Katsushi Ikeuchi, Object shape and reflectance modeling from observation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.379-387, August 1997 [doi>10.1145/258734.258885]
• 35: Imari Sato , Yoichi Sato , Katsushi Ikeuchi, Illumination from Shadows, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.3, p.290-300, March 2003 [doi>10.1109/TPAMI.2003.1182093]
• 36: G. P. Stein, Accurate internal camera calibration using rotation, with analysis of sources of error, Proceedings of the Fifth International Conference on Computer Vision, p.230, June 20-23, 1995
• 37: Richard Szeliski , Heung-Yeung Shum, Creating full view panoramic image mosaics and environment maps, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.251-258, August 1997 [doi>10.1145/258734.258861]
• 38: Seth Teller , Matthew Antone , Zachary Bodnar , Michael Bosse , Satyan Coorg , Manish Jethwa , Neel Master, Calibrated, Registered Images of an Extended Urban Area, International Journal of Computer Vision, v.53 n.1, p.93-107, June 2003 [doi>10.1023/A:1023035826052]
• 39: TORRANCE, K., AND SPARROW, E. 1967. Theory for off-specular reflection from roughened surfaces. JOSA, 57, 1105--1114.
• 40: TSUMURA, N., DANG, M., MAKINO, T., AND MIYAKE, Y. 2003. Estimating the Directions to Light Sources Using Images of Eye for Reconstructing 3D Human Face. In Proc. of IS&T/SID's CIC, 77--81.
• 41: VON HELMHOLTZ, H. 1909. Physiologic Optics, third ed., vol. 1 and 2. Voss, Hamburg, Germany.
• 42: WOODHAM, R. 1978. Photometric Stereo: A Reflectance Map Technique for Determining Surface Orientation from a Single View. In Proc. of SPIE, vol. 155, 136--143.
• 43: YAGI, Y., AND YACHIDA, M. 1991. Real-time Generation of Environmental Map and Obstacle Avoidance using Omnidirectional Image Sensor with Conic Mirror. In Proc. of IEEE CVPR 91, 160--165.
• 44: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Intensity, color, photometry, and thresholding

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture
          Subjects: Imaging geometry

Keywords:
Eye, Illumination Estimation, Relighting