Sketching shiny surfaces: 3D shape extraction and depiction of specular surfaces

ABSTRACT

Many materials including water, plastic, and metal have specular surface characteristics. Specular reflections have commonly been considered a nuisance for the recovery of object shape. However, the way that reflections are distorted across the surface depends crucially on 3D curvature, suggesting that they could, in fact, be a useful source of information. Indeed, observers can have a vivid impression of, 3D shape when an object is perfectly mirrored (i.e., the image contains nothing but specular reflections). This leads to the question what are the underlying mechanisms of our visual system to extract this 3D shape information from a perfectly mirrored object. In this paper we propose a biologically motivated recurrent model for the extraction of visual features relevant for the perception of 3D shape information from images of mirrored objects. We qualitatively and quantitatively analyze the results of computational model simulations and show that bidirectional recurrent information processing leads to better results than pure feedforward processing. Furthermore, we utilize the model output to create a rough nonphotorealistic sketch representation of a mirrored object, which emphasizes image features that are mandatory for 3D shape perception (e.g., occluding contour and regions of high curvature). Moreover, this sketch illustrates that the model generates a representation of object features independent of the surrounding scene reflected in the mirrored object.

REFERENCES

• 1: Attneave, F. 1954. Some informational aspects of visual perception. Psychological Review 61, 183--193.
• 2: Alethea Bair , Donald House , Colin Ware, Perceptually optimizing textures for layered surfaces, Proceedings of the 2nd symposium on Applied perception in graphics and visualization, August 26-28, 2005, A Coroña, Spain [doi>10.1145/1080402.1080414]
• 3: Beck, J. and Prazdny, S. 1981. Highlights and the perception of glossiness. Perception & Psychophysics 30, 4, 407--410.
• 4: Benson, P. J. and Perrett, D. I. 1991. Perception and recognition of photographic quality facial caricatures: Implications for the recognition of natural images. European Journal of Cognitive Psychology 3, 1, 105--135.
• 5: Biederman, I. 1987. Recognition--by--components: A theory of human image understanding. Psychological Review 94, 2, 115--147.
• 6: Blake, A. 1985. Specular stereo. In Proceedings of the 9th International Joint Conference on Artificial Intelligence. 973--976.
• 7: Blake, A. and Brelstaff, G. 1988. Geometry from specularity. In Proceedings of ICCV. 394--403.
• 8: Blake, A. and Bülthoff, H. H. 1990. Does the brain know the physics of specular reflection? Nature (London) 343, 165--168.
• 9: Blake, A. and Bülthoff, H. H. 1991. Shape from specularities: Computation and psychophysics. Philosophical Transactions of the Royal Society of London B 331, 237--252.
• 10: Alfred M. Bruckstein, On shape from shading, Computer Vision, Graphics, and Image Processing, v.44 n.2, p.139-154, Nov. 1988 [doi>10.1016/S0734-189X(88)80002-1]
• 11: Bullier, J. H., McCourt, M. E., and Henry., G. H. 1988. Physiological studies of the feedback connection to the striate cortex from areas 18 and 19 of the cat. Exp. Brain Res. 70, 90--98.
• 12: Brian Cabral , Leith Casey Leedom, Imaging vector fields using line integral convolution, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.263-270, September 1993 [doi>10.1145/166117.166151]
• 13: Hong Chen , Ziqiang Liu , Chuck Rose , Yingqing Xu , Heung-Yeung Shum , David Salesin, Example-based composite sketching of human portraits, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987673]
• 14: John P. Collomosse , Peter M. Hall, Cubist Style Rendering from Photographs, IEEE Transactions on Visualization and Computer Graphics, v.9 n.4, p.443-453, October 2003 [doi>10.1109/TVCG.2003.1260739]
• 15: Cassidy J. Curtis , Sean E. Anderson , Joshua E. Seims , Kurt W. Fleischer , David H. Salesin, Computer-generated watercolor, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.421-430, August 1997 [doi>10.1145/258734.258896]
• 16: Daugman, J. 1988. Complete discrete 2d gabor transforms by neural networks for image analysis and compression. Trans. Acoustics, Speech, and Signal Proc. 36, 7, 1169--1179.
• 17: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 18: Doug DeCarlo , Anthony Santella, Stylization and abstraction of photographs, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 19: Oliver Deussen , Thomas Strothotte, Computer-generated pen-and-ink illustration of trees, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.13-18, July 2000 [doi>10.1145/344779.344792]
• 20: Do Carmo, M. P. 1976. Differential Geometry of Curves and Surfaces. Prentice Hall, Englewood Cliffs, NJ.
• 21: Frédo Durand , Victor Ostromoukhov , Mathieu Miller , Francois Duranleau , Julie Dorsey, Decoupling Strokes and High-Level Attributes for Interactive Traditional Drawing, Proceedings of the 12th Eurographics Workshop on Rendering Techniques, p.71-82, June 25-27, 2001
• 22: Dwyer, F. M. J. 1967. Adapting visual illustrations for effective learning. Harvard Educational Review 37, 250--263.
• 23: Feldman, J. and Singh, M. 2005. Information along contours and object boundaries. Psychological Review 112, 243--252.
• 24: Fleming, R., Torralba, A., and Adelson, E. Three Dimensional Shape Perception (in press). Springer, Chapter Shape From Sheen.
• 25: Fleming, R., Torralba, A., Dror, R. O., and Adelson, E. H. 2003. How image statistics drive shape-from-texture and shape-from-specularity. J. Vision 3, 9, 73--73.
• 26: Fleming, R. W., Torralba, A., and Adelson, E. H. 2004. Specular reflections and the perception of shape. J. Vision 4, 9, 798--820.
• 27: Fraisse, P. and Elkin, E. H. 1963. Etude génétique de l'influence des modes de présentation sur le seuil de reconnaissance d'objets familiers. Année Psychologique 63, 1--12.
• 28: Amy Gooch , Bruce Gooch , Peter Shirley , Elaine Cohen, A non-photorealistic lighting model for automatic technical illustration, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.447-452, July 1998 [doi>10.1145/280814.280950]
• 29: Bruce Gooch , Erik Reinhard , Amy Gooch, Human facial illustrations: Creation and psychophysical evaluation, ACM Transactions on Graphics (TOG), v.23 n.1, p.27-44, January 2004 [doi>10.1145/966131.966133]
• 30: Grossberg, S. and Mingolla, E. 1985. Neural dynamics of form perception: Boundary completion, illusory figures and neon color spreading. Psych. Rev. 92, 2, 173--211.
• 31: Hadamard, J. 1902. Sur les problémes aux dérivées partielles et leur signification physique. Princeton University Bulletin, 49--52.
• 32: Hartung, B. and Kersten, D. 2003. How does the perception of shape interact with the perception of shiny material? J. Vis. 3, 9 (10), 59--59.
• 33: Alejo Hausner, Simulating decorative mosaics, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.573-580, August 2001 [doi>10.1145/383259.383327]
• 34: James Hays , Irfan Essa, Image and video based painterly animation, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987676]
• 35: Aaron Hertzmann, Painterly rendering with curved brush strokes of multiple sizes, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.453-460, July 1998 [doi>10.1145/280814.280951]
• 36: Horn, B. and Brooks, M. 1985. Shape and source from shading. In Proceedings of International Joint Conference on Artificial Intelligence. 932--936.
• 37: Berthold K. P. Horn , Michael J. Brooks, Shape from shading, MIT Press, Cambridge, MA, 1989
• 38: Hubel, D. H. and Wiesel, T. N. 1968. Receptive fields and functional architecture of monkey striate cortex. J. Physiology (London) 195, 215--243.
• 39: Ikeuchi, K. 1981. Recognition of 3-d objects using the extended gaussian image. In Proc. of the 7th International Joint Conference on Artificial Intelligence. 595--600.
• 40: Interrante, V. and Kim, S. 2001. Investigating the effect of texture orientation on shape perception. Proc. Human Vision and Electronic Imaging 6, 330--339.
• 41: Interrante, V., Kim, S., and Hagh-Shenas, H. 2002. Conveying 3d shape with texture: Recent advances and experimental findings. Proc. Human Vision and Electronic Imaging 7, 197--206.
• 42: Wang Jin , Bao Hujun , Zhou Weihua , Peng Qunsheng , Xu Yingqing, Automatic image-based pencil sketch rendering, Journal of Computer Science and Technology, v.17 n.3, p.347-355, May 2002 [doi>10.1007/BF02947313]
• 43: Kang, H., He, W., Chui, C., and Chakraborty, U. 2005. Interactive sketch generation. The Visual Computer 21, 9, 821--830.
• 44: Kim, S., Hagh-Shenas, H., and Interrante, V. 2003. Showing shape with texture: Two directions seem better than one. In Proc. Human Vision and Electronic Imaging 8, 332--339.
• 45: Koenderink, J. and van Doorn, A. 1980. Photometric invariants related to solid shape. Optica Acta 27, 7, 981--996.
• 46: Koenderink, J. J. 1984. What does the occluding contour tell us about solid shape. Perc. 13, 312--330.
• 47: Peter Litwinowicz, Processing images and video for an impressionist effect, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.407-414, August 1997 [doi>10.1145/258734.258893]
• 48: Longuet-Higgins, M. S. 1960. Reflection and refraction at a random moving surface. i. pattern and paths of specular points. Journal of the Optical Society of America 50, 9, 838--844.
• 49: Neumann, H. and Sepp, W. 1999. Recurrent v1-v2 interactions in early boundary processing. Biol. Cybern. 81, 425--444.
• 50: Norman, J. F., Todd, J. T., and Orban, G. A. 2004. Perception of three-dimensional shape from specular highlights, deformations of shading, and other types of visual information. Psychological Science 15, 565--570.
• 51: Michael Oren , Shree K. Nayar, A Theory of Specular Surface Geometry, International Journal of Computer Vision, v.24 n.2, p.105-124, Sept. 1997 [doi>10.1023/A:1007954719939]
• 52: Rhodes, G., Brennan, S., and Carey, S. 1987. Identification and ratings of caricatures: Implications for mental representations of faces. Cognitive Psychology 19, 473--497.
• 53: Ryan, T. and Schwartz, C. 1956. Speed of perception as a function of mode of presentation. American Journal of Psychology 69, 60--69.
• 54: Dimitrios Samaras , Dimitris Metaxas, Coupled Lighting Direction and Shape Estimation from Single Images, Proceedings of the International Conference on Computer Vision-Volume 2, p.868, September 20-25, 1999
• 55: A. C. Sanderson , L. E. Weiss , S. K. Nyar, Structured Highlight Inspection of Specular Surfaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.10 n.1, p.44-55, January 1988 [doi>10.1109/34.3866]
• 56: Silvio Savarese , Li Fei-Fei , Pietro Perona, What do reflections tell us about the shape of a mirror?, Proceedings of the 1st Symposium on Applied perception in graphics and visualization, August 07-08, 2004, Los Angeles, California [doi>10.1145/1012551.1012571]
• 57: Savarese, S. and Perona, P. 2001. Local analysis for 3d reconstruction of specular surfaces. In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 738--745.
• 58: Silvio Savarese , Pietro Perona, Local Analysis for 3D Reconstruction of Specular Surfaces - Part II, Proceedings of the 7th European Conference on Computer Vision-Part II, p.759-774, May 28-31, 2002
• 59: Michio Shiraishi , Yasushi Yamaguchi, An algorithm for automatic painterly rendering based on local source image approximation, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.53-58, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340923]
• 60: Detlev Stalling , Hans-Christian Hege, Fast and resolution independent line integral convolution, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.249-256, September 1995 [doi>10.1145/218380.218448]
• 61: Tinbergen, N. 1951. The Study of Instinct. Clarendon, Oxford.
• 62: Tinbergen, N. and Perdeck, A. C. 1950. On the stimulus situation releasing the begging response in the newly hatched herring gull chick (larus argentatus argentatus pont). Behaviour 3, 1--39.
• 63: Todd, J. and Mingolla, E. 1983. Perception of surface curvature and direction of illumination from patterns of shading. Journal of Experimental Psychology: Human Perception and Performance 10, 740--745.
• 64: Todd, J., Norman, J., Koenderink, J., and Kappers, A. M. L. 1997. Effects of texture, illumination and surface reflectance on stereoscopic shape perception. Perception 26, 806--822.
• 65: Todd, J. T. 2004. The visual perception of 3d shape. Trends in Cognitive Sciences 9, 3, 115--121.
• 66: Todd, J. T., Norman, J. F., and Mingolla, E. 2004. Lightness constancy in the presence of specular highlights. Psychological Science 15, 1, 33--39.
• 67: von der Heydt, R., Peterhans, E., and Baumgartner, G. 1984. Illusory contours and cortical neuron responses. Science 224, 1260--1262.
• 68: Shigefumi Yamamoto , Xiaoyang Mao , Atsumi Imamiya, Enhanced LIC Pencil Filter, Proceedings of the International Conference on Computer Graphics, Imaging and Visualization (CGIV'04), p.251-256, July 26-29, 2004 [doi>10.1109/CGIV.2004.21]
• 69: Ruo Zhang , Ping-Sing Tsai , James Edwin Cryer , Mubarak Shah, Shape from Shading: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.21 n.8, p.690-706, August 1999 [doi>10.1109/34.784284]
• 70: Jiang Yu Zheng , Akio Murata, Acquiring a Complete 3D Model from Specular Motion under the Illumination of Circular-Shaped Light Sources, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.913-920, August 2000 [doi>10.1109/34.868691]
• 71: Qinfen Zheng , Rama Chellappa, Estimation of Illuminant Direction, Albedo, and Shape from Shading, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13 n.7, p.680-702, July 1991 [doi>10.1109/34.85658]
• 72: Andrew Zisserman , Peter Giblin , Andrew Blake, The information available to a moving observer from specularities, Image and Vision Computing, v.7 n.1, p.38-42, February 1989 [doi>10.1016/0262-8856(89)90018-8]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Invariants

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation
          Subjects: Line and curve generation

Keywords:
3D shape perception, Nonphotorealistic rendering, human visual perception, perfectly specular surfaces