Depth-of-field-based alpha-matte extraction

ABSTRACT

In compositing applications, objects depicted in images frequently have to be separated from their background, so that they can be placed in a new environment. Alpha mattes are important tools aiding the selection of objects, but cannot normally be created in a fully automatic way. We present an algorithm that requires as input two images---one where the object is in focus, and one where the background is in focus---and then automatically produces an alpha matte indicating which pixels belong to the object. This algorithm is inspired by human visual processing and involves nonlinear response compression, center-surround mechanisms as well as a filling-in stage. The output can then be refined with standard computer vision techniques.

REFERENCES

• 1: Adelson, E. H. 1982. Saturation and adaptation in the rod system. Vision Research 22, 1299--1312.
• 2: Boykov, Y., and Jolly, M.-P. 2001. Interactive graph cuts for optimal boundary and region segmentation of objects in nd images. In Proceedings of IEEE International Conference on Computer Vision.
• 3: Vicent Caselles , Ron Kimmel , Guillermo Sapiro, Geodesic Active Contours, International Journal of Computer Vision, v.22 n.1, p.61-79, Feb./March 1997 [doi>10.1023/A:1007979827043]
• 4: Chuang, Y.-Y., Curless, B., Salesin, D., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proceedings of Computer Vision and Pattern Recognition (CVPR 2001), vol. II, 264--271.
• 5: Yung-Yu Chuang , Aseem Agarwala , Brian Curless , David H. Salesin , Richard Szeliski, Video matting of complex scenes, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 6: Dakin, S. C., and Bex, P. J. 2003. Natural image statistics mediate brightness filling in. Proceedings of the Royal Society of London, B 270, 2341--2348.
• 7: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 8: Dowling, J. E. 1987. The retina: an approachable part of the brain. Belknap Press, Cambridge.
• 9: James H. Elder, Are Edges Incomplete?, International Journal of Computer Vision, v.34 n.2-3, p.97-122, Nov. 1999 [doi>10.1023/A:1008183703117]
• 10: John Ens , Peter Lawrence, An Investigation of Methods for Determining Depth from Focus, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.2, p.97-108, February 1993 [doi>10.1109/34.192482]
• 11: Greig, D., Porteous, B., and Seheult, A. 1989. Exact MAP estimation for binary images. Journal of the Royal Statistical Society, B 51, 271--279.
• 12: Grossberg, S., and Mingolla, E. 1985. Neural dynamics of form perception: boundary adaptation, illusory figures, and neon color spreading. Psychological Review 92, 173--211.
• 13: Hood, D. C., and Finkelstein, M. A. 1979. Comparison of changes in sensitivity and sensation: implications for the response-intensity function of the human photopic system. Journal of Experimental Pssychology: Human Perceptual Performance 5, 391--405.
• 14: Horn, B. K. P., 1968. Focusing. memo 160.
• 15: ITU. 1990. International Telecommunication Union ITU-R Recommendation BT.709, Basic Parameter Values for the HDTV Standard for the Studio and for International Programme Exchange. Geneva. Formerly CCIR Rec. 709.
• 16: Kleinschmidt, J., and Dowling, J. E. 1975. Intracellular recordings from gecko photoreceptors during light and dark adaptation. J gen Physiol 66, 617--648.
• 17: Krauskopf, J. 1963. Effect of retinal stabilization on the appearance of heterochromatic targets. J Opt Soc Am 53, 741--744.
• 18: Kuffler, S. W. 1953. Discharge patterns and functional organization of mammalian retina. Journal of Neurophysiology 16, 37--68.
• 19: Yin Li , Jian Sun , Chi-Keung Tang , Heung-Yeung Shum, Lazy snapping, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 20: David Marr, Vision: A Computational Investigation into the Human Representation and Processing of Visual Information, Henry Holt and Co., Inc., New York, NY, 1982
• 21: Marshall, J. A., Burbeck, C. A., Ariely, D., Rolland, J. P., and Martin, K. E. 1996. Occlusion edge blur: a cue to relative visual depth. Journal of the Optical Society of America A 13, 681--688.
• 22: Morgan McGuire , Wojciech Matusik , Hanspeter Pfister , John F. Hughes , Frédo Durand, Defocus video matting, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 23: Tomoo Mitsunaga , Taku Yokoyama , Takashi Totsuka, AutoKey: human assisted key extraction, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.265-272, September 1995 [doi>10.1145/218380.218450]
• 24: Eric N. Mortensen , William A. Barrett, Intelligent scissors for image composition, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.191-198, September 1995 [doi>10.1145/218380.218442]
• 25: Mortensen, E. N., and Barrett, W. A. 1999. Toboganbased intelligent scissors with a four parameter edge model. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 452--458.
• 26: Naka, K. I., and Rushton, W. A. H. 1966. S-potentials from luminosity units in the retina of fish (cyprinidae). J Physiol 185, 587--599.
• 27: S. K. Nayar , Y. Nakagawa, Shape from Focus, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.16 n.8, p.824-831, August 1994 [doi>10.1109/34.308479]
• 28: Neumann, H., Pessoa, L., and Hansen, T. 2001. Visual filling-in for computing perceptual surface properties. Biological Cybernetics 85, 355--369.
• 29: Palmer, S. E. 1999. Vision Science: Photons to Phenomenology. The MIT Press, Cambridge, Massachusetts.
• 30: Sumanta N. Pattanaik , Jack Tumblin , Hector Yee , Donald P. Greenberg, Time-dependent visual adaptation for fast realistic image display, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.47-54, July 2000 [doi>10.1145/344779.344810]
• 31: A. P. Pentland, A new sense for depth of field, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.9 n.4, p.523-531, July 1987 [doi>10.1109/TPAMI.1987.4767940]
• 32: P. Perona , J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.7, p.629-639, July 1990 [doi>10.1109/34.56205]
• 33: Erik Reinhard , Kate Devlin, Dynamic Range Reduction Inspired by Photoreceptor Physiology, IEEE Transactions on Visualization and Computer Graphics, v.11 n.1, p.13-24, January 2005 [doi>10.1109/TVCG.2005.9]
• 34: Erik Reinhard , Michael Stark , Peter Shirley , James Ferwerda, Photographic tone reproduction for digital images, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 35: Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers, San Francisco.
• 36: Carsten Rother , Vladimir Kolmogorov , Andrew Blake, "GrabCut": interactive foreground extraction using iterated graph cuts, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 37: Ruderman, D. L. 1997. Origins of scaling in natural images. Vision Research 37, 3385--3398.
• 38: Ruderman, D. L. 1997. The statistics of natural images. Network: Computation in Neural Systems 5, 4, 517--548.
• 39: Schlick, C. 1994. Quantization techniques for the visualization of high dynamic range pictures. In Photorealistic Rendering Techniques, Springer-Verlag Berlin Heidelberg New York, P. Shirley, G. Sakas, and S. Müller, Eds., 7--20.
• 40: Murali Subbarao , Gopal Surya, Depth from defocus: a spatial domain approach, International Journal of Computer Vision, v.13 n.3, p.271-294, Dec. 1994 [doi>10.1007/BF02028349]
• 41: Subbarao, M. 1988. Parallel depth recovery by changing camera parameters. In Proc. of the International Conference on Computer Vision, 149--155.
• 42: Jian Sun , Jiaya Jia , Chi-Keung Tang , Heung-Yeung Shum, Poisson matting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 43: Jack Tumblin , Jessica K. Hodgins , Brian K. Guenter, Two methods for display of high contrast images, ACM Transactions on Graphics (TOG), v.18 n.1, p.56-94, Jan. 1999 [doi>10.1145/300776.300783]
• 44: Walls, G. 1954. The filling-in process. American Journal of Optometry 31, 329--340.
• 45: James Z. Wang , Jia Li , Robert M. Gray , Gio Wiederhold, Unsupervised Multiresolution Segmentation for Images with Low Depth of Field, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.1, p.85-90, January 2001 [doi>10.1109/34.899949]
• 46: Ward, G. 2003. Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures. Journal of Graphics Tools 8, 2, 17--30.
• 47: Masahiro Watanabe , Shree K. Nayar, Rational Filters for Passive Depth from Defocus, International Journal of Computer Vision, v.27 n.3, p.203-225, May 1, 1998 [doi>10.1023/A:1007905828438]
• 48: Joachim Weickert, A Review of Nonlinear Diffusion Filtering, Proceedings of the First International Conference on Scale-Space Theory in Computer Vision, p.3-28, July 02-04, 1997
• 49: Witkin, A. P. 1983. Scale-space filtering. In Proceedings of the Eighth International Joint Conference on Artificial Intelligence, 2, 1019--1022.
• 50: Won, C. S., Pyun, K., and Gray, R. M. 2002. Automatic object segmentation in images with low depth of field. In Proceedings of IEEE International Conference on Image Processing, III.805 - III.808.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
        I.4.10 Image Representation

Keywords:
alpha-mattes, depth of field, high dynamic range imaging, human visual perception