Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

We describe a learning-based method for low-level vision problems—estimating scenes from images. We generate a synthetic world of scenes and their corresponding rendered images, modeling their relationships with a Markov network. Bayesian belief propagation allows us to efficiently find a local maximum of the posterior probability for the scene, given an image. We call this approach VISTA—Vision by Image/Scene TrAining.

We apply VISTA to the “super-resolution” problem (estimating high frequency details from a low-resolution image), showing good results. To illustrate the potential breadth of the technique, we also apply it in two other problem domains, both simplified. We learn to distinguish shading from reflectance variations in a single image under particular lighting conditions. For the motion estimation problem in a “blobs world”, we show figure/ground discrimination, solution of the aperture problem, and filling-in arising from application of the same probabilistic machinery.

REFERENCES

• 1: Adelson, E.H. 1995. Personal communication.
• 2: Barrow, H.G. and Tenenbaum, J.M. 1981. Computational vision. <i>Proc. IEEE</i>, 69(5):572-595.
• 3: Bell, A.J. and Sejnowski, T.J. 1997. The independent components of natural scenes are edge filters. <i>Vision Research</i>, 37(23):3327- 3338.
• 4: Berger, J.O. 1985. <i>Statistical Decision Theory and Bayesian Analysis </i>. Springer: Berlin.
• 5: Besag, J. 1974. Spatial interaction and the statistical analysis of lattice systems (with discussion). <i>J. Royal Statist. Soc. B</i>, 36:192-326.
• 6: Binford, T., Levitt, T. and Mann, W. 1988. Bayesian inference in model-based machine vision. In <i>Uncertainty in Artificial Intelligence </i>, J.F. Lemmer and L.M. Kanal (Eds.), Morgan Kaufmann: Los Alos, CA.
• 7: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 8: Burt, P.J. and Adelson, E.H. 1983. The Laplacian pyramid as a compact image code. <i>IEEE Trans. Comm.</i>, 31(4):532-540.
• 9: Carandini, M. and Heeger, D.J. 1994, Summation and division by neurons in primate visual cortex. <i>Science</i>, 264:1333-1336.
• 10: J. S. de Bonet , P. Viola, Texture Recognition Using a Non-Parametric Multi-Scale Statistical Model, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.641, June 23-25, 1998
• 11: David J. Field, What is the goal of sensory coding?, Neural Computation, v.6 n.4, p.559-601, July 1994 [doi>10.1162/neco.1994.6.4.559]
• 12: Freeman, W.T. 1994. The generic viewpoint assumption in a framework for visual perception. <i>Nature</i>, 368(6471):542-545.
• 13: Freeman, W.T., Haddon, J.A., and Pasztor, E.C. 2001. Learning motion analysis. In <i>Statistical Theories of the Brain</i>, R. Rao, B. Olshausen, and M. Lewicki (Eds.), MIT Press, Cambridge, MA. See also http://www.merl.com/reports/TR2000-32.
• 14: William T. Freeman , Egon C. Pasztor, Learning to estimate scenes from images, Proceedings of the 1998 conference on Advances in neural information processing systems II, p.775-781, July 1999
• 15: William T. Freeman , Paul A. Viola, Bayesian model of surface perception, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.787-793, July 1998, Denver, Colorado, United States
• 16: Brendan J. Frey, Graphical models for machine learning and digital communication, MIT Press, Cambridge, MA, 1998
• 17: Frey, B.J. 2000. Filling in scenes by propagating probabilities through layers and into appearance models. In <i>Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition</i>, Hilton Head Island, S.C.
• 18: Davi Geiger , Federico Girosi, Parallel and Deterministic Algorithms from MRFs: Surface Reconstruction, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13 n.5, p.401-412, May 1991 [doi>10.1109/34.134040]
• 19: Geman, S. and Geman, D. 1984. Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images. <i>IEEE Pattern Analysis and Machine Intelligence</i>, 6:721-741.
• 20: David J. Heeger , James R. Bergen, Pyramid-based texture analysis/synthesis, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.229-238, September 1995 [doi>10.1145/218380.218446]
• 21: Berthold K. Horn, Robot Vision, McGraw-Hill Higher Education, 1986
• 22: Berthold K. P. Horn , Michael J. Brooks, Shape from shading, MIT Press, Cambridge, MA, 1989
• 23: Hurlbert, A.C. and Poggio, T.A. 1988. Synthesizing a color algorithm from examples. <i>Science</i>, 239:482-485.
• 24: Michael Isard , Andrew Blake, Contour Tracking by Stochastic Propagation of Conditional Density, Proceedings of the 4th European Conference on Computer Vision-Volume I, p.343-356, April 15-18, 1996
• 25: Jahne, B. 1991. <i>Digital Image Processing</i>. Springer-Verlag: Berlin.
• 26: Michael I. Jordan, Learning in graphical models, MIT Press, Cambridge, MA, 1999
• 27: Jordan, M.I., Kearns, M.J., and Solla, S.A. (Eds.), MIT Press, Cambridge, MA. See also http://www.merl.com/reports/TR98-05.
• 28: Kersten, D., O'Toole, A.J., Sereno, M.E., Knill, D.C., and Anderson, J.A. 1987. Associative learning of scene parameters from images. <i>Applied Optics</i>, 26(23):4999-5006.
• 29: J Kittler , J Illingworth, Relaxation labelling algorithms-a review, Image and Vision Computing, v.3 n.4, p.206-216, Nov. 1986 [doi>10.1016/0262-8856(85)90009-5]
• 30: David C. Knill , Whitman Richards, Perception as Bayesian inference, Cambridge University Press, New York, NY, 1996
• 31: Kschischang, F.R. and Frey, B.J. 1998. Iterative decoding of compound codes by probability propagation in graphical models. <i>IEEE Journal on Selected Areas in Communication</i>, 16(2):219-230.
• 32: Michael S. Landy , J. Anthony Movshon, Computational models of visual processing, MIT Press, Cambridge, MA, 1991
• 33: Luettgen, M.R., Karl, W.C., and Willsky, A.S. 1994. Efficient multi-scale regularization with applications to the computation of optical flow. <i>IEEE Trans. Image Processing</i>, 3(1):41-64.
• 34: McEliece, R., MacKay, D., and Cheng, J. 1998. Turbo decoding as as an instance of pearl's 'Belief Propagation' algorithm. <i>IEEE J. on Sel. Areas in Comm.</i>, 16(2):140-152.
• 35: Olshausen, B.A. and Field, D.J. 1996. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. <i>Nature</i>, 381:607-609.
• 36: Judea Pearl, Probabilistic reasoning in intelligent systems: networks of plausible inference, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1988
• 37: Alex Pentland , Bradley Horowitz, A practical approach to fractal-based image compression, Digital images and human vision, MIT Press, Cambridge, MA, 1993
• 38: Tomaso Poggio , Vincent Torre , Christof Koch, Computational vision and regularization theory, Nature, v.317 n.26, p.314-319, Sept. 1985 [doi>10.1038/317314a0]
• 39: Polvere, M. 1998. Mars v. 1.0, A quadtree based fractal image coder/decoder. http://inls.ucsd.edu/y/Fractals/.
• 40: Rosenfeld, A., Hummel, R.A., and Zucker, S.W. 1976. Scene labeling by relaxation operations. <i>IEEE Trans. Systems, Man, Cybern</i>, 6(6):420-433.
• 41: Saund, E. 1999. Perceptual organization of occluding contours generated by opaque surfaces. In <i>Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition.</i>, Ft. Collins, CO.
• 42: Schultz, R.R. and Stevenson, R.L. 1994. A Bayesian approach to image expansion for improved definition. <i>IEEE Trans. Image Processing </i>, 3(3):233-242.
• 43: Simoncelli, E.P. 1997. Statistical models for images: Compression, restoration and synthesis. In <i>31st Asilomar Conf. on Sig., Sys. and Computers</i>, Pacific Grove, CA.
• 44: Sinha, P. and Adelson, E.H. 1993. Recovering reflectance and illumination in a world of painted polyhedra. In <i>Proc. 4th Intl. Conf. Comp. Vis.</i>, pp. 156-163.
• 45: Padhraic Smyth , David Heckerman , Michael I. Jordan, Probabilistic independence networks for hidden Markov probability models, Neural Computation, v.9 n.2, p.227-269, Feb. 15, 1997 [doi>10.1162/neco.1997.9.2.227]
• 46: Richard Szeliski, Bayesian Modeling of Uncertainty in Low-Level Vision, Kluwer Academic Publishers, Norwell, MA, 1989
• 47: Weiss, Y. 1997. Interpreting images by propagating Bayesian beliefs. <i>Adv. in Neural Information Processing Systems</i>, Vol. 9. pp. 908- 915.
• 48: Yair Weiss, Belief Propagation and Revision in Networks with Loops, Massachusetts Institute of Technology, Cambridge, MA, 1997
• 49: Weiss, Y. and Freeman, W.T. 1999. Correctness of belief propagation in Gaussian graphical models of arbitrary topology. Technical Report UCB.CSD-99-1046, Berkeley Computer Science Dept. www.cs.berkeley.edu/~yweiss/gaussTR. ps.gz.
• 50: Weiss, Y. and Freeman, W.T. 2001. On the optimality of solutions of the max-product belief propagation algorithm in arbitrary graphs. <i>IEEE Trans. Info. Theory</i>. Special issue on codes on Graphs and Iterative Algorithms. See also: http://www.merl.com/reports/TR 99-39.
• 51: Yedidia, J.S., Freeman, W.T., and Weiss, Y. 2000. Generalized belief propagation. Technical Report 2000-26, MERL, Mitsubishi Electric Research Labs., www.merl.com.
• 52: Song Chun Zhu , David Mumford, Prior Learning and Gibbs Reaction-Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.11, p.1236-1250, November 1997 [doi>10.1109/34.632983]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION

General Terms:
Design, Experimentation, Theory

Keywords:
belief propagation, low-level vision, motion estimation, shading and reflectance, super-resolution, vision and learning