Deep photo: model-based photograph enhancement and viewing

ABSTRACT

In this paper, we introduce a novel system for browsing, enhancing, and manipulating casual outdoor photographs by combining them with already existing georeferenced digital terrain and urban models. A simple interactive registration process is used to align a photograph with such a model. Once the photograph and the model have been registered, an abundance of information, such as depth, texture, and GIS data, becomes immediately available to our system. This information, in turn, enables a variety of operations, ranging from dehazing and relighting the photograph, to novel view synthesis, and overlaying with geographic information. We describe the implementation of a number of these applications and discuss possible extensions. Our results show that augmenting photographs with already available 3D models of the world supports a wide variety of new ways for us to experience and interact with our everyday snapshots.

REFERENCES

• 1: Michael Ashikhmin, Synthesizing natural textures, Proceedings of the 2001 symposium on Interactive 3D graphics, p.217-226, March 2001 [doi>10.1145/364338.364405]
• 2: Chen, B., Ramos, G., Ofek, E., Cohen, M., Drucker, S., and Nister, D. 2008. Interactive techniques for registering images to digital terrain and building models. Microsoft Research Technical Report MSR-TR-2008-115.
• 3: Peter L. Cho, 3D Organization of 2D Urban Imagery, Proceedings of the 36th Applied Imagery Pattern Recognition Workshop (aipr 2007), p.3-8, October 10-12, 2007 [doi>10.1109/AIPR.2007.12]
• 4: A. Criminisi , I. Reid , A. Zisserman, Single View Metrology, International Journal of Computer Vision, v.40 n.2, p.123-148, Nov. 2000 [doi>10.1023/A:1026598000963]
• 5: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 6: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 7: Iddo Drori , Daniel Cohen-Or , Hezy Yeshurun, Fragment-based image completion, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 8: E-on Software, 2008. Vue 6 Infinite. http://www.e-onsoftware.com/products/vue/vue_6_infinite.
• 9: Alexei A. Efros , Thomas K. Leung, Texture Synthesis by Non-Parametric Sampling, Proceedings of the International Conference on Computer Vision-Volume 2, p.1033, September 20-25, 1999
• 10: Raanan Fattal, Single image dehazing, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 11: Christian Früh , Avideh Zakhor, Constructing 3D City Models by Merging Aerial and Ground Views, IEEE Computer Graphics and Applications, v.23 n.6, p.52-61, November 2003 [doi>10.1109/MCG.2003.1242382]
• 12: A. Gruen , Thomas S. Huang, Calibration and Orientation of Cameras in Computer Vision, Springer-Verlag New York, Inc., Secaucus, NJ, 2001
• 13: James Hays , Alexei A. Efros, Scene completion using millions of photographs, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 14: Aaron Hertzmann , Charles E. Jacobs , Nuria Oliver , Brian Curless , David H. Salesin, Image analogies, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.327-340, August 2001 [doi>10.1145/383259.383295]
• 15: Derek Hoiem , Alexei A. Efros , Martial Hebert, Automatic photo pop-up, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 16: Youichi Horry , Ken-Ichi Anjyo , Kiyoshi Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.225-232, August 1997 [doi>10.1145/258734.258854]
• 17: Kaftory, R., Schechner, Y. Y., and Zeevi, Y. Y. 2007. Variational distance-dependent image restoration. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2007, 1--8.
• 18: Kang, S. B. 1998. Depth painting for image-based rendering applications. Tech. rep., Compaq Cambridge Research Lab.
• 19: Jean-François Lalonde , Derek Hoiem , Alexei A. Efros , Carsten Rother , John Winn , Antonio Criminisi, Photo clip art, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 20: Céline Loscos , George Drettakis , Luc Robert, Interactive Virtual Relighting of Real Scenes, IEEE Transactions on Visualization and Computer Graphics, v.6 n.4, p.289-305, October 2000 [doi>10.1109/2945.895874]
• 21: McCartney, E. J. 1976. Optics of the Atmosphere: Scattering by Molecules and Particles. John Wiley and Sons, New York, NY, USA.
• 22: Narasimhan, S. G., and Nayar, S. K. 2003. Contrast restoration of weather degraded images. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 6, 713--724.
• 23: Narasimhan, S. G., and Nayar, S. K. 2003. Interactive (de)weathering of an image using physical models. IEEE Workshop on Color and Photometric Methods in Computer Vision.
• 24: NASA, 2008. The landsat program. http://landsat.gsfc.nasa.gov/.
• 25: NASA, 2008. Shuttle radar topography mission. http://www2.jpl.nasa.gov/srtm/.
• 26: Shree K. Nayar , Srinivasa G. Narasimhan, Vision in Bad Weather, Proceedings of the International Conference on Computer Vision-Volume 2, p.820, September 20-25, 1999
• 27: David Nistér , Henrik Stewénius, A Minimal Solution to the Generalised 3-Point Pose Problem , Journal of Mathematical Imaging and Vision, v.27 n.1, p.67-79, January 2007 [doi>10.1007/s10851-006-0450-y]
• 28: Oakley, J. P., and Satherley, B. L. 1998. Improving image quality in poor visibility conditions using a physical model for contrast degradation. IEEE Transactions on Image Processing 7, 2, 167--179.
• 29: Byong Mok Oh , Max Chen , Julie Dorsey , Frédo Durand, Image-based modeling and photo editing, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.433-442, August 2001 [doi>10.1145/383259.383310]
• 30: Reichmann, M., 2001. The art of photography. http://www.luminous-landscape.com/essays/theartof.shtml.
• 31: Schechner, Y. Y., and Averbuch, Y. 2007. Regularized image recovery in scattering media. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 9, 1655--1660.
• 32: Schechner, Y. Y., Narasimhan, S. G., and Nayar, S. K. 2003. Polarization-based vision through haze. Applied Optics 42, 3, 511--525.
• 33: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• 34: H. -Y. Shum , M. Han , R. Szeliski, Interactive Construction of 3D Models from Panoramic Mosaics, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.427, June 23-25, 1998
• 35: Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 36: Stamos, I., and Allen, P. K. 2000. 3-D model construction using range and image data. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1998, 531--536.
• 37: Kalyan Sunkavalli , Wojciech Matusik , Hanspeter Pfister , Szymon Rusinkiewicz, Factored time-lapse video, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 38: Tan, R. T. 2008. Visibility in bad weather from a single image. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2008, to appear.
• 39: Kentaro Toyama , Ron Logan , Asta Roseway, Geographic location tags on digital images, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957046]
• 40: Jue Wang , Maneesh Agrawala , Michael F. Cohen, Soft scissors: an interactive tool for realtime high quality matting, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 41: Yizhou Yu , Jitendra Malik, Recovering photometric properties of architectural scenes from photographs, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.207-217, July 1998 [doi>10.1145/280814.280874]
• 42: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]
• 43: Zhang, L., Dugas-Phocion, G., Samson, J.-S., and Seitz, S. M. 2002. Single-view modelling of free-form scenes. The Journal of Visualization and Computer Animation 13, 4, 225--235.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Modeling and recovery of physical attributes

General Terms:
Design, Performance

Keywords:
dehazing, image completion, image-based modeling, image-based rendering, photo browsing, relighting