Image-based modeling and photo editing

ABSTRACT

We present an image-based modeling and editing system that takes a single photo as input. We represent a scene as a layered collection of depth images, where each pixel encodes both color and depth. Starting from an input image, we employ a suite of user-assisted techniques, based on a painting metaphor, to assign depths and extract layers. We introduce two specific editing operations. The first, a “clone brushing tool,” permits the distortion-free copying of parts of a picture, by using a parameterization optimization technique. The second, a “texture-illuminance decoupling filter,” discounts the effect of illumination on uniformly textured areas, by decoupling large- and small-scale features via bilateral filtering. Our system enables editing from different viewpoints, extracting and grouping of image-based objects, and modifying the shape, color, and illumination of these objects.

REFERENCES

• BV99: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• Can: Canoma. http://www.canoma.com.
• Che95: Shenchang Eric Chen, QuickTime VR: an image-based approach to virtual environment navigation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.29-38, September 1995 [doi>10.1145/218380.218395]
• CW93: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• Deb98: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• DM97: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• DRB97: George Drettakis , Luc Robert , Sylvain Bougnoux, Interactive Common Illumination for Computer Augmented Reality, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.45-56, June 16-18, 1997
• DTM96: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• EG01: James H. Elder , Richard M. Goldberg, Image Editing in the Contour Domain, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.3, p.291-296, March 2001 [doi>10.1109/34.910881]
• FGR93: A. Fournier, A. Gunawan, and C. Romanzin. Common illumination between real and computer generated scenes. Graphics Interface, 1993.
• FLR+95: O. Faugeras, S. Laveau, L. Robert, G. Csurka, and C. Zeller. 3-d reconstruction of urban scenes from sequences of images. In A. Gruen, O. Kuebler, and P. Agouris, editors, Automatic Extraction of Man-Made Objects from Aerial and Space Images. Birkhauser, 1995.
• GDCV98: Jonas Gomes , Lucia Darsa , Bruno Costa , Luiz Velho, Warping and morphing of graphical objects, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• GGSC96: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• HAA97: Youichi Horry , Ken-Ichi Anjyo , Kiyoshi Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.225-232, August 1997 [doi>10.1145/258734.258854]
• HH90: Pat Hanrahan , Paul Haeberli, Direct WYSIWYG painting and texturing on 3D shapes, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.215-223, September 1990, Dallas, TX, USA
• IMT99: Takeo Igarashi , Satoshi Matsuoka , Hidehiko Tanaka, Teddy: a sketching interface for 3D freeform design, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.409-416, July 1999 [doi>10.1145/311535.311602]
• Kan98: S. Kang. Depth painting for image-based rendering applications. Tech. report, CRL, Compaq Cambridge Research Lab, 1998. http://www.research.microsoft.com/Users/sbkang/publications/index.html.
• LCZ99: D. Liebowitz, A. Criminisi, and A. Zisserman. Creating architectural models from images. In Proc. of Eurographics, 1999.
• LDR00: Céline Loscos , George Drettakis , Luc Robert, Interactive Virtual Relighting of Real Scenes, IEEE Transactions on Visualization and Computer Graphics, v.6 n.4, p.289-305, October 2000 [doi>10.1109/2945.895874]
• LF94: S. Laveau and O. Faugeras. 3-D scene representation as a collection of images and fundamental matrices. In Proc. of 12th Int. Conf. on Pattern Recognition, volume 1, pages 689-691, 1994.
• LFD+99: C. Loscos, M.C. Frasson, G. Drettakis, B. Walter, X. Granier, and P. Poulin. Interactive virtual relighting and remodeling of real scenes. Eurographics Rendering Workshop, 1999.
• LH96: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• LM98: Bruno Lévy , Jean-Laurent Mallet, Non-distorted texture mapping for sheared triangulated meshes, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.343-352, July 1998 [doi>10.1145/280814.280930]
• Mal89: Jean-Laurent Mallet, Discrete smooth interpolation, ACM Transactions on Graphics (TOG), v.8 n.2, p.121-144, April 1989 [doi>10.1145/62054.62057]
• MB95: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• McM97: Leonard McMillan, Jr., An image-based approach to three-dimensional computer graphics, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1997
• Met: MetaCreations. http://www.metacreations.com.
• MMB97: William R. Mark , Leonard McMillan , Gary Bishop, Post-rendering 3D warping, Proceedings of the 1997 symposium on Interactive 3D graphics, p.7-ff., April 27-30, 1997, Providence, Rhode Island, United States [doi>10.1145/253284.253292]
• NB93: S. K. Nayar and R. M. Bolle. Computing reflectance ratios from an image. Pattern recognition, 7, 1993.
• Pal99: S. Palmer. Vision Science : Photons to Phenomenology. MIT Press, 1999.
• Pho: Photomodeler. http://www.photomodeler.com.
• PM90: P. Perona , J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.7, p.629-639, July 1990 [doi>10.1109/34.56205]
• POF98: P. Poulin, M. Ouimet, and M.C. Frasson. Interactively modeling with photogrammetry. In Eurographics Workshop on Rendering, 1998.
• PRJ97: Pierre Poulin , Karim Ratib , Marco Jacques, Sketching Shadows and Highlights to Position Lights, Proceedings of the 1997 Conference on Computer Graphics International, p.56, June 23-27, 1997
• PSVF92: W. Press, S.Teukolsky, W. Vetterling, and B. Flannery. Numerical Recipes. Cambridge Univ. Pr., 2nd edition, 1992.
• Rea: Realviz. Image modeler. http://www.realviz.com.
• SD95: Fraņois Sillion , George Drettakis, Feature-based control of visibility error: a multi-resolution clustering algorithm for global illumination, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.145-152, September 1995 [doi>10.1145/218380.218434]
• SGHS98: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• SK98: Steven M. Seitz , Kiriakos N. Kutulakos, Plenoptic Image Editing, Proceedings of the Sixth International Conference on Computer Vision, p.17, January 04-07, 1998
• SKvW+92: Mark Segal , Carl Korobkin , Rolf van Widenfelt , Jim Foran , Paul Haeberli, Fast shadows and lighting effects using texture mapping, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.249-252, July 1992
• TM98: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• WA94: J. Y. A. Wang and E. H. Adelson. Representing moving images with layers. IEEE Trans. on Image Processing, 3(5):625-638, 1994.
• Wil98: L. Williams. Image jets, level sets and silhouettes. Workshop on Image-Based Modeling and Renderingt, http://wwwgraphics. stanford.edu/workshops/ibr98/, March 1998.
• YDMH99: Yizhou Yu , Paul Debevec , Jitendra Malik , Tim Hawkins, Inverse global illumination: recovering reflectance models of real scenes from photographs, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.215-224, July 1999 [doi>10.1145/311535.311559]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

General Terms:
Algorithms, Design, Theory