3D-aware image editing for out of bounds photography

ABSTRACT

In this paper, we propose algorithms to manipulate 2D images in a way that is consistent with the 3D geometry of the scene that they capture. We present these algorithms in the context of creating "Out of Bounds" (OOB) images - compelling, depth-rich images generated from single, conventional 2D photographs (fig. 1). Starting from a single image our tool enables rapid OOB prototyping; i.e. the ability to quickly create and experiment with many different variants of the OOB effect before deciding which one best expresses the users' artistic intentions. We achieve this with a flexible work-flow driven by an intuitive user interface.

The rich 3D perception of the final composition is achieved by exploiting two strong cues -- occlusions and shadows. A realistic-looking 3D frame is interactively inserted in the scene between segmented foreground objects and the background to generate novel occlusions and enhance the scene's perception of depth. This perception is further enhanced by adding new, realistic cast shadows. The key contributions of this paper are: (i) new algorithms for inserting simple 3D objects like frames in 2D images requiring minimal camera calibration, and (ii) new techniques for the realistic synthesis of cast shadows, even for complex 3D objects. These algorithms, although presented for OOB photography, may be directly used in general image composition tasks.

With our tool, untrained users can turn ordinary photos into compelling OOB images in seconds. In contrast with existing workflows, at any time the artist can modify any aspect of the composition while avoiding time-consuming pixel painting operations. Such a tool has important commercial applications, and is much more suitable for OOB prototyping than existing image editors.

REFERENCES

• 1: J. Blinn, Me and My (Fake) Shadow, IEEE Computer Graphics and Applications, v.8 n.1, p.82-86, January 1988
• 2: X. Cao, Y. Shen, M. Shah, and H. Foroosh. Single view compositing with shadows. The Visual Computer, 21(8--10):639--648, 2005.
• 3: P. Cavanaugh. The artist as neuroscientist. Nature, 434:301--307, 2005.
• 4: P. Cavanaugh and Y. Leclerc. Shape from shadows. J. Experimental Psychology, 15:3--27, 1989.
• 5: Yung-Yu Chuang , Dan B Goldman , Ke Colin Zheng , Brian Curless , David H. Salesin , Richard Szeliski, Animating pictures with stochastic motion textures, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 6: Accurate visual metrology from single and multiple uncalibrated images, Springer-Verlag New York, Inc., New York, NY, 2001
• 7: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 8: Christopher DeCoro , Forrester Cole , Adam Finkelstein , Szymon Rusinkiewicz, Stylized shadows, Proceedings of the 5th international symposium on Non-photorealistic animation and rendering, August 04-05, 2007, San Diego, California [doi>10.1145/1274871.1274884]
• 9: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer visiond, Cambridge University Press, New York, NY, 2001
• 10: Derek Hoiem , Alexei A. Efros , Martial Hebert, Automatic photo pop-up, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 11: Youichi Horry , Ken-Ichi Anjyo , Kiyoshi Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.225-232, August 1997 [doi>10.1145/258734.258854]
• 12: Takeo Igarashi , Satoshi Matsuoka , Hidehiko Tanaka, Teddy: a sketching interface for 3D freeform design, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.409-416, July 1999 [doi>10.1145/311535.311602]
• 13: D. Kersten, D. Knill, P. Mamassian, and Bulthoff. Illusory motion from shadows. Nature, 379(6560):31, 1996.
• 14: Jean-François Lalonde , Derek Hoiem , Alexei A. Efros , Carsten Rother , John Winn , Antonio Criminisi, Photo clip art, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 15: D. Liebowitz, A. Criminisi, and A. Zisserman. Creating architectural models from images. In Proc. Eurographics, pages 39--50, 1999.
• 16: Herve Lombaert , Yiyong Sun , Leo Grady , Chenyang Xu, A Multilevel Banded Graph Cuts Method for Fast Image Segmentation, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.259-265, October 17-20, 2005 [doi>10.1109/ICCV.2005.13]
• 17: David Marr, Vision: A Computational Investigation into the Human Representation and Processing of Visual Information, Henry Holt and Co., Inc., New York, NY, 1982
• 18: Ankit Mohan , Jack Tumblin , Prasun Choudhury, Editing Soft Shadows in a Digital Photograph, IEEE Computer Graphics and Applications, v.27 n.2, p.23-31, March 2007 [doi>10.1109/MCG.2007.30]
• 19: Byong Mok Oh , Max Chen , Julie Dorsey , Frédo Durand, Image-based modeling and photo editing, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.433-442, August 2001 [doi>10.1145/383259.383310]
• 20: Mukta Prasad , Andrew Fitzgibbon, Single View Reconstruction of Curved Surfaces, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1345-1354, June 17-22, 2006 [doi>10.1109/CVPR.2006.281]
• 21: Carsten Rother , Vladimir Kolmogorov , Andrew Blake, "GrabCut": interactive foreground extraction using iterated graph cuts, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 22: Tai-Pang Wu , Chi-Keung Tang , Michael S. Brown , Heung-Yeung Shum, Natural shadow matting, ACM Transactions on Graphics (TOG), v.26 n.2, p.8-es, June 07 [doi>10.1145/1243980.1243982]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications