Parallax photography: creating 3D cinematic effects from stills

ABSTRACT

We present an approach to convert a small portion of a light field with extracted depth information into a cinematic effect with simulated, smooth camera motion that exhibits a sense of 3D parallax. We develop a taxonomy of the cinematic conventions of these effects, distilled from observations of documentary film footage and organized by the number of subjects of interest in the scene. We present an automatic, content-aware approach to apply these cinematic conventions to an input light field. A face detector identifies subjects of interest. We then optimize for a camera path that conforms to a cinematic convention, maximizes apparent parallax, and avoids missing information in the input. We describe a GPU-accelerated, temporally coherent rendering algorithm that allows users to create more complex camera moves interactively, while experimenting with effects such as focal length, depth of field, and selective, depth-based desaturation or brightening. We evaluate and demonstrate our approach on a wide variety of scenes and present a user study that compares our 3D cinematic effects to their 2D counterparts.

REFERENCES

• 1: Chris Buehler , Michael Bosse , Leonard McMillan , Steven Gortler , Michael Cohen, Unstructured lumigraph rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.425-432, August 2001 [doi>10.1145/383259.383309]
• 2: A. Criminisi, P. Perez, and K. Toyama. Object removal by exemplar-based inpainting. In Proc. IEEE Computer Vision and Pattern Recognition (CVPR), Jun. 2003.
• 3: J. Demers. Depth of field a survey of techniques. In R. Fernando, editor, GPU Gems 1, chapter 23, pages 375--390. Mar. 2004.
• 4: P. F. Felzenszwalb and D. P. Huttenlocher. Distance transforms of sampled functions. Technical Report TR2004--1963, Cornell University, 2004.
• 5: T. Georgiev, C. Zheng, S. Nayar, D. Salesin, B. Curless, and C. Intwala. Spatio-angular resolution trade-offs in integral photography. Proc. of Eurographics Symposium on Rendering, 2006.
• 6: S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. B. Cohen. The lumigraph. ACM Trans. Graph., pages 43--54, 1996.
• 7: E. Hammon. Practical post-process depth of field. In H. Nguyen, editor, GPU Gems 3, chapter 28. Addison Wesley, 2007.
• 8: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer visiond, Cambridge University Press, New York, NY, 2001
• 9: Li-wei He , Michael F. Cohen , David H. Salesin, The virtual cinematographer: a paradigm for automatic real-time camera control and directing, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.217-224, August 1996 [doi>10.1145/237170.237259]
• 10: Benno Heigl , Reinhard Koch , Marc Pollefeys , Joachim Denzler , Luc J. Van Gool, Plenoptic Modeling and Rendering from Image Sequences Taken by Hand-Held Camera, Mustererkennung 1999, 21. DAGM-Symposium, p.94-101, September 15-17, 1999
• 11: Derek Hoiem , Alexei A. Efros , Martial Hebert, Automatic photo pop-up, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 12: Youichi Horry , Ken-Ichi Anjyo , Kiyoshi Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.225-232, August 1997 [doi>10.1145/258734.258854]
• 13: A. Isaksen, L. McMillan, and S. J. Gortler. Dynamically reparameterized light fields. ACM Trans. Graph., pages 297--306, 2000.
• 14: S. B. Kang and H.-Y. Shum. A review of image-based rendering techniques. In IEEE/SPIE Visual Communications and Image Processing 2000, pages 2--13, 2002.
• 15: M. Kass, A. Lefohn, and J. D. Owens. Interactive depth of field using simulated diffusion. Technical Report 06-01, Pixar Animation Studios, Jan. 2006.
• 16: Andreas Klaus , Mario Sormann , Konrad Karner, Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure, Proceedings of the 18th International Conference on Pattern Recognition, p.15-18, August 20-24, 2006 [doi>10.1109/ICPR.2006.1033]
• 17: Anat Levin , Rob Fergus , Frédo Durand , William T. Freeman, Image and depth from a conventional camera with a coded aperture, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 18: Marc Levoy, Light Fields and Computational Imaging, Computer, v.39 n.8, p.46-55, August 2006 [doi>10.1109/MC.2006.270]
• 19: M. Levoy and P. Hanrahan. Light field rendering. ACM Trans. Graph., pages 31--42, 1996.
• 20: Chia-Kai Liang , Tai-Hsu Lin , Bing-Yi Wong , Chi Liu , Homer H. Chen, Programmable aperture photography: multiplexed light field acquisition, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 21: Francesc Moreno-Noguer , Peter N. Belhumeur , Shree K. Nayar, Active refocusing of images and videos, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 22: Ren Ng, Fourier slice photography, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 23: R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, and P. Hanrahan. Light field photography with a hand-held plenoptic camera. Technical Report CSTR 2005--02, Stanford University, 2005.
• 24: Byong Mok Oh , Max Chen , Julie Dorsey , Frédo Durand, Image-based modeling and photo editing, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.433-442, August 2001 [doi>10.1145/383259.383310]
• 25: Marc Pollefeys , Luc Van Gool , Maarten Vergauwen , Frank Verbiest , Kurt Cornelis , Jan Tops , Reinhard Koch, Visual Modeling with a Hand-Held Camera, International Journal of Computer Vision, v.59 n.3, p.207-232, September-October 2004 [doi>10.1023/B:VISI.0000025798.50602.3a]
• 26: Kari Pulli , Michael Cohen , Tom Duchamp , Hugues Hoppe , Linda G. Shapiro , Werner Stuetzle, View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data, Proceedings of the Eurographics Workshop on Rendering Techniques '97, p.23-34, June 16-18, 1997
• 27: Daniel Scharstein , Richard Szeliski, A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, International Journal of Computer Vision, v.47 n.1-3, p.7-42, April-June 2002 [doi>10.1023/A:1014573219977]
• 28: Steven M. Seitz , Brian Curless , James Diebel , Daniel Scharstein , Richard Szeliski, A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.519-528, June 17-22, 2006 [doi>10.1109/CVPR.2006.19]
• 29: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• 30: Jonathan Richard Shewchuk, Triangle: Engineering a 2D Quality Mesh Generator and Delaunay Triangulator, Selected papers from the Workshop on Applied Computational Geormetry, Towards Geometric Engineering, p.203-222, May 27-28, 1996
• 31: Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 32: D. Taylor. Virtual camera movement: The way of the future? American Cinematographer, 77(9):93--100, Sept. 1996.
• 33: Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
• 34: K. C. Zheng, A. Colburn, A. Agarwala, M. Agrawala, B. Curless, D. Salesin, and M. Cohen. A consistent segmentation approach to image-based rendering. Technical Report CSE-09-03-02, University of Washington, 2009.
• 35: C. Lawrence Zitnick , Sing Bing Kang , Matthew Uyttendaele , Simon Winder , Richard Szeliski, High-quality video view interpolation using a layered representation, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Graphics data structures and data types

Keywords:
image-based rendering, photo and image editing