Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging

ABSTRACT

We present a non-photorealistic rendering approach to capture and convey shape features of real-world scenes. We use a camera with multiple flashes that are strategically positioned to cast shadows along depth discontinuities in the scene. The projective-geometric relationship of the camera-flash setup is then exploited to detect depth discontinuities and distinguish them from intensity edges due to material discontinuities.We introduce depiction methods that utilize the detected edge features to generate stylized static and animated images. We can highlight the detected features, suppress unnecessary details or combine features from multiple images. The resulting images more clearly convey the 3D structure of the imaged scenes.We take a very different approach to capturing geometric features of a scene than traditional approaches that require reconstructing a 3D model. This results in a method that is both surprisingly simple and computationally efficient. The entire hardware/software setup can conceivably be packaged into a self-contained device no larger than existing digital cameras.

REFERENCES

• 1: David Akers , Frank Losasso , Jeff Klingner , Maneesh Agrawala , John Rick , Pat Hanrahan, Conveying Shape and Features with Image-Based Relighting, Proceedings of the 14th IEEE Visualization 2003 (VIS'03), p.46, October 22-24, 2003 [doi>10.1109/VISUAL.2003.1250392]
• 2: AVENUE AMY, 2002. Curious Pictures.
• 3: Stanley Thomas Birchfield , Carlo Tomasi, Depth and motion discontinuities, 1999
• 4: Yung-Yu Chuang , Dan B Goldman , Brian Curless , David H. Salesin , Richard Szeliski, Shadow matting and compositing, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 5: COHEN, M. F., COLBURN, A., AND DRUCKER, S. 2003. Image stacks. Tech. Rep. MSR-TR-2003-40, Microsoft Research.
• 6: M. Daum , G. Dudek, On 3-D Surface Reconstruction Using Shape from Shadows, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.461, June 23-25, 1998
• 7: Doug DeCarlo , Anthony Santella, Stylization and abstraction of photographs, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 8: Doug DeCarlo , Adam Finkelstein , Szymon Rusinkiewicz , Anthony Santella, Suggestive contours for conveying shape, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 9: Frédo Durand, An invitation to discuss computer depiction, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508550]
• 10: James H. Elder, Are Edges Incomplete?, International Journal of Computer Vision, v.34 n.2-3, p.97-122, Nov. 1999 [doi>10.1023/A:1008183703117]
• 11: Raanan Fattal , Dani Lischinski , Michael Werman, Gradient domain high dynamic range compression, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 12: Rogerio Feris , Matthew Turk , Ramesh Raskar , Karhan Tan , Gosuke Ohashi, Exploiting Depth Discontinuities for Vision-Based Fingerspelling Recognition, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 10, p.155, June 27-July 02, 2004
• 13: David A. Forsyth , Jean Ponce, Computer Vision: A Modern Approach, Prentice Hall Professional Technical Reference, 2002
• 14: Davi Geiger , Bruce Ladendorf , Alan L. Yuille, Occlusions and Binocular Stereo, Proceedings of the Second European Conference on Computer Vision, p.425-433, May 19-22, 1992
• 15: Bruce Gooch , Amy Gooch, Non-Photorealistic Rendering, A. K. Peters, Ltd., Natick, MA, 2001
• 16: Aaron Hertzmann, Painterly rendering with curved brush strokes of multiple sizes, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.453-460, July 1998 [doi>10.1145/280814.280951]
• 17: A. Huertas , R. Nevatia, Detecting buildings in aerial images, Computer Vision, Graphics, and Image Processing, v.41 n.2, p.131-152, Feb, 1988 [doi>10.1016/0734-189X(88)90016-3]
• 18: HUGGINGS, P., CHEN, H., BELHUMEUR, P., AND ZUCKER, S. 2001. Finding Folds: On the Appearance and Identification of Occlusion. In IEEE CVPR, vol. 2, 718--725.
• 19: IRVIN, R., AND MCKEOWN, D. 1989. Methods for exploiting the relationship between buildings and their shadows in aerial imagery. IEEE Transactions on Systems, Man and Cybernetics 19, 6, 1564--1575.
• 20: Scott F. Johnston, Lumo: illumination for cel animation, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508538]
• 21: KANG, S. B., SZELISKI, R., AND CHAI, J. 2001. Handling occlusions in dense multi-view stereo. In IEEE CVPR, vol. 1, 102--110.
• 22: KRIEGMAN, D., AND BELHUMEUR, P. 2001. What Shadows Reveal About Object Structure. Journal of the Optical Society of America, 1804--1813.
• 23: LANGER, M., DUDEK, G., AND ZUCKER, S. 1995. Space Occupancy using Multiple Shadow Images. International Conference on Intelligent Robots and Systems, 390--396.
• 24: Chungan Lin , Ramakant Nevatia, Building detection and description from a single intensity image, Computer Vision and Image Understanding, v.72 n.2, p.101-121, Nov. 1998 [doi>10.1006/cviu.1998.0724]
• 25: PAPADEMETRIS, X., AND BELHUMEUR, P. N. 1996. Estimation of motion boundary location and optical flow using dynamic programming. In Proc. Int. Conf. on Image Processing.
• 26: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical recipes in C (2nd ed.): the art of scientific computing, Cambridge University Press, New York, NY, 1992
• 27: Ramesh Raskar , Adrian Ilie , Jingyi Yu, Image fusion for context enhancement and video surrealism, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987671]
• 28: RAVIV, D., PAO, Y., AND LOPARO, K. A. 1989. Reconstruction of Three-dimensional Surfaces from Two-dimensional Binary Images. In IEEE Transactions on Robotics and Automation, vol. 5(5), 701--710.
• 29: Takafumi Saito , Tokiichiro Takahashi, Comprehensible rendering of 3-D shapes, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.197-206, September 1990, Dallas, TX, USA
• 30: SATO, I., SATO, Y., AND IKEUCHI, K. 2001. Stability issues in recovering illumination distribution from brightness in shadows. IEEE Conf. on CVPR, 400--407.
• 31: SAVARESE, S., RUSHMEIER, H., BERNARDINI, F., AND PERONA, P. 2001. Shadow Carving. In ICCV.
• 32: Daniel Scharstein , Richard Szeliski, A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, International Journal of Computer Vision, v.47 n.1-3, p.7-42, April-June 2002 [doi>10.1023/A:1014573219977]
• 33: SHIRAI, Y., AND TSUJI, S. 1972. Extraction of the Line Drawing of 3-Dimensional Objects by Sequential Illumination from Several Directions. Pattern Recognition 4, 4, 345--351.
• 34: Thomas Strothotte , Stefan Schlechtweg, Non-photorealistic computer graphics: modeling, rendering, and animation, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2002
• 35: Ping Tan , Stephen Lin , Long Quan , Heung-Yeung Shum, Highlight Removal by Illumination-Constrained Inpainting, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.164, October 13-16, 2003
• 36: TAN, K., KOBLER, J., DIETZ, P., FERIS, R., AND RASKAR, R. 2004. Shape-Enhanced Surgical Visualizations and Medical Illustrations with Multi-Flash Imaging. In MERL TR/38.
• 37: TOYAMA, K., KRUMM, J., BRUMITT, B., AND MEYERS, B. 1999. Wallflower: Principles and Practice of Background Maintenance. In ICCV, 255--261.
• 38: WAKING LIFE, 2001. Waking Life, the movie.
• 39: WEISS, Y. 2001. Deriving intrinsic images from image sequences. In Proceedings of ICCV, vol. 2, 68--75.
• 40: David Kwun-Moo Yang , John Ronald Kender, Shape from darkness under error, 1996

INDEX TERMS

Keywords:
depth edges, image enhancement, non-photorealistic rendering