Multiscale shape and detail enhancement from multi-light image collections

ABSTRACT

We present a new image-based technique for enhancing the shape and surface details of an object. The input to our system is a small set of photographs taken from a fixed viewpoint, but under varying lighting conditions. For each image we compute a multiscale decomposition based on the bilateral filter and then reconstruct an enhanced image that combines detail information at each scale across all the input images. Our approach does not require any information about light source positions, or camera calibration, and can produce good results with 3 to 5 input images. In addition our system provides a few high-level parameters for controlling the amount of enhancement and does not require pixel-level user input. We show that the bilateral filter is a good choice for our multiscale algorithm because it avoids the halo artifacts commonly associated with the traditional Laplacian image pyramid. We also develop a new scheme for computing our multiscale bilateral decomposition that is simple to implement, fast O(N² log N) and accurate.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: David Akers , Frank Losasso , Jeff Klingner , Maneesh Agrawala , John Rick , Pat Hanrahan, Conveying Shape and Features with Image-Based Relighting, Proceedings of the 14th IEEE Visualization 2003 (VIS'03), p.46, October 22-24, 2003 [doi>10.1109/VISUAL.2003.1250392]
• 3: Soonmin Bae , Sylvain Paris , Frédo Durand, Two-scale tone management for photographic look, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 4: Barash, D., and Comaniciu, D. 2004. A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift. Journal of Image and Vision Computing 22, 1, 73--81.
• 5: Burt, P. J., and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. on Comm. 31, 4, 532--540.
• 6: Burt, P., and Kolczynski, R. 1993. Enhanced image capture through fusion. Proc. ICCV, 173--182.
• 7: Burt, P. J. 1981. Fast filter transforms for image processing. Computer Graphics, Image Processing 6, 20--51.
• 8: Prasun Choudhury , Jack Tumblin, The trilateral filter for high contrast images and meshes, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 9: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 10: Frédo Durand , Julie Dorsey, Fast bilateral filtering for the display of high-dynamic-range images, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 11: Elmar Eisemann , Frédo Durand, Flash photography enhancement via intrinsic relighting, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 12: Raanan Fattal , Dani Lischinski , Michael Werman, Gradient domain high dynamic range compression, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 13: Graham D. Finlayson , Steven D. Hordley , Cheng Lu , Mark S. Drew, On the Removal of Shadows from Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.1, p.59-68, January 2006 [doi>10.1109/TPAMI.2006.18]
• 14: Holschneider, M., Kronland-Martinet, R., Morlet, J., and Tchamitchian, P. 1989. A real-time algorithms for signal analysis with the help of the wavelet transform. Wavelets, Time-Frequency Methods and Phase Space, 286--297.
• 15: Erum Arif Khan , Erik Reinhard , Roland W. Fleming , Heinrich H. Bülthoff, Image-based material editing, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 16: Li, H., Manjunath, B., and Mitra, S. 1994. Multi-sensor image fusion using the wavelet transform. Proc. ICIP 1, 51--55.
• 17: Yuanzhen Li , Lavanya Sharan , Edward H. Adelson, Compressing and companding high dynamic range images with subband architectures, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 18: Stphane Mallat, A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way, Academic Press, 2008
• 19: Tom Malzbender , Dan Gelb , Hans Wolters, Polynomial texture maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.519-528, August 2001 [doi>10.1145/383259.383320]
• 20: Vincent Masselus , Philip Dutré , Frederik Anrys, The free-form light stage, Proceedings of the 13th Eurographics workshop on Rendering, June 26-28, 2002, Pisa, Italy
• 21: Ankit Mohan , Jack Tumblin , Bobby Bodenheimer , Reynold Bailey , Cindy Grimm, Table-top computed lighting for practical digital photography, ACM SIGGRAPH 2005 Sketches, July 31-August 04, 2005, Los Angeles, California [doi>10.1145/1187112.1187203]
• 22: Mudge, M., Voutaz, J.-P., Schroer, C., and Lum, M. 2005. Reflection transformation imaging and virtual representations of coins from the Hospice of the Grand St. Bernard. In Proc. VAST.
• 23: Mudge, M., Malzbender, T., Schroer, C., and Lum, M. 2006. New reflection transformation imaging methods for rock art and multiple-viewpoint display. In Proc. VAST.
• 24: Ogden, J., Adelson, E., Bergen, J., and Burt, P. 1985. Pyramid-based computer graphics. RCA Engineer 30, 5, 4--15.
• 25: Byong Mok Oh , Max Chen , Julie Dorsey , Frédo Durand, Image-based modeling and photo editing, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.433-442, August 2001 [doi>10.1145/383259.383310]
• 26: Paris, S., and Durand, F. 2006. A fast approximation of the bilateral filter using a signal processing approach. Tech. Rep. MIT-CSAIL-TR-2006-073.
• 27: Sumanta N. Pattanaik , James A. Ferwerda , Mark D. Fairchild , Donald P. Greenberg, A multiscale model of adaptation and spatial vision for realistic image display, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.287-298, July 1998 [doi>10.1145/280814.280922]
• 28: Georg Petschnigg , Richard Szeliski , Maneesh Agrawala , Michael Cohen , Hugues Hoppe , Kentaro Toyama, Digital photography with flash and no-flash image pairs, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 29: Pham, T., and van Vliet, L. 2005. Separable Bilateral Filtering for Fast Video Preprocessing. Proc. ICME, 454--457.
• 30: Ramesh Raskar , Adrian Ilie , Jingyi Yu, Image fusion for context enhancement and video surrealism, Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering, June 07-09, 2004, Annecy, France [doi>10.1145/987657.987671]
• 31: Ramesh Raskar , Kar-Han Tan , Rogerio Feris , Jingyi Yu , Matthew Turk, Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 32: Erik Reinhard , Michael Stark , Peter Shirley , James Ferwerda, Photographic tone reproduction for digital images, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 33: Szymon Rusinkiewicz , Michael Burns , Doug DeCarlo, Exaggerated shading for depicting shape and detail, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 34: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 35: Jack Tumblin , Greg Turk, LCIS: a boundary hierarchy for detail-preserving contrast reduction, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.83-90, July 1999 [doi>10.1145/311535.311544]
• 36: Weiss, Y. 2001. Deriving intrinsic images from image sequences. In ICCV, 68--75.
• 37: Ben Weiss, Fast median and bilateral filtering, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 38: Winnemöller, H., Mohan, A., Tumblin, J., and Gooch, B. 2005. Light waving: Estimating light positions from photographs alone. Computer Graphics Forum 24, 3, 433--438.
• 39: Holger Winnemöller , Sven C. Olsen , Bruce Gooch, Real-time video abstraction, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 40: Woodham, R. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1, 139--144.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement
          Subjects: Geometric correction

Additional Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.2 Nonnumerical Algorithms and Problems
          Subjects: Geometrical problems and computations

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.5 Computational Geometry and Object Modeling
          Subjects: Curve, surface, solid, and object representations
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Size and shape

General Terms:
Algorithms, Design, Measurement

Keywords:
NPR, bilateral filter, image enhancement, multiscale image processing, relighting, shape depiction