Image fusion for context enhancement and video surrealism

ABSTRACT

We present a class of image fusion techniques to automatically combine images of a scene captured under different illumination. Beyond providing digital tools for artists for creating surrealist images and videos, the methods can also be used for practical applications. For example, the non-realistic appearance can be used to enhance the context of nighttime traffic videos so that they are easier to understand. The context is automatically captured from a fixed camera and inserted from a day-time image (of the same scene). Our approach is based on a gradient domain technique that preserves important local perceptual cues while avoiding traditional problems such as aliasing, ghosting and haloing. We presents several results in generating surrealistic videos and in increasing the information density of low quality nighttime videos.

REFERENCES

• 1: BOSTONWEBCAMS, 2003. (Observe the cameras at night). http://www.boston.com/traffic/cameras/artery.htm.
• 2: BRAUN, M. 1992. Picturing Time. University of Chicago.
• 3: CHUANG, Y., CURLESS, B., SALESIN, D., AND R., S. 2001. A Bayesian Approach to Digital Matting. In Proceedings of CVPR, vol. 2, 264--271.
• 4: COHEN, M., 2003. Image Stacks. Presentation at MIT, Cambridge, USA, October 2003.
• 5: Doug DeCarlo , Anthony Santella, Stylization and abstraction of photographs, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 6: DICARLO, J., AND WANDELL, B. 2000. Rendering High Dynamic Range Images. In Proceedings of SPIE: Image Sensors, vol. 3965, 392--401.
• 7: Frédo Durand , Julie Dorsey, Fast bilateral filtering for the display of high-dynamic-range images, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 8: ESSA, I., 2002. Graphical Display of Motion-Capture System using a Shape-time Style Rendering.
• 9: Raanan Fattal , Dani Lischinski , Michael Werman, Gradient domain high dynamic range compression, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 10: Sidney Fels , Kenji Mase, Interactive video cubism, Proceedings of the 1999 workshop on new paradigms in information visualization and manipulation in conjunction with the eighth ACM internation conference on Information and knowledge management, p.78-82, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/331770.331789]
• 11: Graham D. Finlayson , Steven D. Hordley , Mark S. Drew, Removing Shadows from Images, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.823-836, May 28-31, 2002
• 12: FREEMAN, B., AND ZHANG, H. 2003. Shapetime photography. In Proceedings of CVPR, vol. 2, 264--271.
• 13: Michael L. Gleicher , Rachel M. Heck , Michael N. Wallick, A framework for virtual videography, Proceedings of the 2nd international symposium on Smart graphics, p.9-16, June 11-13, 2002, Hawthorne, New York [doi>10.1145/569005.569007]
• 14: HAEBERLI, P., 1994. A Multifocus Method for Controlling Depth of Field. Availavle at: http://www.sgi.com/grafica/depth/index.html.
• 15: Aaron Hertzmann , Ken Perlin, Painterly rendering for video and interaction, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.7-12, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340917]
• 16: Allison W. Klein , Tyler Grant , Adam Finkelstein , Michael F. Cohen, Video mosaics, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508534]
• 17: Allison W. Klein , Peter-Pike J. Sloan , Adam Finkelstein , Michael F. Cohen, Stylized video cubes, Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 21-22, 2002, San Antonio, Texas [doi>10.1145/545261.545264]
• 18: Peter Litwinowicz, Processing images and video for an impressionist effect, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.407-414, August 1997 [doi>10.1145/258734.258893]
• 19: Barbara J. Meier, Painterly rendering for animation, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.477-484, August 1996 [doi>10.1145/237170.237288]
• 20: MERRIAM-WEBSTER. 2001. Collegiate Dictionary.
• 21: MUYBRIDGE, E. 1985. Horses and other animals in motion. Dover.
• 22: PÉREZ, P., GANGNET, M., AND BLAKE, A. 2003. Poisson image editing. In Proceedings of SIGGRAPH 2003, 313--318.
• 23: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical recipes in C (2nd ed.): the art of scientific computing, Cambridge University Press, New York, NY, 1992
• 24: Paul Rademacher , Gary Bishop, Multiple-center-of-projection images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.199-206, July 1998 [doi>10.1145/280814.280871]
• 25: RASKAR, R., YU, J., AND ILIE, A., 2003. Stylized Images using Variable Illumination. Unpublished, January 2003.
• 26: RASKAR, R., YU, J., AND ILIE, A., 2003. Video Surveillance with NPR Image Fusion. http://www.merl.com/projects/NPRfusion/, March 2003.
• 27: Erik Reinhard , Michael Stark , Peter Shirley , James Ferwerda, Photographic tone reproduction for digital images, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 28: Stephen M. Smith , J. Michael Brady, SUSAN—A New Approach to Low Level Image Processing, International Journal of Computer Vision, v.23 n.1, p.45-78, May 1997 [doi>10.1023/A:1007963824710]
• 29: SOCOLINSKY, D., AND WOLFF, L. 1999. A New Visualization Paradigm for Multispectral Imagery and Data Fusion. In Proceedings of IEEE CVPR, 319--324.
• 30: TOYAMA, K., KRUMM, J., BRUMITT, B., AND MEYERS, B. 1999. Wallflower: Principles and Practice of Background Maintenance. In ICCV, 255--261.
• 31: WEISS, Y. 2001. Deriving Intrinsic Images From Image Sequences. In Proceedings of ICCV, vol. 2, 68--75.

INDEX TERMS

Keywords:
gradient domain approach, image fusion, surrealism