High performance imaging using large camera arrays

ABSTRACT

The advent of inexpensive digital image sensors and the ability to create photographs that combine information from a number of sensed images are changing the way we think about photography. In this paper, we describe a unique array of 100 custom video cameras that we have built, and we summarize our experiences using this array in a range of imaging applications. Our goal was to explore the capabilities of a system that would be inexpensive to produce in the future. With this in mind, we used simple cameras, lenses, and mountings, and we assumed that processing large numbers of images would eventually be easy and cheap. The applications we have explored include approximating a conventional single center of projection video camera with high performance along one or more axes, such as resolution, dynamic range, frame rate, and/or large aperture, and using multiple cameras to approximate a video camera with a large synthetic aperture. This permits us to capture a video light field, to which we can apply spatiotemporal view interpolation algorithms in order to digitally simulate time dilation and camera motion. It also permits us to create video sequences using custom non-uniform synthetic apertures.

REFERENCES

• 1: Don Anderson, FireWire system architecture (2nd ed.): IEEE 1394a, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 2: Bayer, B., 1976. Color imaging array. U. S. Patent 3,971,065.
• 3: Black, M., and Anandan, P. 1993. A framework for the robust estimation of optical flow. In Proc. ICCV 1993, 231--236.
• 4: M. Brown , D. G. Lowe, Recognising Panoramas, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1218, October 13-16, 2003
• 5: Jin-Xiang Chai , Shing-Chow Chan , Heung-Yeung Shum , Xin Tong, Plenoptic sampling, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.307-318, July 2000 [doi>10.1145/344779.344932]
• 6: Paul E. Debevec , Jitendra Malik, Recovering high dynamic range radiance maps from photographs, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.369-378, August 1997 [doi>10.1145/258734.258884]
• 7: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 8: Aaron Isaksen , Leonard McMillan , Steven J. Gortler, Dynamically reparameterized light fields, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.297-306, July 2000 [doi>10.1145/344779.344929]
• 9: Kang, S., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video. In Proc. SIGGRAPH 2003, 319--325.
• 10: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 11: Wojciech Matusik , Chris Buehler , Ramesh Raskar , Steven J. Gortler , Leonard McMillan, Image-based visual hulls, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.369-374, July 2000 [doi>10.1145/344779.344951]
• 12: Peter Rander , P. J. Narayanan , Takeo Kanade, Virtualized reality: constructing time-varying virtual worlds from real world events, Proceedings of the 8th conference on Visualization '97, p.277-ff., October 18-24, 1997, Phoenix, Arizona, United States
• 13: Schechner, Y., and Nayar, S. 2001. Generalized mosaicing. In Proc. ICCV 2001, 17--24.
• 14: E. Shechtman , Yaron Caspi , Michal Irani, Increasing Space-Time Resolution in Video, Proceedings of the 7th European Conference on Computer Vision-Part I, p.753-768, May 28-31, 2002
• 15: S. Mann, and R. W. Picard. 1994. Being 'undigital' with digital cameras: Extending dynamic range by combining differently exposed pictures. Tech. Rep. 323, M.I.T. Media Lab Perceptual Computing Section, Boston, Massachusetts. Also appears, IS&T's 48th annual conference, Cambridge, Massachusetts, May 1995.
• 16: J. Stewart , J. Yu , S. J. Gortler , L. McMillan, A new reconstruction filter for undersampled light fields, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 17: Szeliski, R. 1994. Image mosaicing for tele-reality applications. In WACV 1994, 44--53.
• 18: Tao, H., Sawhney, H., and Kumar, R. 2001. A global matching framework for stereo computation. In Proc. ICCV 2001, 532--539.
• 19: Taylor, D. 1996. Virtual camera movement: The way of the future? American Cinematographer 77, 9 (September), 93--100.
• 20: Vaish, V., Wilburn, B., Joshi, N., and Levoy, M. 2004. Using plane + parallax for calibrating dense camera arrays. In Proc. CVPR 2004, 2--9.
• 21: Vaish, V., Garg, G., Talvala, E., Antunez, E., Wilburn, B., Horowitz, M., and Levoy, M. 2005. Synthetic aperture focusing using a shear-warp factorization of the viewing transform. In Proc. A3DISS 2005.
• 22: Wilburn, B., Smulski, M., Lee, H., and Horowitz, M. 2002. The light field video camera. In Media Processors 2002, vol. 4674 of Proc. SPIE, 29--36.
• 23: Wilburn, B., Joshi, N., Vaish, V., Levoy, M., and Horowitz, M. 2004. High speed video using a dense array of cameras. In Proc. CVPR 2004, 294--301.
• 24: Jason C. Yang , Matthew Everett , Chris Buehler , Leonard McMillan, A real-time distributed light field camera, Proceedings of the 13th Eurographics workshop on Rendering, June 26-28, 2002, Pisa, Italy
• 25: Zhang, C., and Chen, T. 2004. A self-reconfigurable camera array. In Eurographics Symposium on Rendering, 243--254.
• 26: Zhang, C., and Chen, T. 2004. View-dependent non-uniform sampling for image-based rendering. In Proc. ICIP 2004, 2471--2474.
• 27: Zhang, Y., and Kambhamettu, C. 2001. On 3d scene flow and structure estimation. In Proc. CVPR 2001, 778--785.
• 28: Zitnick, C., Kang, S., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. In Proc. SIGGRAPH 2004, 600--608.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture

Keywords:
camera arrays, spatiotemporal sampling, synthetic aperture