Plenoptic sampling

ABSTRACT

This paper studies the problem of plenoptic sampling in image-based rendering (IBR). From a spectral analysis of light field signals and using the sampling theorem, we mathematically derive the analytical functions to determine the minimum sampling rate for light field rendering. The spectral support of a light field signal is bounded by the minimum and maximum depths only, no matter how complicated the spectral support might be because of depth variations in the scene. The minimum sampling rate for light field rendering is obtained by compacting the replicas of the spectral support of the sampled light field within the smallest interval. Given the minimum and maximum depths, a reconstruction filter with an optimal and constant depth can be designed to achieve anti-aliased light field rendering. Plenoptic sampling goes beyond the minimum number of images needed for anti-aliased light field rendering. More significantly, it utilizes the scene depth information to determine the minimum sampling curve in the joint image and geometry space. The minimum sampling curve quantitatively describes the relationship among three key elements in IBR systems: scene complexity (geometrical and textural information), the number of image samples, and the output resolution. Therefore, plenoptic sampling bridges the gap between image-based rendering and traditional geometry-based rendering. Experimental results demonstrate the effectiveness of our approach.

REFERENCES

• 1: E.H. Adelson and J. Bergen. The plenoptic function and the elements of early vision. In Computational Models of Visual Processing, pages 3-20. MIT Press, Cambridge, MA, 1991.
• 2: M. Bass, editor. Handbook of Optics. McGraw-Hill, New York, 1995.
• 3: R.C. Bolles, H. H. Baker, and D. H. Marimont. Epipolar-plane image analysis: An approach to determining structure from motion. International Journal of Computer Vision, 1:7-55, 1987.
• 4: E. Camahort, A. Lerios, and D. Fussell. Uniformly sampled light fields. In Proc. 9th Eurographics Workshop on Rendering, pages 117-130, 1998.
• 5: J.-X. Chai and H.-Y. Shum. Parallel projections for stereo reconstruction. In Proc. CVPR 2000, 2000.
• 6: Chun-Fa Chang , Gary Bishop , Anselmo Lastra, LDI tree: a hierarchical representation for image-based rendering, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.291-298, July 1999 [doi>10.1145/311535.311571]
• 7: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• 8: T. Feng and H.-Y. Shum. An optical analysis of light field rendering. Technical report, Microsoft Research, MSR-TR-2000-38, May 2000.
• 9: B. Girod. Motion compensation: visual aspects, accuracy, and fundamental limits. In Motion Analysis and Image Sequence Processing. Kluwer, 1995.
• 10: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 11: M. Halle. Holographic stereograms as discrete imaging systems. In Proc. SPIE Vol.2176, Practical Holography VIII, pages 73-84, May 1994.
• 12: Václav Hlavác , Ales Leonardis , Thomás Werner, Automatic Selection of Reference Views for Image-Based Scene Representations, Proceedings of the 4th European Conference on Computer Vision-Volume I, p.526-535, April 15-18, 1996
• 13: A. Isaksen, L. McMillan, and S. Gortler. Dynamically reparameterized light fields. Technical report, Technical Report MIT-LCS-TR-778, May 1999.
• 14: S. Kang. A survey of image-based rendering techniques. In VideoMetrics, SPIE Vol. 3641, pages 2-16, 1999.
• 15: Jed Lengyel, The Convergence of Graphics and Vision, Computer, v.31 n.7, p.46-53, July 1998 [doi>10.1109/2.689676]
• 16: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 17: Z.-C. Lin and H.-Y. Shum. On the numbers of samples needed in light field rendering with constant-depth assumption. In Proc. CVPR 2000, 2000.
• 18: William R. Mark , Leonard McMillan , Gary Bishop, Post-rendering 3D warping, Proceedings of the 1997 symposium on Interactive 3D graphics, p.7-ff., April 27-30, 1997, Providence, Rhode Island, United States [doi>10.1145/253284.253292]
• 19: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• 20: H. Schirmacher, W. Heidrich, and H. Seidel. Adaptive acquisition of lumigraphs from synthetic scenes. In Eurographics '99, pages 151-159, Sept 1999.
• 21: Steven M. Seitz , Charles R. Dyer, View morphing, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.21-30, August 1996 [doi>10.1145/237170.237196]
• 22: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• 23: Heung-Yeung Shum , Li-Wei He, Rendering with concentric mosaics, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.299-306, July 1999 [doi>10.1145/311535.311573]
• 24: Peter-Pike Sloan , Michael F. Cohen , Steven J. Gortler, Time critical lumigraph rendering, Proceedings of the 1997 symposium on Interactive 3D graphics, p.17-ff., April 27-30, 1997, Providence, Rhode Island, United States [doi>10.1145/253284.253296]
• 25: A. Murat Tekalp, Digital video processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION

General Terms:
Design, Experimentation, Measurement, Performance, Theory

Keywords:
image-based rendering, plenoptic functions, plenoptic sampling, sampling, spectral analysis