Plenoptic stitching: a scalable method for reconstructing 3D interactive walk throughs

ABSTRACT

Interactive walkthrough applications require detailed 3D models to give users a sense of immersion in an environment. Traditionally these models are built using computer-aided design tools to define geometry and material properties. But creating detailed models is time-consuming and it is also difficult to reproduce all geometric and photometric subtleties of real-world scenes. Computer vision attempts to alleviate this problem by extracting geometry and photogrammetry from images of the real-world scenes. However, these models are still limited in the amount of detail they recover.

Image-based rendering generates novel views by resampling a set of images of the environment without relying upon an explicit geometric model. Current such techniques limit the size and shape of the environment, and they do not lend themselves to walkthrough applications. In this paper, we define a parameterization of the 4D plenoptic function that is particularly suitable for interactive walkthroughs and define a method for its sampling and reconstructing. Our main contributions are: 1) a parameterization of the 4D plenoptic function that supports walkthrough applications in large, arbitrarily shaped environments; 2) a simple and fast capture process for complex environments; and 3) an automatic algorithm for reconstruction of the plenoptic function.

REFERENCES

• Adelson91: Adelson E.H. and Bergen J., In "The Plenoptic Function and the Elements of Early Vision", In Computational Models of Visual Processing, MIT Press, Cambridge, MA, 3-20, 1991.
• Aliaga01: Aliaga D., "Accurate Catadioptric Calibration for Real-time Pose Estimation in Room-size Environments", IEEE International Conference on Computer Vision (ICCV 01), July, 2001.
• Chen93: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• Chen95: Shenchang Eric Chen, QuickTime VR: an image-based approach to virtual environment navigation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.29-38, September 1995 [doi>10.1145/218380.218395]
• Debevec96: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• Faugeras93: Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, Cambridge, MA, 1993
• Faugeras98: Olivier Faugeras , Luc Robert , Stéphane Laveau , Gabriella Csurka , Cyril Zeller , Cyrille Gauclin , Imad Zoghlami, 3-D reconstruction of urban scenes from image sequences, Computer Vision and Image Understanding, v.69 n.3, p.292-309, March 1998 [doi>10.1006/cviu.1998.0665]
• Geyer98: C. Geyer and K. Daniliidis, "Catadioptric Camera Calibration", IEEE International Conference on Computer Vision (ICCV 98), pp. 398-404, 1998.
• Gortler96: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• Gottschalk96: S. Gottschalk , M. C. Lin , D. Manocha, OBBTree: a hierarchical structure for rapid interference detection, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.171-180, August 1996 [doi>10.1145/237170.237244]
• Kang96: Sing Bing Kang , Richard Szeliski, 3-D Scene Data Recovery using Omnidirectional Multibaseline Stereo, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.364, June 18-20, 1996
• Levoy96: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• LeGall91: Didier Le Gall, MPEG: a video compression standard for multimedia applications, Communications of the ACM, v.34 n.4, p.46-58, April 1991 [doi>10.1145/103085.103090]
• Max95: Max N. and Ohsaki K., "Rendering Trees from Precomputed Z- Buffer Views", Rendering Techniques '95: Proceedings of the 6th Eurographics Workshop on Rendering, 45-54, 1995.
• McMillan95: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• Nalwa96: Nalwa V.S., A True Omnidirectional Viewer, Technical Report, Bell Laboratories, Holmdel, NJ, 1996.
• Nayar97: Shree K. Nayar, Catadioptric Omnidirectional Camera, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.482, June 17-19, 1997
• Rademacher98: Paul Rademacher , Gary Bishop, Multiple-center-of-projection images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.199-206, July 1998 [doi>10.1145/280814.280871]
• Roberts97: Roberts D.R. and Marshall A.D., "A Review of Viewpoint Planning", Technical Report 97008, University of Wales, College of Cardiff, Department of Computer Science, 1997.
• Shi94: Shi J. and Tomasi C., "Good Features to Track", IEEE Computer Vision and Pattern Recognition (CVPR 94), 593-600, 1994.
• Shum99: Heung-Yeung Shum , Li-Wei He, Rendering with concentric mosaics, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.299-306, July 1999 [doi>10.1145/311535.311573]
• Szeliski96: Szeliski R., "Video mosaics for virtual environments", IEEE Computer Graphics and Applications, 22-30, 1996.
• Szeliski97: Richard Szeliski , Heung-Yeung Shum, Creating full view panoramic image mosaics and environment maps, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.251-258, August 1997 [doi>10.1145/258734.258861]
• Takahasi00: Takahashi T., Kawasaki H., Ikeuchi K., and Sakauchi M., "Arbitrary View Position and Direction Rendering for Large-Scale Scenes", IEEE Computer Vision and Pattern Recognition (CVPR 00), 296-303, 2000.
• Taylor00: Camillo J. Taylor, Video Plus, Proceedings of the IEEE Workshop on Omnidirectional Vision, p.3, June 16-16, 2000
• Tomasi91: Tomasi C. and Kanade T., "Detection and Tracking of Point Features", Carnegie Mellon University Technical Report CMU-CS-91-132, 1991.
• Wallace91: Gregory K. Wallace, The JPEG still picture compression standard, Communications of the ACM, v.34 n.4, p.30-44, April 1991 [doi>10.1145/103085.103089]
• Ward92: Mark Ward , Ronald Azuma , Robert Bennett , Stefan Gottschalk , Henry Fuchs, A demonstrated optical tracker with scalable work area for head-mounted display systems, Proceedings of the 1992 symposium on Interactive 3D graphics, p.43-52, June 1992, Cambridge, Massachusetts, United States [doi>10.1145/147156.147162]
• Yu98: Yu Y. and Malik J., "Recovering photometric properties of architectural scenes from photographs", Computer Graphics (SIGGRAPH 96), 207-218, 1996.
• Ziv77: Ziv J. and Lempel A., "A universal algorithm for sequential data compression", IEEE Transactions on Information Theory, IT-23, 337-343, 1977.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

General Terms:
Algorithms, Design, Theory

Keywords:
image-based rendering, interactive walkthroughs, omnidirectional, plenoptic function, virtual environments