Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

This paper presents a complete system for constructing panoramic image mosaics from sequences of images. Our mosaic representation associates a transformation matrix with each input image, rather than explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to construct a full view panorama, we introduce a rotational mosaic representation that associates a rotation matrix (and optionally a focal length) with each input image. A patch-based alignment algorithm is developed to quickly align two images given motion models. Techniques for estimating and refining camera focal lengths are also presented.

In order to reduce accumulated registration errors, we apply global alignment (block adjustment) to the whole sequence of images, which results in an optimally registered image mosaic. To compensate for small amounts of motion parallax introduced by translations of the camera and other unmodeled distortions, we use a local alignment (deghosting) technique which warps each image based on the results of pairwise local image registrations. By combining both global and local alignment, we significantly improve the quality of our image mosaics, thereby enabling the creation of full view panoramic mosaics with hand-held cameras.

We also present an inverse texture mapping algorithm for efficiently extracting environment maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture-mapped polyhedron surrounding the origin, we can explore the virtual environment using standard 3D graphics viewers and hardware without requiring special-purpose players.

REFERENCES

• 1: Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. <i>International Journal of Computer Vision</i>, 2(3):283-310.
• 2: Anandan, P., Sawhrey, H., Shashua, A., and Simoncelli, E. (Eds.) 1995. In <i>IEEE Workshop on Representations of Visual Scenes</i>, Cambridge, Massachusetts, IEEE Computer Society Press.
• 3: Ayache, N. 1989. <i>Vision Stéréoscopique et Perception Multisensorielle </i>. InterEditions: Paris.
• 4: James R. Bergen , P. Anandan , Keith J. Hanna , Rajesh Hingorani, Hierarchical Model-Based Motion Estimation, Proceedings of the Second European Conference on Computer Vision, p.237-252, May 19-22, 1992
• 5: Michael J. Black , Anand Rangarajan, On the unification of line processes, outlier rejection, and robust statistics with applications in early vision, International Journal of Computer Vision, v.19 n.1, p.57-91, July 1996 [doi>10.1007/BF00131148]
• 6: Peter J. Burt , Edward H. Adelson, A multiresolution spline with application to image mosaics, ACM Transactions on Graphics (TOG), v.2 n.4, p.217-236, October 1983 [doi>10.1145/245.247]
• 7: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• 8: Shenchang Eric Chen, QuickTime VR: an image-based approach to virtual environment navigation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.29-38, September 1995 [doi>10.1145/218380.218395]
• 9: Ming-Chao Chiang , Terrance E. Boult, Efficient Image Warping and Super-Resolution, Proceedings of the 3rd IEEE Workshop on Applications of Computer Vision (WACV '96), p.56, December 02-04, 1996
• 10: Danielsson, P.E. 1980. Euclidean distance mapping. <i>Computer Graphics and Image Processing</i>, 14:227-248.
• 11: J. Davis, Mosaics of Scenes with Moving Objects, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.354, June 23-25, 1998
• 12: Pascal Fua , Yvan G. Leclerc, Using 3-Dimensional Meshes To Combine Image-Based and Geometry-Based Constraints, Proceedings of the Third European Conference-Volume II on Computer Vision, p.281-291, May 02-06, 1994
• 13: Gene H. Golub , Charles F. Van Loan, Matrix computations (3rd ed.), Johns Hopkins University Press, Baltimore, MD, 1996
• 14: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 15: Ned Greene, Environment mapping and other applications of world projections, IEEE Computer Graphics and Applications, v.6 n.11, p.21-29, November 1986 [doi>10.1109/MCG.1986.276658]
• 16: Hansen, M., Anandan, P., Dana, K., van der Wal, G., and Burt, P. 1994. Real-time scene stabilization and mosaic construction. In <i>IEEE Workshop on Applications of Computer Vision (WACV'94)</i>, Sarasota, Florida, pp. 54-62.
• 17: Richard I. Hartley, Self-calibration from multiple views with a rotating camera, Proceedings of the third European conference on Computer vision (vol. 1), p.471-478, June 1994, Stocklholm, Sweden
• 18: Heckbert, P. 1989. <i>Fundamentals of Texture Mapping and Image Warping</i>. Master's Thesis, The University of California at Berkeley.
• 19: Horn, B.K.P. 1987. Closed-form solution of absolute orientation using unit quaternions. <i>Journal of the Optical Society of America A</i>, 4(4):629-642.
• 20: Horn, B.K.P., Hilden, H.M., and Negahdaripour, S. 1988. Closed-form solution of absolute orientation using orthonormal matrices. <i>Journal of the Optical Society of America A</i>, 5(7):1127-1135.
• 21: Michal Irani , Shmuel Peleg, Improving resolution by image registration, CVGIP: Graphical Models and Image Processing, v.53 n.3, p.231-239, May 1991 [doi>10.1016/1049-9652(91)90045-L]
• 22: M. Irani , P. Anandan , S. Hsu, Mosaic based representations of video sequences and their applications, Proceedings of the Fifth International Conference on Computer Vision, p.605, June 20-23, 1995
• 23: Irani, M., Hsu, S., and Anandan, P. 1995b. Video compression using mosaic representations. <i>Signal Processing: Image Communication </i>, 7:529-552.
• 24: K. Kanatani, Analysis of 3-D Rotation Fitting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.16 n.5, p.543-549, May 1994 [doi>10.1109/34.291441]
• 25: Kang, S.B. 1997. A survey of image-based rendering techniques. Technical Report 97/4, Digital Equipment Corporation, Cambridge Research Lab.
• 26: Sing Bing Kang , Richard Weiss, Characterization of errors in compositing panoramic images, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.103, June 17-19, 1997
• 27: Kumar, R., Anandan, P., and Hanna, K. 1994. Shape recovery from multiple views: A parallax based approach. In <i>Image Understanding Workshop</i>, Morgan Kaufmann Publishers: Monterey, CA, pp. 947-955.
• 28: Rakesh Kumar , P. Anandan , michal Irani , James Bergen , Keith Hanna, Representation of Scenes from Collections of Images, Proceedings of the IEEE Workshop on Representation of Visual Scenes, p.10, June 21-21, 1995
• 29: Didier Le Gall, MPEG: a video compression standard for multimedia applications, Communications of the ACM, v.34 n.4, p.46-58, April 1991 [doi>10.1145/103085.103090]
• 30: Lee, M.-C. et al. 1997. A layered video object coding system using sprite and affine motion model. <i>IEEE Transactions on Circuits and Systems for Video Technology</i>, 7(1):130-145.
• 31: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 32: Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application in stereo vision. In <i>Seventh International Joint Conference on Artificial Intelligence (IJCAI-81)</i>, Vancouver, pp. 674-679.
• 33: Malde, H.E. 1983. Panoramic photographs. <i>American Scientist</i>, 71(2):132-140.
• 34: Mann, S. and Picard, R.W. 1994. Virtual bellows: Constructing high-quality images from video. In <i>First IEEE International Conference on Image Processing (ICIP-94)</i>, Austin, Texas, pp. 363-367.
• 35: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• 36: Meehan, J. 1990. <i>Panoramic Photography</i>. Watson-Guptill.
• 37: Milgram, D.L. 1977. Adaptive techniques for photomosaicking. <i>IEEE Transactions on Computers</i>, C-26(11):1175-1180.
• 38: Moffitt, F.H. and Mikhail, E.M. 1980. <i>Photogrammetry</i>, 3rd edn. Harper & Row: New York.
• 39: Richard M. Murray , S. Shankar Sastry , Li Zexiang, A Mathematical Introduction to Robotic Manipulation, CRC Press, Inc., Boca Raton, FL, 1994
• 40: Shree K. Nayar, Catadioptric Omnidirectional Camera, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.482, June 17-19, 1997
• 41: Gregory M. Nielson, Scattered Data Modeling, IEEE Computer Graphics and Applications, v.13 n.1, p.60-70, January 1993 [doi>10.1109/38.180119]
• 42: S. Peleg , J. Herman, Panoramic mosaics by manifold projection, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.338, June 17-19, 1997
• 43: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical recipes in C (2nd ed.): the art of scientific computing, Cambridge University Press, New York, NY, 1992
• 44: Quam, L.H. 1984. Hierarchical warp stereo. In <i>Image Understanding Workshop</i>, Science Applications International Corporation, New Orleans, Louisiana, pp. 149-155.
• 45: Azriel Rosenfeld , Avinash C. Kak, Digital Picture Processing, Academic Press, Inc., Orlando, FL, 1982
• 46: Yoichi Sato , Mark D. Wheeler , Katsushi Ikeuchi, Object shape and reflectance modeling from observation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.379-387, August 1997 [doi>10.1145/258734.258885]
• 47: Sawhney, H.S. 1994. Simplifying motion and structure analysis using planar parallax and image warping. In <i>Twelfth International Conference on Pattern Recognition (ICPR'94)</i>, Jerusalem, Israel, IEEE Computer Society Press, pp. 403-408.
• 48: Harpreet S. Sawhney , Serge Ayer, Compact Representations of Videos Through Dominant and Multiple Motion Estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.8, p.814-830, August 1996 [doi>10.1109/34.531801]
• 49: Harpreet S. Sawhney , Steve Hsu , Rakesh Kumar, Robust Video Mosaicing through Topology Inference and Local to Global Alignment, Proceedings of the 5th European Conference on Computer Vision-Volume II, p.103-119, June 02-06, 1998
• 50: Shi, J. and Tomasi, C. 1994. Good features to track. In <i>IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'94)</i>, Seattle, Washington, IEEE Computer Society, pp. 593-600.
• 51: Shum, H.-Y. and Szeliski, R. 1997. Panoramic image mosaicing. Technical Report MSR-TR-97-23, Microsoft Research.
• 52: Heung-Yeung Shum , Richard Szeliski, Construction and Refinement of Panoramic Mosaics with Global and Local Alignment, Proceedings of the Sixth International Conference on Computer Vision, p.953, January 04-07, 1998
• 53: G. P. Stein, Accurate internal camera calibration using rotation, with analysis of sources of error, Proceedings of the Fifth International Conference on Computer Vision, p.230, June 20-23, 1995
• 54: Szeliski, R. 1994. Image mosaicing for tele-reality applications. In <i>IEEE Workshop on Applications of Computer Vision (WACV'94)</i>, Sarasota, Florida, IEEE Computer Society, pp. 44-53.
• 55: Richard Szelicki , Sing Bing Kang, Direct Method for Visual Scene Reconstruction, Proceedings of the IEEE Workshop on Representation of Visual Scenes, p.26, June 21-21, 1995
• 56: Szeliski, R. 1996. Video mosaics for virtual environments. <i>IEEE Computer Graphics and Applications</i>, 16(2):22-30.
• 57: Richard Szeliski , Heung-Yeung Shum, Creating full view panoramic image mosaics and environment maps, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.251-258, August 1997 [doi>10.1145/258734.258861]
• 58: Qi Tian , Michael N Huhns, Algorithms for subpixel registration, Computer Vision, Graphics, and Image Processing, v.35 n.2, p.220-233, Aug. 1986 [doi>10.1016/0734-189X(86)90028-9]
• 59: Tsai, R.Y. 1987. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. <i>IEEE Journal of Robotics and Automation</i>, RA-3(4):323-344.
• 60: Hank Weghorst , Gary Hooper , Donald P. Greenberg, Improved Computational Methods for Ray Tracing, ACM Transactions on Graphics (TOG), v.3 n.1, p.52-69, Jan. 1984 [doi>10.1145/357332.357335]
• 61: Lee Westover, Footprint evaluation for volume rendering, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.367-376, September 1990, Dallas, TX, USA
• 62: Lance Williams, Pyramidal parametrics, ACM SIGGRAPH Computer Graphics, v.17 n.3, p.1-11, July 1983
• 63: George Wolberg, Digital Image Warping, IEEE Computer Society Press, Los Alamitos, CA, 1994
• 64: Wolf, P.R. 1974. <i>Elements of Photogrammetry</i>. McGraw-Hill: New York.
• 65: Daniel N. Wood , Adam Finkelstein , John F. Hughes , Craig E. Thayer , David H. Salesin, Multiperspective panoramas for cel animation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.243-250, August 1997 [doi>10.1145/258734.258859]
• 66: Yalin Xiong , Ken Turkowski, Creating Image-Based VR Using a Self-Calibrating Fisheye Lens, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.237, June 17-19, 1997

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Range data

General Terms:
Algorithms, Experimentation, Measurement, Theory

Keywords:
global alignment, image mosaics, local alignment, panoramas, parametric motion estimation, virtual environment modeling