Discovering panoramas in web videos

ABSTRACT

While methods for stitching panoramas have been successful given proper source images, providing these source images still remains a burden. In this paper, we present a method to discover panoramic source images within widely available web videos. The challenge comes from the fact that many of these videos are not recorded intentionally for stitching panoramas. Our method aims to find segments within a video that work as panorama sources. Specifically, we determine a video segment to be a valid panorama source according to the following three criteria. First, its camera motion should cover a wide field-of-view of the scene. Second, its frames should be "mosaicable", which states that the inter-frame motion should observe the underlying conditions for stitching a panorama. Third, its frames should have good image quality. Based on these criteria, we formulate discovering panoramas in a video as an optimization problem that aims to find an optimal set of video segments as panorama sources. After discovering these panorama sources, we synthesize regular scene panoramas using them. When significant dynamics is detected in the sources, we fuse the dynamics into the scene panoramas to make activity synopses to convey the dynamics. Our experiment of querying panoramas from YouTube confirms the feasibility of using web videos as panorama sources and demonstrates the effectiveness of our method.

REFERENCES

• 1: P. Bhat, C. L. Zitnick, N. Snavely, A. Agarwala, M. Agrawala, B. Curless, M. Cohen, and S. B. Kang. Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering, 2007.
• 2: M. Brown , D. G. Lowe, Recognising Panoramas, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1218, October 13-16, 2003
• 3: Matthew Brown , David G. Lowe, Automatic Panoramic Image Stitching using Invariant Features, International Journal of Computer Vision, v.74 n.1, p.59-73, August 2007 [doi>10.1007/s11263-006-0002-3]
• 4: Peter J. Burt , Edward H. Adelson, A multiresolution spline with application to image mosaics, ACM Transactions on Graphics (TOG), v.2 n.4, p.217-236, October 1983 [doi>10.1145/245.247]
• 5: Yaron Caspi , Anat Axelrod , Yasuyuki Matsushita , Alon Gamliel, Dynamic stills and clip trailers, The Visual Computer: International Journal of Computer Graphics, v.22 n.9, p.642-652, September 2006 [doi>10.1007/s00371-006-0046-y]
• 6: Liang-Hua Chen , Yu-Chun Lai , Hong-Yuan Liao, Video Scene Extraction Using Mosaic Technique, Proceedings of the 18th International Conference on Pattern Recognition, p.723-726, August 20-24, 2006 [doi>10.1109/ICPR.2006.1173]
• 7: C. Cotsaces, N. Nikolaidis, and I. Pitas. Video shot detection and condensed representation. a review. Signal Processing Magazine, IEEE, 23(2):28--37, March 2006.
• 8: N. Damera-Venkata, T. Kite, W. Geisler, B. Evans, and A. Bovik. Image quality assessment based on a degradation model. IEEE Transactions on Image Processing, 9(4):636--650, 2000.
• 9: Martin A. Fischler , Robert C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, v.24 n.6, p.381-395, June 1981 [doi>10.1145/358669.358692]
• 10: Flickr. http://www.flickr.com/.
• 11: P. Heckbert. Fundamentals of texture mapping and image warping. Master's thesis, UC-Berkeley, Computer Science Division, EECS Department, 1989.
• 12: http://en.wikipedia.org/wiki/Panorama.
• 13: M. Irani , P. Anandan , S. Hsu, Mosaic based representations of video sequences and their applications, Proceedings of the Fifth International Conference on Computer Vision, p.605, June 20-23, 1995
• 14: Yan Ke , Xiaoou Tang , Feng Jing, The Design of High-Level Features for Photo Quality Assessment, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.419-426, June 17-22, 2006 [doi>10.1109/CVPR.2006.303]
• 15: Don Kimber , Jonathan Foote , Surapong Lertsithichai, FlyAbout: spatially indexed panoramic video, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500192]
• 16: X. Li. Blind image quality assessment. In IEEE ICIP, pages 449--453, 2002.
• 17: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 18: B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proc. of International Joint Conference on Artificial Intelligence pages 674--679, 1981.
• 19: R. Marzotto, A. Fusiello, and V. Murino. High resolution video mosaicing with global alignment. In IEEE CVPR, pages 692--698, 2004.
• 20: Yasuyuki Matsushita , Eyal Ofek , Xiaoou Tang , Heung-Yeung Shum, Full-Frame Video Stabilization, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.50-57, June 20-26, 2005 [doi>10.1109/CVPR.2005.166]
• 21: Shmuel Peleg , Benny Rousso , Alex Rav-Acha , Assaf Zomet, Mosaicing on Adaptive Manifolds, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.10, p.1144-1154, October 2000 [doi>10.1109/34.879794]
• 22: M. Rasheed, Z.and Shah. Scene detection in hollywood movies and tv shows. In IEEE CVPR, pages 343--348, 2003.
• 23: Drew Steedly , Chris Pal , Richard Szeliski, Efficiently Registering Video into Panoramic Mosaics, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1300-1307, October 17-20, 2005 [doi>10.1109/ICCV.2005.86]
• 24: G. Strang. Wavelet transforms versus fourier transforms. Bull. Amer. Math. Soc., 28:288--305, 1993.
• 25: Richard Szeliski, Video Mosaics for Virtual Environments, IEEE Computer Graphics and Applications, v.16 n.2, p.22-30, March 1996 [doi>10.1109/38.486677]
• 26: Richard Szeliski, Image alignment and stitching: a tutorial, Foundations and Trends® in Computer Graphics and Vision, v.2 n.1, p.1-104, January 2006 [doi>10.1561/0600000009]
• 27: Richard Szeliski , Heung-Yeung Shum, Creating full view panoramic image mosaics and environment maps, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.251-258, August 1997 [doi>10.1145/258734.258861]
• 28: Yukinobu Taniguchi , Akihito Akutsu , Yoshinobu Tonomura, PanoramaExcerpts: extracting and packing panoramas for video browsing, Proceedings of the fifth ACM international conference on Multimedia, p.427-436, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266396]
• 29: Laura Teodosio , Walter Bender, Salient stills, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.1 n.1, p.16-36, February 2005 [doi>10.1145/1047936.1047940]
• 30: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 31: H. Tong, M. Li, H. Zhang, and C. Zhang. Blur detection for digital images using wavelet transform. In IEEE ICME, 2004.
• 32: H. Tong, M. Li, H.-J. Zhang, and C. Zhang. No-reference quality assessment for jpeg2000 compressed images. In IEEE ICIP, pages 24--27, 2004.
• 33: Bala R. Vatti, A generic solution to polygon clipping, Communications of the ACM, v.35 n.7, p.56-63, July 1992 [doi>10.1145/129902.129906]
• 34: Z. Wang, A. Bovik, and B. Evans. Blind measurement of blocking artifacts in images. In IEEE ICIP, pages 981--984, 2000.
• 35: Z. Wang, H. Sheikh, and A. Bovik. No-reference perceptual quality assessment of jpeg compressed images. In IEEE ICIP, pages Vol I: 477--480, 2002.
• 36: Z. Wang, G. Wu, H. Sheikh, E. Simoncelli, E.-H. Yang, and A. Bovik. Quality-aware images. IEEE Transactions on Image Processing, 15(6):1680--1689, 2006.
• 37: Wei-Qi Yan , Mohan S Kankanhalli, Detection and removal of lighting & shaking artifacts in home videos, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641031]
• 38: YouTube. http://www.youtube.com/.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Algorithms

Keywords:
activity synopsis, discovering panorama, scene panorama