FSCAV: fast seam carving for size adaptation of videos

ABSTRACT

The presentation of multimedia data and especially of high resolution videos on small mobile devices is still a great challenge today. Both cropping of borders and scaling of frames may result in the removal of essential content of videos or lost details due to the reduced size of the visual content. Another major problem emerges if the aspect ratio of the original video and the display of the mobile device differ. User evaluations indicate that changing the aspect ratio may reduce the visual quality of videos significantly. In this paper, we present the new FSCAV algorithm (Fast Seam Carving for Size Adaptation of Videos) to adapt the size of videos to the limited display resolution and different aspect ratios of handheld mobile devices. The general idea of the seam carving algorithm for still images is to remove seams in images so that the essential content is preserved. We extended this technique which works very well for images to create videos without jitter or visible artifacts. A major feature of our FSCAV algorithm is the low computational complexity which enables an efficient adaptation of videos to small screens. Nevertheless, severe distortions are clearly visible in some shots of the adapted videos. We present a new heuristic to identify shots with such a low visual quality. If the quality drops below a threshold, a different adaptation technique is used for this shot (e.g., scaling or cropping). User evaluations confirm a very high visual quality of our approach.

REFERENCES

• 1: S. Avidan and A. Shamir. Seam carving for content-aware image resizing. ACM Transactions on Graphics, SIGGRAPH 2007, 26(3), 2007.
• 2: Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In IEEE Transactions on PAMI, volume 26(9), pages 1124--1137, September 2004.
• 3: H. El-Alfy, D. Jacobs, and L. Davis. Multi-scale video cropping. MM '07: ACM international conference on Multimedia, pages 97--106, 2007.
• 4: X. Fan, X. Xie, H.-Q. Zhou, and W.-Y. Ma. Looking into video frames on small displays. MM '03: ACM international conference on Multimedia, pages 247--250, 2003.
• 5: D. Farin. Automatic Video Segmentation Employing Object/Camera Modeling. PhD thesis, Technische Universiteit Eindhoven, Einhoven, The Netherlands, 2005.
• 6: D. Farin, T. Haenselmann, S. Kopf, G. Kühne, and W. Effelsberg. Segmentation and classification of moving video objects. In B. Furht and O. Marques, editors, Handbook of Video Databases: Design and Applications, volume 8 of Internet and Communications Series, pages 561--591. CRC Press, Boca Raton, FL, USA, September 2003.
• 7: M. Fischler and R. Bolles. Random sample concensus: A paradigm for model fitting with applications to image analysis and automated cartography. In Communications ACM, volume 24(6), pages 381--395. ACM Press, 1981.
• 8: C. Harris and M. Stephens. A combined corner and edge detector. In Proceedings of Alvey Vision Conference, pages 147--151, 1988.
• 9: S. Kopf and W. Effelsberg. Mobile cinema: Canonical processes for video adaptation. In Multimedia Systems, volume 14(6), pages 369--375. Springer, December 2008.
• 10: S. Kopf, T. Haenselmann, D. Farin, and W. Effelsberg. Automatic generation of summaries for the web. In Proceedings of IS&T/SPIE conference on Storage and Retrieval for Media Databases, volume 5307, pages 417--428, Januar 2004.
• 11: S. Kopf, F. Lampi, T. King, and W. Effelsberg. Automatic scaling and cropping of videos for devices with limited screen resolution. In ACM Multimedia (video program session), pages 957--958, October 2006.
• 12: S. Kopf, F. Lampi, T. King, and W. Effelsberg. Automatic scaling and cropping of videos for devices with limited screen resolution. In Proceedings of the 14th ACM international conference on Multimedia, pages 957--958. ACM Press, Oktober 2006.
• 13: F. Liu and M. Gleicher. Video retargeting: Automating pan and scan. MM '06: ACM international conference on Multimedia, pages 241--250, 2006.
• 14: D. G. Lowe. Distinctive image features from scale-invariant keypoints. In International Journal of Computer Vision, volume 60(2), pages 91--110. Kluwer Academic Publishers, November 2004.
• 15: H. Moravec. Visual mapping by a robot rover. In Proceedings of the 6th International Joint Conference on Artificial Intelligence, pages 599 -- 601, August 1979.
• 16: M. Rubinstein, S. Avidan, and A. Shamir. Improved seam carving for video retargeting. ACM Transactions on Graphics, SIGGRAPH 2008, 27(3), 2008.
• 17: A. Shamir and S. Avidan. Seam carving for media retargeting. Commun. ACM, 52(1):77--85, 2009.
• 18: S. M. Smith and J. M. Brady. Susan - new approach to low level image processing. In International Journal of Computer Vision (IJCV), volume 23(1), pages 45 -- 78, May 1997.
• 19: C. Tao, J. Jia, and H. Sun. Active window oriented dynamic video retargeting. Proceedings of the Workshop on Dynamical Vision, ICCV 2007, 2007.
• 20: J. Wang, M. Reinders, R. Lagendijk, J. Lindenberg, and M. Kankanhalli. Video content presentation on tiny devices. ICME '04: IEEE International Conference on Multimedia and Expo, pages 1711--1714, 2004.
• 21: L. Wolf, M. Guttmann, and D. Cohen-Or. Non-homogeneous content-driven video-retargeting. In Proceedings of the Eleventh IEEE International Conference on Computer Vision (ICCV-07), 2007.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Algorithms

Keywords:
seam carving, video adaptation, video retargeting