Consumer video retargeting: context assisted spatial-temporal grid optimization

ABSTRACT

Pervasive multimedia devices require accurate video retargeting, especially in connected consumer electronics platforms. In this paper, we present a context assisted spatialtemporal grid scheme for consumer video retargeting. First, we parse consumer videos from low-level features to highlevel visual concepts, combining visual attention into a more accurate importance description. Then, a semantic importance map is built up representing the spatial importance and temporal continuity, which is incorporated with a 3D rectilinear grid scaleplate to map frames to the target display, thereby keeping the aspect ratio of semantically salient objects as well as the perceptual coherency. Extensive evaluations were done on two popular video genres, sports and advertisements. The comparison with state-of-the-art approaches on both images and videos have demonstrated the advantages of the proposed approach.

REFERENCES

• 1: Technical report, comScore 2008.
• 2: A. Albiol, M. J. C. Full'a, and A. Albiol. Detection of tv commercials. In ICASSP, pages 541--544, 2004.
• 3: A. Albiol, M. J. C. Fulla, A. Albiol, and L. Torres. Commercials detection using hmms. In Int. Workshop on Image Analysis for Multimedia Interactive Services, 2004.
• 4: J. Assfalg, M. Bertini, C. Colombo, A. D. Bimbo, and W. Nunziati. Semantic annotation of soccer videos: automatic highlights identification. Comput. Vis. Image Underst., 2003.
• 5: S. Avidan and A. Shamir. Seam carving for content-aware image resizing. In ACM Trans. Graph, 2007.
• 6: P. Chang, M. Han, and Y. Gong. Extract highlights from baseball game video with hidden markov models. In ICIP, 2002.
• 7: L.-Y. Duan, J. Wang, Y. Zheng, J. S. Jin, H. Lu, and C. Xu. Segmentation, categorization, and identification of commercials from tv streams using multimodal analysis. In ACM Multimedia, 2006.
• 8: L.-Y. Duan, M. Xu, Q. Tian, and C. sheng Xu. Nonparametric color characterization using mean shift. In ACM Multimedia, 2003.
• 9: L.-Y. Duan, M. Xu, Q. Tian, C. Xu, and J. S. Jin. A unified framework for semantic shot classification in sports video. In IEEE Transactions on Multimedia, volume 7(6), 2005.
• 10: A. Ekin, A. M. Tekalp, and R. Mehrotra. Automatic soccer video analysis and summarization. IEEE Transactions on Image Processing, 2003.
• 11: X. Fan, X. Xie, H.-Q. Zhou, and W.-Y. Ma. Looking into video frames on small displays. In ACM Multimedia, 2003.
• 12: X.-S. Hua, L. Lu, and H.-J. Zhang. Robust learning-based tv commercial detection. In ICME, 2005.
• 13: C. Huang, H. Ai, Y. Li, and S. Lao. Vector boosting for rotation invariant multi-view face detection. In ICCV, 2005.
• 14: F. Li. Knowledge transfer in learning to recognize visual objects classes. In ICDL, 2006.
• 15: R. Lienhart, C. Kuhmunch, , and W. Effelsberg. On the detection and recognition of television commercials. In Int. Conf. on Multimedia Computing and Systems, 1997.
• 16: F. Liu and M. Gleicher. Video retargeting automating pan and scan. In Acm Multimedia, 2006.
• 17: T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum. Learning to detect a salient object. In CVPR, 2007.
• 18: A. Loui, J. Luo, S.-F. Chang, D. Ellis, W. Jiang, L. Kennedy, K. Lee, and A. Yanagawa. Kodak's consumer video benchmark data set: concept definition and annotation. In ACM MIR, 2007.
• 19: D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
• 20: F. Megino, J. Sanchez, and V. Lopez. Virtual camera tools for an image2video application. In International Workshop on Image Analysis for Multimedia Interactive Services, 2008.
• 21: T. Mei, X.-S. Hua, and S. Li. Contextual in-image advertising. In ACM Multimedia, 2008.
• 22: T. Mei, X.-S. Hua, L. Yang, and S. Li. Videosense - towards effective online video advertising. In ACM Multimedia, 2007.
• 23: M. Mizutani, S. Ebadollahi, and S.-F. Chang. Commercial detection in heterogeneous video streams using fused multi-modal and temporal features. In ICASSP, 2005.
• 24: M. Rubinstein, A. Shamir, and S. Avidan. Improved seam carving for video retargeting. In Siggraph, 2008.
• 25: V. Setlur and S. Takagi. Automatic image retargetting. In MUM, 2005.
• 26: Y.-P. Tan, D. D. Saur, S. R. Kulkarni, and P. J. Ramadge. Rapid estimation of camera motion from compressed video with application to video annotation. IEEE Trans. on CSVT, 2000.
• 27: C. Tao, J. Jia, and H. Sun. Active window oriented dynamic video retargeting. In Workshop On Dynamical Vision,ICCV, 2007.
• 28: D. Thomas, D. Philippe, and N. Hermann. Pan, zoom, scan time-coherent, trained automatic video cropping. In CVPR, 2008.
• 29: L. Vincent and P. Soille. Watersheds in digital spaces: An efficient algorithm based on immersion simulations. In IEEE PAMI, 1991.
• 30: J.Wang, M. J. Reinders, R. L. Lagendijk, J. Lindenberg, and M. S.Kankanhalli. Video content representation on tiny devices. In ICME, 2004.
• 31: Y.-S. Wang, C.-L. Tai, O. Sorkine, and T.-Y. Lee. Optimized scale-and-stretch for image resizing. In ACM Trans. Graph, 2008.
• 32: L.Wolf, M. Guttmann, and D. Cohen-Or. Non-homogeneous content-driven video retargeting. In ICCV, 2007.
• 33: L. Xie, P. Xu, S.-F. Chang, A. Divakaran, and H. Sun. Structure analysis of soccer video with domain knowledge and hidden markov models. Pattern Recogn. Lett., 2004.
• 34: J. Yang, Y. G. Jiang, A. G. Hauptmann, and C. W. Ngo. Evaluating bag-of-visual-words representations in scene classification. In MIR, 2007.
• 35: H. Zhang, A. Kankanhalli, and S. Smoliar. Automatic partitioning of full-motion video. In ACM Multimedia Systems, 1993.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms

Keywords:
context analysis, video retargeting