A generic virtual content insertion system based on visual attention analysis

ABSTRACT

This paper presents a generic Virtual Content Insertion (VCI) system based on visual attention analysis. VCI is an emerging application of video analysis and has been used in video augmentation and advertisement insertion. There are three critical issues for a VCI system: when (time), where (place) and how (method) to insert the Virtual Content (VC) into the video. Our system selects the insertion time and place by performing temporal and spatial attention analysis, which predicts the attention change along time and the attended region over space. In order to enable the inserted VC to be noticed by audience while not to interrupt the audience's viewing experience to the original content, the VC should be inserted at the time when the video content attracts much audience attention and at the place where attracts less. Dynamic insertion is performed by using Global Motion Estimation (GME) and affine transformation. Our VCI system is able to obtain an optimal balance between the notice of the VC by audience and disruption of viewing experience to the original content. Extensive subjective evaluations based on user study on the VCI result have verified the effectiveness of the system.

REFERENCES

• 1: Xinguo Yu , Xin Yan , Tran Thi Phuong Chi , Loong Fah Cheong, Inserting 3D projected virtual content into broadcast tennis video, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180767]
• 2: K. Wan, C. Xu, "Automatic Content Placement in Sports Highlights", IEEE International Conference on Multimedia & Expo, pp: 1893--1896, 2006.
• 3: C. Xu, K. W. Wan, S. H. Bui, Q. Tian, "Implanting Virtual Advertisement into Broadcast Soccer Video", Pacific-Rim Conference on Multimedia, pp: 264--271, 2004.
• 4: Yiqun Li , Kong Wah Wan , Xin Yan , Changsheng Xu, Real time advertisement insertion in baseball video based on advertisement effect, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101221]
• 5: M. Tamir, et al., "Method and Apparatus for Automatic Electronic Replacement of Billboards in a Video Image", US Patent 6,292,227, 2001.
• 6: S. Deshpande, et al., "Method and Apparatus for Including Virtual ADs in Video Presentations", US Patent, 7,158,666, 2007.
• 7: Tao Mei , Xian-Sheng Hua , Linjun Yang , Shipeng Li, VideoSense: towards effective online video advertising, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291467]
• 8: H. Liu, S. Jiang, Q. Huang and C. Xu. "Lower Attentive Region Detection for Virtual Content Insertion", IEEE International Conference on Multimedia & Expo, 2008.
• 9: S. Das, et al., "Method of Tracking Scene Motion for Live Video Insertion Systems", US Patent 5,808,695, 1998.
• 10: Kreitman, et al., "Method and System for Perspectively Distorting an Image and Implanting Same into a Video Stream", US Patent 5,731,846, 1998.
• 11: Y.-F. Ma, X.-S. Hua, L. Lu, H.-J. Zhang. "A Generic Framework of User Attention Model and Its Application in Video Summarization", IEEE Trans on Multimedia, Vol. 7, No. 5, pp: 907--919, 2005.
• 12: Yun Zhai , Mubarak Shah, Visual attention detection in video sequences using spatiotemporal cues, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180824]
• 13: Olivier Le Meur , Patrick Le Callet , Dominique Barba , Dominique Thoreau, A Coherent Computational Approach to Model Bottom-Up Visual Attention, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.5, p.802-817, May 2006 [doi>10.1109/TPAMI.2006.86]
• 14: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 15: Huiying Liu , Shuqiang Jiang , Qingming Huang , Changsheng Xu , Wen Gao, Region-based visual attention analysis with its application in image browsing on small displays, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291298]
• 16: Yiqun Hu , Deepu Rajan , Liang-Tien Chia, Robust subspace analysis for detecting visual attention regions in images, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101306]
• 17: Ying Li , Yu-Fei Ma , Hong-Jiang Zhang, Salient region detection and tracking in video, Proceedings of the 2003 International Conference on Multimedia and Expo, p.269-272, July 06-09, 2003
• 18: Yaoru Sun , Robert Fisher, Object-based visual attention for computer vision, Artificial Intelligence, v.146 n.1, p.77-123, May 2003 [doi>10.1016/S0004-3702(02)00399-5]
• 19: T. N. Topper, "Selection Mechanisms in Human and Machine Vision", Ph.D. Thesis, University of Waterloo, 1991.
• 20: N. D. B. Bruce, "Features That Draw Visual Attention: an Information Theoretic Perspective", Neurocomputing, Vol. 65-66, pp: 125--133, 2005.
• 21: María T. López , Antonio Fernández-Caballero , Miguel A. Fernández , José Mira , Ana E. Delgado, Visual surveillance by dynamic visual attention method, Pattern Recognition, v.39 n.11, p.2194-2211, November, 2006 [doi>10.1016/j.patcog.2006.04.018]
• 22: Laurent Itti , Pierre Baldi, A Principled Approach to Detecting Surprising Events in Video, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.631-637, June 20-26, 2005 [doi>10.1109/CVPR.2005.40]
• 23: L. Itti. "Automatic Foveation for Video Compression Using a Neurobiological Model of Visual Attention", IEEE Transactions on Image Processing, Vol. 13, No. 10, pp: 1304--1318, 2004.
• 24: G. Qiu, X. Gu, Z. Chen, Q. Chen and C. Wang, "An Information Theoretic Model of Spatiotemporal Visual Saliency", IEEE International Conference on Multimedia & Expo, pp: 1806--1809, 2007.
• 25: J. You, G. Liu, L. Sun, H. Li, "A Multiple Visual Models Based Perceptive Analysis Framework for Multilevel Video Summarization", IEEE Trans. Circuits and Systems for Video Technology, Vol. 17, No. 3, pp: 273--285, 2007.
• 26: C. Liu, H. Liu, S. Jiang, Q. Huang, Y. Zheng and W. Zhang: "JDL at Trecvid 2006 Shot Boundary Detection", TRECVID 2006 Workshop (2006).
• 27: R. E. Soodak, "Two-dimensional modeling of visual receptive fields using Gaussian subunits", Proc. Antl. Acad, Sci. USA, Vol. 83, pp: 9259--9263, 1986.
• 28: E. T. Rolls, N. C. Aggelopoulos, and F. Zheng, "Effective size of receptive fields of inferior temporal visual cortex neurons in natural scenes", The Journal of Neuroscience, Vol. 23, No. 1, pp:339--348, 2003.
• 29: L.-Y. Duan, M. Xu, Q. Tian, C.-S. Xu, J. S. Jin, "A Unified Framework for Semantic Shot Classification in Sports Video", IEEE Trans on Multimedia, Vol. 7, No. 6, pp: 1066--1083, 2005.
• 30: F. Dufaux and J. Konrad, "Efficient, robust, and fast global motion estimation for video coding," IEEE Trans. Image Processing, vol. 9, no. 3, pp. 497--501, 2000.
• 31: X. Chen, J. Yang, J. Zhang, and A. Waibel. "Automatic Detection and Recognition of Signs from Natural Scenes", IEEE Transactions on Image Processing, Vol. 13, No.1, pp: 87--99, 2004.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Perceptual reasoning

General Terms:
Algorithms, Experimentation

Keywords:
virtual content insertion, visual attention