A user attention model for video summarization

ABSTRACT

Automatic generation of video summarization is one of the key techniques in video management and browsing. In this paper, we present a generic framework of video summarization based on the modeling of viewer's attention. Without fully semantic understanding of video content, this framework takes advantage of understanding of video content, this framework takes advantage of computational attention models and eliminates the needs of complex heuristic rules in video summarization. A set of methods of audio-visual attention model features are proposed and presented. The experimental evaluations indicate that the computational attention based approach is an effective alternative to video semantic analysis for video summarization.

REFERENCES

• 1: H.J Zhang, et al, "An integrated system for content-based video retrieval and browsing," Pattern Recognition, vol. 30, no.4, pp. 643--658, 1997.
• 2: W. Wolf, "Key frame selection by motion analysis," Proc. of ICASSP'96, vol.2, pp. 1228--1231, 1996.
• 3: Y. Zhuang, et al, "Adaptive key frame extraction using unsupervised clustering," Proc. of ICIP'98, 1998.
• 4: A. Girgensohn, and J. Boreczky, "Time-constrained key frame selection technique," Proc. of ICMCS, pp. 756--761, 1999.
• 5: F. Dufaux, "Key frame selection to represent a video," Proc. of ICME 2000.
• 6: P. Campisi, A. Longari and A. Neri, "Automatic key frame selection using a wavelet based approach," Proc. of SPIE, vol. 3813, pp. 861--872, July 1999.
• 7: Changick Kim , Jenq-Neng Hwang, An integrated scheme for object-based video abstraction, Proceedings of the eighth ACM international conference on Multimedia, p.303-311, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354512]
• 8: Nosa Omoigui , Liwei He , Anoop Gupta , Jonathan Grudin , Elizabeth Sanocki, Time-compression: systems concerns, usage, and benefits, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.136-143, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303017]
• 9: Michael A. Smith, Video Skimming and Characterization through the Combination of Image and Language Understanding Techniques, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.775, June 17-19, 1997
• 10: Jeho Nam , Ahmed H. Tewfik, Dynamic video summarization and visualization, Proceedings of the seventh ACM international conference on Multimedia (Part 2), p.53-56, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319878.319892]
• 11: A. Stefanidis , P. Partsinevelos , P. Agouris , P. Doucette, Summarizing Video Datasets in the Spatiotemporal Domain, Proceedings of the 11th International Workshop on Database and Expert Systems Applications, p.906, September 06-08, 2000
• 12: X. Orriols, X. Binefa, "An EM algorithm for video summarization, generative model approach," Proc. of ICCV, 2001.
• 13: Y.H. Gong, X. Liu, "Video Summarization Using Singular Value Decomposition," Proc. of CVPR, June, 2000.
• 14: L. Itti, C. Koch, "Computational Modeling of Visual Attention," Nature Reviews Neuroscience, Vol. 2, No. 3, pp. 194--203, Mar. 2001.
• 15: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 16: L. Itti, C. Koch, "A Comparison of Feature Combination Strategies for Saliency-Based Visual Attention Systems," Proc. of SPIE Human Vision and Electronic Imaging IV (HVEI'99), San Jose, CA, Vol. 3644, pp. 473--82, Jan 1999.
• 17: Albert Ali Salah , Ethem Alpaydin , Lale Akarun, A Selective Attention-Based Method for Visual Pattern Recognition with Application to Handwritten Digit Recognition and Face Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.3, p.420-425, March 2002 [doi>10.1109/34.990146]
• 18: Maik Bollmann , Rainer Hoischen , Bärbel Mertsching, Integration of Static and Dynamic Scene Features Guiding Visual Attention, Mustererkennung 1997, 19. DAGM-Symposium, p.483-490, September 15-17, 1997
• 19: Y.F Ma, H.J. Zhang, "A Model of Motion Attention for Video Skimming," Proc. of ICIP, 2002.
• 20: Y.F. Ma, H.J. Zhang "A New Perceived Motion based Shot Content Representation," Proc. of ICIP, 2001.
• 21: Stan Z. Li , Long Zhu , ZhenQiu Zhang , Andrew Blake , HongJiang Zhang , Harry Shum, Statistical Learning of Multi-view Face Detection, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.67-81, May 28-31, 2002
• 22: Lie Lu , Hao Jiang , HongJiang Zhang, A robust audio classification and segmentation method, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500173]
• 23: L. Lu, Stan Li, H. J. Zhang, "Content-based Audio Segmentation Using Support Vector Machines". Proc. of ICME 2001, pp956--959, Tokyo, Japan, 2001.
• 24: HongJiang Zhang , Atreyi Kankanhalli , Stephen W. Smoliar, Automatic partitioning of full-motion video, Multimedia Systems, v.1 n.1, p.10-28, June 1993 [doi>10.1007/BF01210504]

INDEX TERMS

Keywords:
attention model, skimming, video content analysis, video summarization