User-adaptive home video summarization using personal photo libraries

ABSTRACT

In this paper, we propose a user-adaptive video summarization system which accounts for individual preferences by analyzing contents of the user's personal photo library. Nowadays, it is common practice for people to keep thousands of photos in their PCs, taken with their digital cameras. Due to the similarities in characteristics between video and still photos, we assume that these libraries can be used to infer each user's preferences on video summarization. The system uses image classification techniques to determine which sections of the movie are "important" to the user, and performs summarization by cutting off sections regarded as unimportant, while taking care not to overly disrupt the continuity of the video. We have implemented a prototype of the system, and conducted a series of evaluation tests to assess its effectiveness. The results show that our overall approach has the potential to serve as a powerful automatic/semi-automatic video summarization solution.

REFERENCES

• 1: http://www.apple.com/ilife/imovie/
• 2: http://eyespot.com/
• 3: http://www.flickr.com/
• 4: http://www.jumpcut.com/
• 5: http://www.youtube.com/
• 6: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800]
• 7: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 8: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 9: Ju, S. X., Black, M. J., Minneman, S., Kimber, D. Summarization of Video-taped Presentations: Automatic Analysis of Motion and Gesture. In IEEE Transactions on Circuits and Systems for Video Technology, 1998.
• 10: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 11: Yu-Fei Ma , Lie Lu , Hong-Jiang Zhang , Mingjing Li, A user attention model for video summarization, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641116]
• 12: Tao Mei , Xian-Sheng Hua , He-Qin Zhou, Tracking users' capture intention: a novel complementary view for home video content analysis, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101269]
• 13: Mori, Y., Takahashi, H., Oka, R. Image-to-word transformation based on dividing and vector quantizing images with words. In Proc. of MISRM 1999.
• 14: Parshin, V., Chen, L. Video Summarization Based on User-Defined Constraints and Preferences. In Proceedings of RIAO 2004.
• 15: Greg Pass , Ramin Zabih, Comparing images using joint histograms, Multimedia Systems, v.7 n.3, p.234-240, May 1999 [doi>10.1007/s005300050125]
• 16: Smith, M. A., Kanade, T. Video Skimming and Characterization through the Combination of Image and Language Understanding Techniques. In CMU Technology Report, 1997.
• 17: Yow, D., Yeo, B. L., Yeung, M., Liu, G. Analysis and Presentation of Soccer Highlights from Digital Video. In Proc. of Second Asian Conference on Computer Vision, 1995.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS

General Terms:
Algorithms, Performance

Keywords:
image classification, personal photo library, video summarization