Video summarization using personal photo libraries

ABSTRACT

In this paper, we propose a video summarization system which takes into account users' individual preferences by using their personal photo libraries. Nowadays it is common, especially among people of younger generations, to store thousands of photos inside their PCs and manage them using software such as iPhoto and Picasa. These personal photo libraries contain rich information about the user's tastes, personalities, and lifestyles. Since still photos are in many aspects similar to video as a medium, we assume that these personal photo libraries can be used to estimate users' preferences on video summarization.Our system first divides a movie into short segments, and uses image classification techniques to judge whether each segment is meaningful to the user or not. If many photos with contents similar to the segment can be found in the user's photo library, the segment is judged as being "important" to the user. Conventional image classification techniques use public or commercial photo databases as training data, while our system uses personal photo libraries. This difference leads to the need of several modifications in the classification process.We have implemented a prototype version of our system, and have validated the effectiveness of our approach through evaluating both the accuracy of our image classification algorithm, and users' subjective satisfaction levels of the summarization results.

REFERENCES

• 1: Carson, C., Belongie, S., Greenspan, H. Malik, J. Blobworld: Image segmentation using Expectation-Maximization and its application to image querying. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 1999.
• 2: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 3: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 4: Ju, S. X., Black, M. J., Minneman, S., Kimber, D. Summarization of Video-taped Presentations: Automatic Analysis of Motion and Gesture. In IEEE Transactions on Circuits and Systems for Video Technology, 1998.
• 5: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 6: Yu-Fei Ma , Lie Lu , Hong-Jiang Zhang , Mingjing Li, A user attention model for video summarization, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641116]
• 7: Tao Mei , Xian-Sheng Hua , He-Qin Zhou, Tracking users' capture intention: a novel complementary view for home video content analysis, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101269]
• 8: Mori, Y., Takahashi, H., Oka, R. Image-to-word transformation based on dividing and vector quantizing images with words. In First International Workshop on Multimedia Intelligent Storage and Retrieval Management (MISRM 99), 1999.
• 9: Parshin, V., Chen, L. Video Summarization Based on User-Defined Constraints and Preferences. In Proceedings of RIAO 2004, 2004.
• 10: Greg Pass , Ramin Zabih, Comparing images using joint histograms, Multimedia Systems, v.7 n.3, p.234-240, May 1999 [doi>10.1007/s005300050125]
• 11: Smith, M. A., Kanade, T. Video Skimming and Characterization through the Combination of Image and Language Understanding Techniques. In CMU Technology Report, 1997.
• 12: Yow, D., Yeo, B. L., Yeung, M., Liu, G. Analysis and Presentation of Soccer Highlights from Digital Video. In Proceedings of Second Asian Conference on Computer Vision, 1995.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS

General Terms:
Algorithms, Performance

Keywords:
image classification, personal photo library, video summarization