A time series clustering based framework for multimedia mining and summarization using audio features

ABSTRACT

Past work on multimedia analysis has shown the utility of detecting specific temporal patterns for different content genres. In this paper, we propose a unified, content-adaptive, unsupervised mining framework to bring out such temporal patterns from different multimedia genres. We formulate the problem of pattern discovery from video as a time series clustering problem. We treat the sequence of low/mid level audio-visual features extracted from the video as a time series and perform a temporal segmentation. The segmentation is based on eigenvector analysis of the affinity matrix constructed from statistical models estimated from the subsequences of the time series. We are thus able to detect transitions and outliers from a sequence of observations from a stationary background process. We define a confidence measure on each of the detected outliers as the probability that it is an outlier. Then, we establish a relationship between the mining parameters and the confidence measure. Furthermore the confidence measure can be used to rank the detected outliers in terms of their departures from the back-ground process. Our experimental results with sequences of low and mid level audio features extracted from sports video show that "highlight" events can be extracted effectively as outliers from a background process using the proposed framework. We proceed to show the effectiveness of the proposed framework in bringing out patterns from surveillance videos without any a priori knowledge. Finally, we show that such temporal segmentation into background and outliers, along with the ranking based on the departure from the background, can be used to generate content summaries of any desired length

REFERENCES

• 1: Aya Aner , John R. Kender, Video Summaries through Mosaic-Based Shot and Scene Clustering, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.388-402, May 28-31, 2002
• 2: Gang Wu , Yi Wu , Long Jiao , Yuan-Fang Wang , Edward Y. Chang, Multi-camera spatio-temporal fusion and biased sequence-data learning for security surveillance, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957126]
• 3: H.Pan, P.Van Beek and M.I.Sezan. Detection of slow-motion replay segments in sports video for highlights generation. Proc. IEEE International Conf. on Acoustics, Speech and Signal Processing (2001).
• 4: Hari Sundaram , Shih-Fu Chang, Determining computable scenes in films and their structures using audio-visual memory models, Proceedings of the eighth ACM international conference on Multimedia, p.95-104, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354440]
• 5: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.731, June 17-19, 1997
• 6: Lawrence Rabiner , Biing-Hwang Juang, Fundamentals of speech recognition, Prentice-Hall, Inc., Upper Saddle River, NJ, 1993
• 7: L.Xie, S.-F.Chang, A.Divakaran, H.Sun. Unsupervised mining of statistical temporal structures in video. Video Mining, Azriel Rosenfeld, David Doermann, Daniel Dementhon Eds, Kluwer Academic Publishers (2003).
• 8: M.P.Wand and M.C.Jones. Kernel smoothing. London:Chapman & Hall (1995).
• 9: Rainer Lienhart, Automatic text recognition for video indexing, Proceedings of the fourth ACM international conference on Multimedia, p.11-20, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244137]
• 10: R.Radhakrishnan, Z.Xiong, A.Divakaran and T.Kan. Time series analysis and segmentation using eigenvectors for mining semantic audio label sequences. ICME (2004).
• 11: S.J.Sheather and M.C.Jones. A reliable data-based bandwidth selection method for kernel density estimation. J.R. Statist. Society (1991).
• 12: Winston Hsu and Shih-Fu Chang. A statistical framework for fusing mid-level perceptual features in news story segmentation. Proc. of ICME (2003).
• 13: Ying Li , C.-C. Jay Kuo, Content-based video analysis, indexing and representation using multimodal information, 2003

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

General Terms:
Algorithms, Design, Experimentation, Theory

Keywords:
audio classification, time series analysis, video summarization