Multi-modal scene segmentation using scene transition graphs

ABSTRACT

In this work the problem of automatic decomposition of video into elementary semantic units, known in the literature as scenes, is addressed. Two multi-modal automatic scene segmentation techniques are proposed, both building upon the Scene Transition Graph (STG). In the first of the proposed approaches, speaker diarization results are used for introducing a post-processing step to the STG construction algorithm, with the objective of discarding scene boundaries erroneously identified according to visual-only dissimilarity. In the second approach, speaker diarization and additional audio analysis results are employed and a separate audio-based STG is constructed, in parallel to the original STG based on visual information. The two STGs are subsequently combined. Preliminary results from the application of the proposed techniques to broadcast videos reveal their improved performance over previous approaches.

REFERENCES

• 1: R. Amaral, H. Meinedo, D. Caseiro, I. Trancoso, and J. Neto. A prototype system for selective dissemination of broadcast news in european portuguese. EURASIP Journal on Advances in Signal Processing, 2007, May 2007.
• 2: J. A. Bondy and U. Murty. Graph Theory with Applications. Macmillan Publishing Group, London, 1976.
• 3: S.-C. Chen, M.-L. Shyu, W. Liao, and C. Zhang. Scene change detection by audio and video clues. In Proc. IEEE ICME, pages 365--368, August 2002.
• 4: N. Goela, K. Wilson, F. Niu, and A. Divakaran. An svm framework for genre-independent scene change detection. In Proc. IEEE ICME, pages 532--535, July 2007.
• 5: A. Hanjalic and R. L. Lagendijk. Automated high-level movie segmentation for advanced video-retrieval systems. IEEE Trans. On Circuits and Systems for Video Technology, 9(4):580--588, June 1999.
• 6: W. Jinqiao, D. Lingyu, L. Qingshan, L. Hanqing, and J. Jin. A multimodal scheme for program segmentation and representation in broadcast video streams. IEEE Trans. on Multimedia, 10(3):393--408, April 2008.
• 7: H. Meinedo. PhD Thesis. IST, Technical University of Lisbon, Portugal, March 2008.
• 8: N. Nitanda, M. Haseyama, and H. Kitajima. Audio signal segmentation and classification for scene-cut detection. In Proc. IEEE Int. Symp. on Circuits and Systems, volume 4, pages 4030--4033, May 2005.
• 9: E. Tsamoura, V. Mezaris, and I. Kompatsiaris. Gradual transition detection using color coherence and other criteria in a video shot meta--segmentation framework. In Proc. IEEE ICIP-MIR 2008, pages 45--48, October 2008.
• 10: J. Vendrig and M. Worring. Systematic evaluation of logical story unit segmentation. IEEE Trans. on Multimedia, 4(4):492--499, December 2002.
• 11: M. Yeung and B.-L. Yeo. Segmentation of video by clustering and graph analysis. Computer Vision and Image Understanding, 71(1):94--109, July 1998.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms

Keywords:
scene segmentation