Automatic discovery of salient segments in imperfect speech transcripts

ABSTRACT

This paper addresses the problem of automatic detection of salient video segments for real-world applications such as corporate training based on associated speech transcriptions. We present a novel segmentation algorithm based on automatic speech recognition (ASR) applied to the audio track of the video. Our feature set consists of word n-grams extracted from the imperfect speech transcriptions. We use a two-pass algorithm that combines a boundary-based method with a content-based method. In the first pass, we analyze the temporal distribution and the rate of arrival of features to compute an initial segmentation. In the second pass, we detect changes in content-bearing words by using the content-bearing features as queries in an information retrieval system. The content-based second pass validates the initial segments and merges them as needed. Variations in the structure of the audio/video content, and the accuracy of ASR have an impact on the feasibility of the segmentation task. For realistic data we observe that we can identify content-rich segments of the audio. In the best scenario a high-level table-of-contents is generated and in the worse scenario a single salient segment is identified. We illustrate the algorithm in detail with some examples and validate the data with manual segmentation boundaries.

REFERENCES

• 1: Allan, J., et al., Topic Detection and Tracking Pilot Study Final Report, Proc. of the DARPA Broadcast News Transcription and Understanding Workshop, February 1998.
• 2: Bach, J.R., et al., Virage image search engine: An open framework for image management, Proc. of SPIE Storage and Retrieval for Still Images and Video Databases IV, Vol. 2670, IS&T/SPIE, February 1996. http://www.virage.com
• 3: Krishna Bharat , Monika R. Henzinger, Improved algorithms for topic distillation in a hyperlinked environment, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.104-111, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290972]
• 4: Eichmann, D., et al., A cluster-based approach to tracking, detection and segmentation of Broadcast News, TDT Evaluation, NIST's 1999.
• 5: Fiscus, J.G., et al., TDT Evaluation, NIST's 1998.
• 6: A. G. Hauptmann, Speech recognition in the Informedia Digital Video Library: uses and limitations, Proceedings of the Seventh International Conference on Tools with Artificial Intelligence, p.288, November 05-08, 1995
• 7: Alexander G. Hauptmann , Danny Lee, Topic labeling of broadcast news stories in the informedia digital video library, Proceedings of the third ACM conference on Digital libraries, p.287-288, June 23-26, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276675.276733]
• 8: Marti A. Hearst, Multi-paragraph segmentation of expository text, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.9-16, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981734]
• 9: Johnson, et al., Spoken Document Retrieval for TREC-7 at Cambridge University, Proc. of the 7th Text Retrieval Conference (TREC-7), 1998.
• 10: Justeson, J.S. and Slava K., Technical Terminology: Some Linguistic Properties and an Algorithm for Identification in Text, in Natural Language Engineering, 1, pp 9-27, 1995.
• 11: Loach, P.D. and Wathen, A.J., On best least-squares approximation of continuous functions using linear splines with free knots, IMA J. Numerical Analysis, 11, pp. 393-409, 1991.
• 12: Schwartz, R., et. al., A Maximum Likelihood Model for Topic Classification in Broadcast News, Eurospeech, Fifth European Conf. on Speech Communication and Technology, September 1997.
• 13: Savitha Srinivasan , Dragutin Petkovic, Phonetic confusion matrix based spoken document retrieval, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.81-87, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345552]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding

General Terms:
Algorithms, Design, Languages