Question answering from lecture videos based on an automatic semantic annotation

ABSTRACT

The number of digital lecture video recordings has increased dramatically. The accessibility, usability and the traceability of their content for students-use is limited. Therefore retrieval of audiovisual lecture recordings is a complex task. Speech recognition is applied to create a tentative and deficient transcription of the video recordings. The imperfect transcription is sufficient to generate semantic metadata serialized in an OWL file. A question answering system based on the automatically generated semantic annotations and a semantic search engine are presented. The annotation process is discussed, evaluated and compared to a perfectly annotated OWL file and, further, to a corrected transcript of the lecture.

REFERENCES

• 1: Franz Baader , Diego Calvanese , Deborah L. McGuinness , Daniele Nardi , Peter F. Patel-Schneider, The description logic handbook: theory, implementation, and applications, Cambridge University Press, New York, NY, 2003
• 2: Ricardo A. Baeza-Yates , Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 3: Marco Bertini , Alberto Del Bimbo , Carlo Torniai , Rita Cucchiara , Costantino Grana, MOM: multimedia ontology manager. A framework for automatic annotation and semantic retrieval of video sequences, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180809]
• 4: M. Engelhardt, A. Hildebrand, D. Lange, and T. C. Schmidt. Reasoning about eLearning Multimedia Objects. In International Workshop on Semantic Web Annotations for Multimedia (SWAMM), 2006.
• 5: Alexander Haubold , John R. Kender, Augmented segmentation and visualization for presentation videos, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101158]
• 6: W. Hürst, T. Kreuzer, and M. Wiesenhütter. A qualitative study towards using large vocabulary automatic speech recognition to index recorded presentations for search and access over the web. In IADIS Internatinal Conference WWW/Internet(ICWI), pages 135--143, 2002.
• 7: A. Jaimes, T. Nagamine, J. Liu, K. Omura, and N. Sebe. Affective meeting video analysis. In IEEE Multimedia and Expo, pages 1412--1415, 2005.
• 8: S. Linckels and C. Meinel. Resolving ambiguities in the semantic interpretation of natural language questions. In Intelligent Data Engineering and Automated Learning (IDEAL), volume 4224 of LNCS, pages 612--619, 2006.
• 9: S. Linckels, H. Sack, and C. Meinel. Optimizing the retrieval of pertinent answers for nl questions with the e-librarian service. In Service Matchmaking and Resource Retrieval in the Semantic Web (SMR2), 2007.
• 10: R. Mertens, H. Schneider, O. Müller, and O. Vornberger. Hypermedia navigation concepts for lecture recordings. In E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, pages 2480--2847, 2004.
• 11: C.-W. Ngo, F. Wang, and T.-C. Pong. Structuring lecture videos for distance learning applications. In Multimedia Software Engineering, pages 215--222, 2003.
• 12: M. Porter. An algorithm for suffix stripping. Program, 14(3):130--137, 1980.
• 13: Stephan Repp , Serge Linckels , Christoph Meinel, Towards to an automatic semantic annotation for multimedia learning objects, Proceedings of the international workshop on Educational multimedia and multimedia education, September 28-28, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290144.1290149]
• 14: S. Repp, J. Waitelonis, H. Sack, and C. Meinel. Segmentation and annotation of audiovisual recordings based on automated speech recognition. In Intelligent Data Engineering and Automated Learning (IDEAL) 2007, Birmingham, UK, December 16-19, volume 4881 of LNCS, pages 620--629. Springer, 2007.
• 15: W. W. W. C. W3C. OWL Web Ontology Language. http://www.w3.org/TR/owl-features/, 2004.
• 16: Patrick Wolf , Wolfgang Putz , Avare Stewart , Arnd Steinmetz , Matthias Hemmje , Erich Neuhold, LectureLounge – experience education beyond the borders of the classroom, International Journal on Digital Libraries, v.4 n.1, p.39-41, August 2004 [doi>10.1007/s00799-003-0053-9]
• 17: N. Yamamoto, J. Ogata, and Y. Ariki. Topic segmentation and retrieval system for lecture videos based on spontaneous speech recognition. In European Conference on Speech Communication and Technology (EUROSPEECH), pages 961--964, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Algorithms, Experimentation, Management, Measurement

Keywords:
learning object, lecture video, multimedia retrieval, ontology, owl, semantic annotation, speech recognition