Detecting topical events in digital video

ABSTRACT

The detection of events is essential to high-level semantic querying of video databases. It is also a very challenging problem requiring the detection and integration of evidence for an event available in multiple information modalities, such as audio, video and language. This paper focuses on the detection of specific types of events, namely, topic of discussion events that occur in classroom/lecture environments. Specifically, we present a query-driven approach to the detection of topic of discussion events with foils used in a lecture as a way to convey a topic. In particular, we use the image content of foils to detect visual events in which the foil is displayed and captured in the video stream. The recognition of a foil in video frames exploits the color and spatial layout of regions on foils using a technique called region hashing. Next, we use the textual phrases listed on a foil as an indication of a topic, and detect topical audio events as places in the audio track where the best evidence for the topical phrases was heard. Finally, we use a probabilistic model of event likelihood to combine the results of visual and audio avent detection that exploits their time cooccurrence. The resulting identification of topical events is evaluated in the domain of classroom lectures and talks.

REFERENCES

• 1: Gregory D. Abowd , Christopher G. Atkeson , Ami Feinstein , Cindy Hmelo , Rob Kooper , Sue Long , Nitin Sawhney , Mikiya Tani, Teaching and learning as multimedia authoring: the classroom 2000 project, Proceedings of the fourth ACM international conference on Multimedia, p.187-198, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244191]
• 2: CORPORATE The SRI MAESTRO Team, MAESTRO: conductor of multimedia analysis technologies, Communications of the ACM, v.43 n.2, p.57-63, Feb. 2000 [doi>10.1145/328236.328147]
• 3: K. Bharat and M. Henzinger. Improved algorithms for topic distillation in a hypexlinked environment. In Proc. 22nd Annual SIGIR Conference, pages 326--327, 1999.
• 4: J.C. Clark and N. Ferrier. Modal control of an attentive vision system. In Proceedings of the International Conference on Computer Vision, pages 514-523. 1988.
• 5: G. Hauptmann, D. Lee, and P.E. Kennedy. Topic labeling of multilingual broadcast news in the informedia digital video library. In Proc. A CM Digital Libraries/SIGIR MIDAS Workshop, 1999.
• 6: Steve Jones , Gordon Paynter, Topic-based browsing within a digital library using keyphrases, Proceedings of the fourth ACM conference on Digital libraries, p.114-121, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313279]
• 7: Christof Koch , Shimon Ullman, Selecting One Among the Many: A Simple Network Implementing Shifts in Selective Visual Attention, Massachusetts Institute of Technology, Cambridge, MA, 1984
• 8: Y. Lamdan and H.J. Wolfish. Geometric hashing: A general and efficient model-based recognition scheme. In Proceedings of the International Conference on Computer Vision, pages 218-249, 1988.
• 9: Sugata Mukhopadhyay , Brian Smith, Passive capture and structuring of lectures, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.477-487, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319690]
• 10: Wayne Niblack, SlideFinder: A Tool for Browsing Presentation Graphics Using Content-Based Retrieval, Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, p.114, June 22-22, 1999
• 11: R. Schwartz et al. A maximum likelihood model for topic classification in broadcast news. In Proc. European Conf. on Speech Communication and Technology, 1997.
• 12: Jeffrey Mark Siskind , Quaid Morris, A Maximum-Likelihood Approach to Visual Event Classification, Proceedings of the 4th European Conference on Computer Vision-Volume II, p.347-360, April 15-18, 1996
• 13: Savitha Srinivasan , Dulce Ponceleon , Dragutin Petkovic , Mahesh Viswanathan, Query Expansion for Imperfect Speech: Applications in Distributed Learning, Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'00), p.50, June 16-16, 2000
• 14: Savitha Srinivasan , Dragutin Petkovic, Phonetic confusion matrix based spoken document retrieval, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.81-87, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345552]
• 15: Savitha Srinivasan , Dragutin Petkovic , Dulce Ponceleon, Towards robust features for classifying audio in the CueVideo system, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.393-400, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319658]
• 16: T. Syeda-Mahmood. Indexing of topics using foils. In IEEE Conf. on Computer Vision and Pattern Recognition, 2000.
• 17: T. F. Syeda-Mahmood , P. Raghavan , N. Megiddo, Interval Hash Tree: An Efficient Index Structure for Searching Object Queries in Large Image Databases, Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'00), p.78, June 16-16, 2000
• 18: T. F. Syeda-Mahmood , Y-Q. Cheng, Indexing Colored Surfaces in Images, Proceedings of the International Conference on Pattern Recognition (ICPR '96) Volume III-Volume 7276, p.8, August 25-29, 1996

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION

General Terms:
Design, Management, Measurement, Performance, Theory

Keywords:
multi-modal fusion, query-driven topic detection, slide detection, topic of discussion events, topical audio events