Exploring temporal consistency for video analysis and retrieval

ABSTRACT

Temporal consistency is ubiquitous in video data, where temporally adjacent video shots usually share similar visual and semantic content.This paper presents a thorough study of temporal consistency defined with respect to semantic concepts and query topics using quantitative measures,and discusses its implications to video analysis and retrieval tasks. We further show that,in interactive settings, using temporal consistency leads to considerable improvement on the performance of semantic concept detection and retrieval of video data.Speci fically,an active learning method with temporal sampling strategy is proposed for building classifiers of semantic concepts,and a temporal reranking method is proposed for improving the efficiency of interactive video search.Both methods outperform existing methods by considerable margins on the TRECVID dataset.

REFERENCES

• 1: LSCOM lexicon definitions and annotations version 1.0. In DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report 217-2006-3, 2006.
• 2: S. Chang, W. Chen, H. Horace, H. Sundaram, and D. Zhong. A fully automated content based video search engine supporting spatio-temporal queries. IEEE Trans. on Circuit System and Video Technology, 8(5):602--615, 1998.
• 3: Ming-yu Chen , Michael Christel , Alexander Hauptmann , Howard Wactlar, Putting active learning into multimedia applications: dynamic definition and refinement of concept classifiers, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101342]
• 4: S. Ebadollahi, L. Xie, S.-F. Chang, and J. Smith. Visual event detection using multi-dimensional concept dynamics. In Proc. IEEE Int'l Conf. on Multimedia and Expo (ICME 2006), 2006.
• 5: R. Khalaf and S. S. Intille. Improving multiple people tracking using temporal consistency. In MIT Dept.of Architecture House N Project Technical Report, 2001.
• 6: R. Lienhart. Comparison of automatic shot boundary detection algorithms. In SPIE Conf.on Storage and Retrieval for Image and Video Databases VII, volume 3656, pages 290--301, 1999.
• 7: M. R. Naphade, T. Kristjansson, B. Frey, and T. Huang. Probabilistic multimedia objects (multijects): A novel approach to video indexing and retrieval in multimedia systems. In Proc. of ICIP, 1998.
• 8: Apostol (Paul) Natsev , Milind R. Naphade , Jelena TešiĆ, Learning the semantics of multimedia queries and concepts from a small number of examples, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101288]
• 9: Yong Rui , Thomas S. Huang , Sharad Mehrotra, Constructing table-of-content for videos, Multimedia Systems, v.7 n.5, p.359-368, September 1999 [doi>10.1007/s005300050138]
• 10: A. Smeaton and P. Over. Trecvid: Benchmarking the effectiveness of infomration retrieval tasks on digital video. In Proc. of the Intl. Conf. on Image and Video Retrieval, 2003.
• 11: Xiaodan Song , Ching-Yung Lin , Ming-Ting Sun, Autonomous visual model building based on image crawling through internet search engines, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026762]
• 12: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 13: L. Xie, S.-F. Chang, A. Divakaran,and H. Sun. Structure analysis of soccer video with hidden markov models. In IEEE Int'l Conf. on Acoustic, Speech and Signal Processing, Orlando, FL, May 2002.
• 14: L. Xie, L. Kennedy, S.-F. Chang, A. Divakaran, H. Sun, and C.-Y. Lin. Layered dynamic mixture model for pattern discovery in asynchronous multi-modal streams. In Int'l Conf. on Acoustic, Speech and Signal Processing, Philadelphia, PA, March 2005.
• 15: Rong Yan , Jun Yang , Alexander G. Hauptmann, Learning query-class dependent weights in automatic video retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027661]
• 16: HongJiang Zhang , Shuang Yeo Tan , Stephen W. Smoliar , Gong Yihong, Automatic parsing and indexing of news video, Multimedia Systems, v.2 n.6, p.256-266, Jan. 1995 [doi>10.1007/BF01225243]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Experimentation, Performance

Keywords:
active learning, interactive search, semantic concept detection, temporal consistency, video retrieval