Exploring knowledge of sub-domain in a multi-resolution bootstrapping framework for concept detection in news video

ABSTRACT

In this paper, we present a model based on a multi-resolution, multi-source and multi-modal (M3) bootstrapping framework that exploits knowledge of sub-domains for concept detection in news video. Because the characteristics and distributions of data in different sub-domains are different, we model and analyze the video in each sub-domain separately using a transductive framework. Along with this framework, we propose a "pseudo-Vapnik combined error bound" to tackle the problem of imbalanced distribution of training data in certain segments of sub-domains. For effective fusion of multi-modal features, we utilize multi-resolution inference and constraints to permit evidences from different modal features to support each other. Finally, we employ a bootstrapping technique to leverage unlabeled data to boost the overall system performance. We test our framework by detecting semantic concepts in the TRECVID 2004 dataset. Experimental results demonstrate that our approach is effective.

REFERENCES

• 1: A. Amir et al, "IBM research TRECVID 2005 video retrieval system", Proceedings of TRECVID 2005, Gaithersburg, MD, November 2005 available at: http://www-nlpir.nist.gov/projects/tvpubs/tv5.papers/
• 2: L. Chaisorn, "A Hierarchical Multi-Modal approach to story segmentation in news video", PhD thesis in National University of Singapore, 2004
• 3: S. F. Chang, R. Manmatha, and T. S. Chua, "Combining text and audio-visual features in video indexing", Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 1005--1008, 2005
• 4: Tat-Seng Chua , Shih-Fu Chang , Lekha Chaisorn , Winston Hsu, Story boundary detection in large broadcast news video archives: techniques, experience and trends, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027679]
• 5: T. S. Chua et al, "TRECVID 2004 Search and Feature Extraction Task by NUS PRIS" Proceedings of (VIDEO) TRECVID 2004, Gaithersburg, MD, November 2004, available at : http://www-nlpir.nist.gov/projects/tvpubs/
• 6: T. S. Chua et al, "TRECVID 2005 by NUS PRIS", Proceeding of TRECVID 2005, Gaithersburg, MD, November 2005, available at http://www-nlpir.nist.gov/projects/tvpubs/
• 7: Huamin Feng , Rui Shi , Tat-Seng Chua, A bootstrapping framework for annotating and retrieving WWW images, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027748]
• 8: Jean-Luc Gauvain , Lori Lamel , Gilles Adda, The LIMSI Broadcast News transcription system, Speech Communication, v.37 n.1-2, p.89-108, May 2002 [doi>10.1016/S0167-6393(01)00061-9]
• 9: A. Hauptmann et al, "Multi-Lingual Broadcast News Retrieval" Proceedings of TRECVID 2006 available at: http://www-nlpir.nist.gov/projects/tvpubs/
• 10: David Hull, Using statistical testing in the evaluation of retrieval experiments, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.329-338, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160758]
• 11: A. K. Jain , M. N. Murty , P. J. Flynn, Data clustering: a review, ACM Computing Surveys (CSUR), v.31 n.3, p.264-323, Sept. 1999 [doi>10.1145/331499.331504]
• 12: Daniel Jurafsky , James H. Martin, Speech and Language Processing (2nd Edition), Prentice-Hall, Inc., Upper Saddle River, NJ, 2006
• 13: M. Lan, C. L. Tan and H. B. Low "Proposing a new term weighting scheme for text categorization", Proceeding of the 21st National Conference on Artificial Intelligence, AAAI-2006
• 14: Chin-Yew Lin, Robust automated topic identification, University of Southern California, Los Angeles, CA, 1997
• 15: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 16: G. J. Qi, X. S. Hua, Y. Song, J. H. Tang, H. J. Zhang, "Transductive Inference with Hierarchical Clustering for Video Annotation" International Conference on Multimedia and Expo, pp.643--646, 2007
• 17: Lawrence A. Rowe , Ramesh Jain, ACM SIGMM retreat report on future directions in multimedia research, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.1 n.1, p.3-13, February 2005 [doi>10.1145/1047936.1047938]
• 18: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 19: Q. Tian, J. Yu, Q. Xue, and N. Sebe, "A New Analysis of the Value of Unlabeled Data in Semi-Supervised Learning for Image Retrieval", Proceedings of IEEE International Conference on Multimedia and Expo (ICME 2004), Vol.2, pp.1019--1022, 2004.
• 20: V. N. Vapnik, "Statistical learning theory", Wiley Interscience New York. pp.120--200, 1998,
• 21: Rong Yan , Milind Naphade, Semi-Supervised Cross Feature Learning for Semantic Concept Detection in Videos, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.657-663, June 20-26, 2005 [doi>10.1109/CVPR.2005.317]
• 22: Rong Yan , Jun Yang , Alexander G. Hauptmann, Learning query-class dependent weights in automatic video retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027661]
• 23: J. Yang, A. Hauptmann, M. Y. Chen, "Finding Person X: Correlating Names with Visual Appearances", International Conference on Image and Video Retrieval (CIVR'04), Dublin City University, Ireland, July 21--23, 2004
• 24: Jun Yang , Rong Yan , Alexander G. Hauptmann, Cross-domain video concept detection using adaptive svms, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291276]
• 25: R. E. Yaniv, and L. Gerzon, "Effective Transductive Learning via PAC-Bayesian Model Selection." Technical Report CS-2004-05, IIT, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms, Experimentation

Keywords:
bootstrapping, domain knowledge, multi-resolution analysis, text semantics, transductive learning, unlabeled data