Semantic context detection based on hierarchical audio models

ABSTRACT

Semantic context detection is one of the key techniques to facilitate efficient multimedia retrieval. Semantic context is a scene that completely represents a meaningful information segment to human beings. In this paper, we propose a novel hierarchical approach that models the statistical characteristics of several audio events, over a time series, to accomplish semantic context detection. The approach consists of two stages: audio event and semantic context detections. HMMs are used to model basic audio events, and event detection is performed in the first stage. Then semantic context detection is achieved based on Gaussian mixture models, which model the correlations among several audio events temporally. With this framework, we bridge the gaps between low-level features and the semantic contexts that last in a time series. The experimental evaluations indicate that the approach is effective in detecting high-level semantics.

REFERENCES

• 1: Zhang, T., and Kuo, C.-C. J. Hierarchical System for Content-based Audio Classification and Retrieval, Proceedings of SPIE, Multimedia Storage and Archiving Systems III, Vol. 3527, 398--409, 1998.
• 2: Cai, R., Lu, L., Zhang, H.-J., and Cai, L.-H. Highlight Sound Effects Detection in Audio Stream. Proceedings of IEEE International Conference on Multimedia & Expo, 2003.
• 3: Lu, L., Zhang, H.-J., and Jiang, H. Content Analysis for Audio Classification and Segmentation. IEEE Transactions on Speech and Audio Processing, Vol. 10, No. 7, 504--516, 2002.
• 4: Naphade, M. R., and Huang, T. S. Extracting Semantics from Aduiovisual Content: The Final Frontier in Multimedia Retrieval. IEEE Transactions on Neural Network, Vol. 13, No. 4, 793--810, 2002.
• 5: Naphade, M. R., and Huang, T. S. A Probabilistic Framework for Semantic Video Indexing, Filtering, and Retrieval. IEEE Transactions on Multimedia, Vol. 3, No. 1, 141--151, 2001.
• 6: Iyengar, G., Nock, H., Neti, C., and Franz, M. Semantic Indexing of Multimedia Using Audio, Text, and Visual Cues. Proceedings of ICME, 369--372, 2002.
• 7: Yu-Fei Ma , Lie Lu , Hong-Jiang Zhang , Mingjing Li, A user attention model for video summarization, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641116]
• 8: Ho, C.-C. A Study of Effective Techniques for User-Centric Video Streaming. Ph.D. dissertation, National Taiwan University, May, 2003.
• 9: Wang, Y., Liu, Z., and Huang, J.-C. Multimedia Content Analysis. IEEE Signal Processing Magazine, Nov. 2000, 12--36, 2000.
• 10: Rabiner, L. R. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proceedings of The IEEE, Vol. 77, No. 2, 257--286, 1989.
• 11: Naphade, M., Mehrotra, R., Ferman, A. M., Warnick, J., Huang, T. S., and Tekalp, A. M. A High Performance Shot Boundary Detection Algorithm Using Multiple Cues. Proceedings of IEEE Int. Conf. Image Processing, Vol. 2, 884--887, 1998.
• 12: Di Zhong , Shih-Fu Chang, Spatio-temporal video search using the object based video representation, Proceedings of the 1997 International Conference on Image Processing (ICIP '97) 3-Volume Set-Volume 1, p.21, October 26-29, 1997
• 13: Lie Lu , Hao Jiang , HongJiang Zhang, A robust audio classification and segmentation method, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500173]
• 14: Naphade, M. R., Kristjansson, T., Frey, B., and Huang, T. S. Probabilistic Multimedia Objects (Multijects): A Novel Approach to Video Indexing and Retrieval in Multimedia System. IEEE Int. Conf. Image Processing, Vol. 3, 536--540, 1998.
• 15: Kschischang, F. R., and Frey, B. J. Factor Graphs and the Sum-Product Algorithm. IEEE Transactions on Information Theory, Vol. 47, No. 2, 498--519, 2001.
• 16: Todd K. Moon, Error Correction Coding: Mathematical Methods and Algorithms, Wiley-Interscience, 2005

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Management

Keywords:
Gaussian mixture model, audio content analysis, audio retrieval, hidden Markov model, semantic context