Latent semantic analysis for an effective region-based video shot retrieval system

ABSTRACT

We present a complete and efficient framework for video shot indexing and retrieval. Video shots are described by their key-frame, themselves described by their regions. Region-based approaches suffer from the complexity of segmentation and comparison tasks. A compact region-based shot representation is usually obtained thanks to vector-quantization method. We thus introduce LSA to reduce the noise inherent to the segmentation and the quantization processes. Then to better capture the content of video shots, we propose two original methods. The first takes advantage of a multi-scale segmentation of frames while the second uses multiple frames to represent a shot. Both approaches require more computation time during the pre-processing but not for indexing and comparison tasks. Indeed the extra information is included in the original signatures of shots. Finally we introduce a relevance feedback loop to optimize the search and propose a new method to optimize the effect of LSA. In the experimental section, we make an evaluation of latent semantic analysis and proposed approaches on two problems, namely object retrieval and semantic content estimation

REFERENCES

• 1: Shih-Fu Chang, W. Chen, H.J. Meng, H. Sundaram, and Di Zhong. A fully automated content-based video search engine supporting spatiotemporal queries. In IEEE Transactions on Circuits and Systems for Video Technology, volume 8, pages 602-- 615, 1998.
• 2: M.R. Naphade, T. Kristjansson, B. Frey, and T.S. Huang. Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval. In IEEE International Conference on Image Processing, volume 3, pages 536--540, 1998.
• 3: Howard D. Wactlar , Takeo Kanade , Michael A. Smith , Scott M. Stevens, Intelligent Access to Digital Video: Informedia Project, Computer, v.29 n.5, p.46-52, May 1996 [doi>10.1109/2.493456]
• 4: Edoardo Ardizzone , Marco La Cascia, Automatic Video Database Indexing and Retrieval, Multimedia Tools and Applications, v.4 n.1, p.29-56, January 1997 [doi>10.1023/A:1009630331620]
• 5: Chad Carson , Megan Thomas , Serge Belongie , Joseph M. Hellerstein , Jitendra Malik, Blobworld: A System for Region-Based Image Indexing and Retrieval, Proceedings of the Third International Conference on Visual Information and Information Systems, p.509-516, June 02-04, 1999
• 6: Feng Jing , Mingjing Li , Hong-Jiang Zhang , Bo Zhang, An effective region-based image retrieval framework, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641106]
• 7: Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391--407, 1990.
• 8: Mikko Kurimo. Indexing audio documents by using latent semantic analysis and som. In Erkki Oja and Samuel Kaski, editors, Kohonen Maps, pages 363--374. Elsevier, 1999.
• 9: Rong Zhao and William I Grosky. From features to semantics: Some preliminary results. In International Conference on Multimedia and Expo, 2000.
• 10: Joo-Hwee Lim, Learning Visual Keywords for Content-Based Retrieval, Proceedings of the IEEE International Conference on Multimedia Computing and Systems Volume II-Volume 2, p.169, June 07-11, 1999
• 11: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 12: P. F. Felzenszwalb , D. P. Huttenlocher, Image Segmentation Using Local Variation, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.98, June 23-25, 1998
• 13: Wei-Ying Ma and Hong Jiang Zhang. Benchmarking of image features for content-based image retrieval. In Thirty-second Asilomar Conference on Signals, System and Computers, volume 1, pages 253--257, 1998.
• 14: Fabrice Souvannavong, Bernard Merialdo, and Benoît Huet. Latent semantic analysis for semantic content detection of video shots. In International Conference on Multimedia and Expo, 2004.
• 15: Fabrice Souvannavong, Bernard Merialdo, and Benoît Huet. Video content modeling with latent semantic analysis. In Third International Workshop on Content-Based Multimedia Indexing, 2003.
• 16: M. Mirmehdi and R. Perissamy. Perceptual image indexing and retrieval. Journal of Visual Communication and Image Representation, 13(4):460--475, December 2002.
• 17: Charles E. Jacobs , Adam Finkelstein , David H. Salesin, Fast multiresolution image querying, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.277-286, September 1995 [doi>10.1145/218380.218454]
• 18: Fabrice Moscheni , Sushil Bhattacharjee , Murat Kunt, Spatiotemporal Segmentation Based on Region Merging, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.9, p.897-915, September 1998 [doi>10.1109/34.713358]
• 19: Daniel DeMenthon. Spatio-temporal segmentation of video by hierarchical mean shift analysis. In Workshop on Statistical Methods in Video Processing, 2002.
• 20: J. Rocchio. Relevance feedback in information retrieval. In G. Salton, editor, The SMART Retrieval System: Experiments in Automatic Document Processing, pages 313--323. Prentice-Hall, 1971.
• 21: Ching-Yung Lin, Belle L. Tseng, and John R. Smith. Video collaborative annotation forum: Establishing ground-truth labels on large multimedia datasets. In Proceedings of the TRECVID 2003 Workshop, 2003.
• 22: Fabrice Souvannavong, Bernard Merialdo, and Benoit Huet. Latent semantic indexing for video content modeling and analysis. In The 12th Text REtrieval Conference (TREC), 2003.
• 23: Kenji Kira , Larry A. Rendell, A Practical Approach to Feature Selection, Proceedings of the Ninth International Workshop on Machine Learning, p.249-256, July 01-03, 1992

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Algorithms, Design

Keywords:
latent semantic analysis, region clustering, region similarity, region-based video retrieval, video analysis