Fast unsupervised alignment of video and text for indexing/names and faces

ABSTRACT

We propose a novel way of combining weakly associated video/audio and text steams in an unsupervised manner which is faster than conventional speech recognition. The technique aligns audio/video and text streams which will enable video search using the associated text. Multimedia of this form includes news broadcast with summaries, parliament proceedings and court trials with transcripts, sports telecast with text commentary, etc. We also show how we can annotate the video with the names of the person appearing in the video which will allow name based indexing/search. We test the technique on a 80 minute video segment downloaded from the website of the International Court of the Former Yugoslavia, with the corresponding transcripts. The proposed technique achieves 88.49% accuracy on sentence level alignments and 95.5% accuracy on the task of assigning names to faces.

REFERENCES

• 1: Ognjen Arandjelovic , Andrew Zisserman, Automatic Face Recognition for Film Character Retrieval in Feature-Length Films, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.860-867, June 20-26, 2005 [doi>10.1109/CVPR.2005.81]
• 2: Kobus Barnard , Matthew Johnson , David Forsyth, Word sense disambiguation with pictures, Proceedings of the HLT-NAACL 2003 workshop on Learning word meaning from non-linguistic data, p.1-5, May 31, 2003 [doi>10.3115/1119212.1119213]
• 3: T. L. Berg, A. C. Berg, J. Edwards, M. Maire, R. White, Y. W. Teh, E. Learned-Miller, and D. A. Forsyth. Names and faces in the news. In computer Vision and Pattern Recognition, pages 848--854, 2004.
• 4: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800]
• 5: M. Everingham, J. Sivic, and A. Zisserman. Hello! my name is... buffy -- automatic naming of characters in tv video. In Proceedings of the British Machine Vision Conference, 2006.
• 6: The hidden markov model toolkit (htk), machine intelligence laboratory, cambridge university engineering department,http://htk.eng.cam.ac.uk/.
• 7: Michael Isard , Andrew Blake, C$ONDENSATION$—Conditional Density Propagation forVisual Tracking, International Journal of Computer Vision, v.29 n.1, p.5-28, Aug. 1998 [doi>10.1023/A:1008078328650]
• 8: Oded Maron , Tomás Lozano-Pérez, A framework for multiple-instance learning, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.570-576, July 1998, Denver, Colorado, United States
• 9: Oded Maron , Aparna Lakshmi Ratan, Multiple-Instance Learning for Natural Scene Classification, Proceedings of the Fifteenth International Conference on Machine Learning, p.341-349, July 24-27, 1998
• 10: K. Mikolajczyk, R. Choudhury, and C. Schmid. Face detection in a video sequence: A temporal approach. In CVPR01, pages II:96--101, 2001.
• 11: Mplayer - the movie player, http://www.mplayerhq.hu/design7/info.html.
• 12: Open computer vision library, http://sourceforge.net/projects/opencvlibrary/.
• 13: Sox - sound exchange, http://sox.sourceforge.net/.
• 14: Cmusphinx: The carnegie mellon sphinx project, http://cmusphinx.sourceforge.net/html/cmusphinx.php.
• 15: M. Turk and A. Pentland. Face recognition using eigenfaces. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, page 586âAŞ591, 1991.
• 16: A. Venkataraman, A. Stolcke, W. Wang, D. Vergyri, V. R. R. Gadde, and J. Zheng. An efficient repair procedure for quick transcriptions. In Proceedings of ICSLP, 2000.
• 17: Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
• 18: Ming-Hsuan Yang, Kernel Eigenfaces vs. Kernel Fisherfaces: Face Recognition Using Kernel Methods, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, p.215, May 20-21, 2002
• 19: Q. Zhang and S. Goldman. Em-dd: An improved multiple-instance learning technique, 2001.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.m Miscellaneous

General Terms:
Algorithms

Keywords:
indexing, multimedia alignment, names and faces