Evaluation of video browser features and user interaction with VAST MM

ABSTRACT

In this paper, we present extensive user studies on browsing and information retrieval in the domain of unstructured videos using the VAST MM video library browser. Our studies were performed over a 3-year period with more than 1,000 participants in the university setting. The majority of students use the video library for retrieval of student presentations in a large engineering design course. Through iterative analysis of context-specific audio, visual, and textual cues, we are able to measure significant improvements on typical retrieval tasks, such as searching for unfamiliar content in a large database with over 300 hours of video. We also present user studies conducted in two videotaped core computer science courses to measure the usefulness of the VAST MM (Video Audio Structure Text MultiMedia) resource for final exam preparation. We find that students who use the lecture video library experience significant improvement in final exam scores.

To better compare video browsers featuring rich content cues to standard video players without cues, we have performed a large experiment to collect measurable data on search tasks. In general, the lack of index cues can be described by an inverse relationship between amount of matching video content and time required to find it. When index cues are available, the relationship is constant, that is, rare content is found in the same time as common content. We evaluate this data and provide additional insight into two common user interaction techniques: audio-visual browsing and visual-only browsing. We show that user preference is uniform, but that audio-visual browsing is significantly more effective for search and retrieval of video data.

REFERENCES

• 1: Sugata Mukhopadhyay , Brian Smith, Passive capture and structuring of lectures, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.477-487, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319690]
• 2: Gregory D. Abowd , Christopher G. Atkeson , Ami Feinstein , Cindy Hmelo , Rob Kooper , Sue Long , Nitin Sawhney , Mikiya Tani, Teaching and learning as multimedia authoring: the classroom 2000 project, Proceedings of the fourth ACM international conference on Multimedia, p.187-198, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244191]
• 3: A. Haubold , J. R. Kender, Analysis and interface for instructional video, Proceedings of the 2003 International Conference on Multimedia and Expo, p.705-708, July 06-09, 2003
• 4: Ming Lin , Jay F. Nunamaker Jr. , Michael Chau , Hsinchun Chen, Segmentation of Lecture Videos Based on Text: A Method Combining Multiple Linguistic Features, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 1, p.10003, January 05-08, 2004
• 5: Alexander Haubold , John R. Kender, Analysis and Visualization of Index Words from Audio Transcripts of Instructional Videos, Proceedings of the IEEE Sixth International Symposium on Multimedia Software Engineering, p.570-573, December 13-15, 2004
• 6: Alexander Haubold , John R. Kender, VAST MM: multimedia browser for presentation video, Proceedings of the 6th ACM international conference on Image and video retrieval, p.41-48, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282286]
• 7: Michael Christel , David Martin, Information Visualization Within a Digital Video Library, Journal of Intelligent Information Systems, v.11 n.3, p.235-257, Nov./Dec. 1998 [doi>10.1023/A:1008638024786]
• 8: Lee H. and Smeaton, A. F. Designing the User Interface for the Físchlár Digital Video Library. In Journal of Digital Information, Volume 2, Issue 4 (May 2002).
• 9: Alexander Haubold, Selection and ranking of text from highly imperfect transcripts for retrieval of video content, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277911]
• 10: Yu, D., Cheung, S. H., Legge, G. E., and Chung S. T. Effect of letter spacing on visual span and reading speed. In Journal of Vision, Vol. 7, Nr. 2, Article 2 (Feb. 2007). 1--10.
• 11: Thorpe, S., Fize, D., and Marlot C. Speed of processing in the human visual system. In Nature, Volume 381, (6 Jun. 1996). 520--522.
• 12: Walpole, R. E., Myers, R. H., Myers, S. L., Ye, K. Probability and Statistics for Engineers and Scientists, 7th Edition, 2002. Prentice Hall, New Jersey.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Dissemination
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

General Terms:
Design, Experimentation, Human Factors, Measurement, Verification

Keywords:
automatic speech recognition, evaluation, measures, presentation video, speaker index, speaker segmentation, streaming video, structure in videos, text augmentation, transcript analysis, user studies, video library, visual segmentation