Effects of audio and visual surrogates for making sense of digital video

ABSTRACT

Video surrogates are meant to help people quickly make sense of the content of a video before downloading or seeking more detailed information. In this paper we present the results of a study comparing the effectiveness of three different surrogates for objects in digital video libraries. Thirty-six people participated in a within subjects user study in which they did five tasks for each of three surrogate alternatives: visual alone (a storyboard), audio alone (spoken description), and combined visual and audio (a storyboard augmented with spoken description). The results show that combined surrogates are more effective, strongly preferred, and do not penalize efficiency. The results also demonstrate that spoken descriptions alone lead to better understanding of the video segments than do visual storyboards alone, although people like to have visual surrogates and use them to confirm interpretations and add context. Participants were able to easily use the combined surrogates even though they were not synchronized, suggesting that synchronization of different media channels may not be necessary in surrogates as it is in full video. The results suggest that multimodal surrogates should be incorporated into video retrieval user interfaces and audio surrogates should be used in small display interfaces. The study also raises questions about the need to synchronize different information channels in multimedia surrogates.

REFERENCES

• 1: Michael G. Christel , Alexander G. Hauptmann , Howard D. Wactlar , Tobun D. Ng, Collages as dynamic summaries for news video, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641120]
• 2: Michael G. Christel , Michael A. Smith , C. Roy Taylor , David B. Winkler, Evolving video skims into useful multimedia abstractions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.171-178, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274670]
• 3: M. G. Christel , A. S. Warmack, The effect of text in storyboards for video navigation, Proceedings of the Acoustics, Speech, and Signal Processing, 2001. on IEEE International Conference, p.1409-1412, May 07-11, 2001 [doi>10.1109/ICASSP.2001.941193]
• 4: Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319--340.
• 5: Wei Ding , Gary Marchionini , Dagobert Soergel, Multimodal surrogates for video browsing, Proceedings of the fourth ACM conference on Digital libraries, p.85-93, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313266]
• 6: Christian Fluhr , Pierre-Alain Moéllic , Patrick Hede, Usage-oriented multimedia information retrieval technological evaluation, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178719]
• 7: Jawaid A. Ghani , Roberta Supnick , Pamela Rooney, The experience of flow in computer-mediated and in face-to-face groups, Proceedings of the twelfth international conference on Information systems, p.229-237, January 1991, New York, New York, United States
• 8: Goodrum, A. (1997). Evaluation of Text-Based and Image-Based Representations for Moving Image Documents. Unpublished doctoral dissertation, University of North Texas.
• 9: Gunther, R., Kazman, R, and MaccGregor, C. (2004) Using 3D sound as a navigational aid in virtual environments. Behaviour and Information Technology. 23(6), 435--446.
• 10: Francis C. Li , Anoop Gupta , Elizabeth Sanocki , Li-wei He , Yong Rui, Browsing digital video, Proceedings of the SIGCHI conference on Human factors in computing systems, p.169-176, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332425]
• 11: Ying Li , Chitra Dorai , Robert Farrell, Creating MAGIC: system for generating learning object metadata for instructional content, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101227]
• 12: Gary Marchionini , Barbara M. Wildemuth , Gary Geisler, The Open Video Digital Library: A Möbius strip of research and practice, Journal of the American Society for Information Science and Technology, v.57 n.12, p.1629-1643, October 2006 [doi>10.1002/asi.v57:12]
• 13: Mayer, R. (2003). Elements of a science of E-learning. Journal of Educational Computing Research, 29(3), 297--313.
• 14: Mayer, R., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology, 90(2), 312--320.
• 15: O'Conner, B. (1985). Access to moving image documents: Background concepts and proposals for surrogates for film and video works. Journal of Documentation, 41(4), 209--220.
• 16: Over, P., Kraaij, W., & Smeaton, A. (2005). TRECVID 2005: An introduction. Proc. TRECVID 2005 (Gaithersburg, MD), 1--14. http://www.cdvp.dcu.ie/Papers/TRECVid2005_Overview.pdf.
• 17: Pavio, A. (1986). Mental representations: A dual coding approach. Oxford: Oxford U. Press.
• 18: Daniel M. Russell , Mark J. Stefik , Peter Pirolli , Stuart K. Card, The cost structure of sensemaking, Proceedings of the INTERACT '93 and CHI '93 conference on Human factors in computing systems, p.269-276, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169209]
• 19: Turner, J. (1994). Determining the subject content of still and moving image documents for storage and retrieval: An experimental investigation. Unpublished doctoral dissertation, University of Toronto.
• 20: Barbara M. Wildemuth , Gary Marchionini , Todd Wilkens , Meng Yang , Gary Geisler , Beth Fowler , Anthony Hughes , Xiangming Mu, Alternative Surrogates for Video Objects in a Digital Library: Users' Perspectives on Their Relative Usability, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.493-507, September 16-18, 2002
• 21: Barbara M. Wildemuth , Gary Marchionini , Meng Yang , Gary Geisler , Todd Wilkens , Anthony Hughes , Richard Gruss, How fast is too fast?: evaluating fast forward surrogates for digital video, Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, May 27-31, 2003, Houston, Texas
• 22: Meng Yang , Gary Marchionini, Deciphering visual gist and its implications for video retrieval and interface design, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057045]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Design, Human Factors

Keywords:
dual coding, multimedia, video surrogates