Deciphering visual gist and its implications for video retrieval and interface design

ABSTRACT

How do people make sense of a video based on viewing a few frames of that video? What elements constitute the "visual gist" in their minds? Answers to these questions will give implications to both content-based video retrieval and the interface design (e.g., key-frame selection) of digital video libraries. A preliminary study was conducted to unravel the issues and 45 subjects participated in the study. After viewing a fast forward surrogate, the subjects were asked to choose pictures which they thought would "belong to" the video. And they were also asked to think aloud during their selection processes. Nine visual gist attributes (e.g., people, objects and actions) were generated using the grounded theory method and their frequencies were also compared and analyzed.

REFERENCES

• 1: Wei Ding , Gary Marchionini , Dagobert Soergel, Multimodal surrogates for video browsing, Proceedings of the fourth ACM conference on Digital libraries, p.85-93, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313266]
• 2: Dufaux, F. Key frame selection to represent a video. ICIP 2000. Vol, II. p. 275--278
• 3: Grodal, T. Emotions, Cognitions, and Narrative Patterns in Film. In Passionate views : film, cognition, and emotion, edited by Plantinga, C. & Smith, G. M. 1999, 127--145
• 4: Corinne Jörgensen, Attributes of images in describing tasks, Information Processing and Management: an International Journal, v.34 n.2-3, p.161-174, 1998 [doi>10.1016/S0306-4573(97)00077-0]
• 5: Levin, D. T. & Simons, D. J. Failure to detect changes to attended objects in motion pictures. Psychological Bulletin, 4, 1997, 501--506
• 6: Lieberman, L. R., & Culpepper, J. T. Words versus objects: comparison of free verbal recall. Pscychol. Rep. 17,1965, 983--988
• 7: Mandler, J & Ritchey G. H. Long term memory for pictures. Journal of Experimental Psychology {Human learning and memory}, 3, 1977. 386--396
• 8: Markkula, M. and Sormumen, E. Searching for photos: journalists' practices in pictorial IR, The Challenge of Image Retrieval Research Workshop, 1998.
• 9: M. Massey , W. Bender, Salient stills: process and practice, IBM Systems Journal, v.35 n.3-4, p.557-573, 1996
• 10: Paivio, A. & Csapo, K. Picture superiority in free recall: Imagery or dual coding? Cognitive Psychology, 5, 1973, 176--206
• 11: Dulce Ponceleon , Savitha Srinivasan , Arnon Amir , Dragutin Petkovic , Dan Diklic, Key to effective video retrieval: effective cataloging and browsing, Proceedings of the sixth ACM international conference on Multimedia, p.99-107, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290760]
• 12: Shepard, R. N. Recognition memory for words, sentences, and pictures. Journal of Verbal Learning and Verbal Behavior, 6, 1967, 156--163
• 13: Simons, D. J. & Levin, D. T., Change blindness. Trends Cognitive Science, 1, 1997, 261--267
• 14: Barbara M. Wildemuth , Gary Marchionini , Todd Wilkens , Meng Yang , Gary Geisler , Beth Fowler , Anthony Hughes , Xiangming Mu, Alternative Surrogates for Video Objects in a Digital Library: Users' Perspectives on Their Relative Usability, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.493-507, September 16-18, 2002
• 15: Barbara M. Wildemuth , Gary Marchionini , Meng Yang , Gary Geisler , Todd Wilkens , Anthony Hughes , Richard Gruss, How fast is too fast?: evaluating fast forward surrogates for digital video, Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, May 27-31, 2003, Houston, Texas
• 16: Wolfe, J.M. Visual memory: what do you know about what you saw? Current Biology, 8(9), 1998, 303--304.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

Keywords:
fast forward, user studies, video retrieval, video surrogate, visual gist understanding