Establishing the utility of non-text search for news video retrieval with real world users

ABSTRACT

TRECVID participants have enjoyed consistent success using storyboard interfaces for shot-based retrieval, as measured by TRECVID interactive search mean average precision (MAP). However, much is lost by only looking at MAP, and especially by neglecting to bring in representatives of the target user communities to conduct such tasks. This paper reports on the use of within-subjects experiments to reduce subject variability and emphasize the examination of specific video search interface features for their effectiveness in interactive retrieval and user satisfaction. A series of experiments is surveyed to illustrate the gradual realization of getting non-experts to utilize non-textual query features through interface adjustments. Notably, the paper explores the use of the search system by government intelligence analysts, concluding that a variety of search methods are useful for news video retrieval and lead to improved satisfaction. This community, dominated by text search system expertise but still new to video and image search, performed better with and favored a system with image and concept query capabilities over an exclusive text-search system. The user study also found that sports topics mean nothing for this user community and tens of relevant shots collected into the answer set are considered enough to satisfy the information need. Lessons learned from these user interactions are reported, with recommendations on both interface improvements for video retrieval systems and enhancing the ecological validity of video retrieval interface evaluations.

REFERENCES

• 1: Michael G. Christel , Ronald M. Conescu, Addressing the challenge of visual information access from digital image and video libraries, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065402]
• 2: Christel, M. G., and Conescu, R. M. Mining Novice User Activity with TRECVID Interactive Retrieval Tasks. In LNCS 4071: Proc. Image and Video Retrieval (CIVR) (Tempe, AZ, July 2006), Springer, Berlin, 2006, 21--30.
• 3: Michael Christel , Neema Moraveji, Finding the right shots: assessing usability and performance of a digital video library interface, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027691]
• 4: Andreas Girgensohn , John Adcock , Matthew Cooper , Lynn Wilcox, Interactive search in large video collections, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056925]
• 5: Alexander G. Hauptmann , Michael G. Christel, Successful approaches in the TREC video retrieval evaluations, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027681]
• 6: Hollink, L., Nguyen, G. P., Koelma, D. C., Schreiber, A. T., and Worring, M. Assessing User Behaviour in News Video Retrieval. IEE Proc. Vision, Image, & Signal Processing 152(6), 2005, 911--918.
• 7: Cameron Marlow , Mor Naaman , Danah Boyd , Marc Davis, HT06, tagging paper, taxonomy, Flickr, academic article, to read, Proceedings of the seventeenth conference on Hypertext and hypermedia, August 22-25, 2006, Odense, Denmark [doi>10.1145/1149941.1149949]
• 8: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 9: National Institute of Standards and Technology, NIST TREC Video Retrieval Evaluation Online Proceedings, 2001--2006, http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html.
• 10: Shneiderman, B., Byrd, D., and Croft, W. B. Clarifying Search: A User-Interface Framework for Text Searches. D-Lib Magazine, 3, 1 (January 1997), http://www.dlib.org.
• 11: Ben Shneiderman , Catherine Plaisant, Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies, Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization, May 23-23, 2006, Venice, Italy [doi>10.1145/1168149.1168158]
• 12: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 13: Snoek, C., Worring, M., Koelma, D., and Smeulders, A. Learned Lexicon-Driven Interactive Video Retrieval. In LNCS 4071: Proc. Image and Video Retrieval (CIVR) (Tempe, AZ, July 2006), Springer, Berlin, 2006, 11--20.
• 14: Snoek, C., Worring, M., Koelma, D., and Smeulders, A. A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval. IEEE Trans. Multimedia 9(2), Feb. 2007, 280--292.
• 15: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 16: Yan, R., and Hauptmann, A. G. Efficient Margin-Based Rank Learning Algorithms for Information Retrieval. In LNCS 4071: Proc. Image and Video Retrieval (CIVR) (Tempe, AZ, July 2006), Springer, Berlin, 2006, 113--122.
• 17: Meng Yang , Barbara M. Wildemuth , Gary Marchionini, The relative effectiveness of concept-based versus content-based video retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027613]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
      H.3.7 Digital Libraries
          Subjects: User issues

General Terms:
Experimentation, Human Factors

Keywords:
TRECVID analysis, interactive video retrieval, user studies