Multi-query interactive image and video retrieval -: theory and practice

ABSTRACT

We propose a new interactive image and video retrieval system called multi-query interactive retrieval, which is designed to jointly optimize the retrieval performance on multiple query topics. The proposed system employs a learning-based hybrid retrieval approach, which can automatically switch between tagging and browsing interface based on user labeling efficiency. To formalize the retrieval process, we use two formal annotation models to track and estimate the retrieval time for each method. Based on the parameters of these models, the system integrates the tagging-based and browsing-based methods in order to minimize overall retrieval time across the full set of query topics. This hybrid multi-topic retrieval approach is demonstrated to be highly effective on two large-scale video collections.

REFERENCES

• 1: Flickr. http://www.flickr.com.
• 2: J. Adcock, A. Girgensohn, M. Cooper, T. Liu, L. Wilcox, and E. Rieffel. Fxpal experiments for trecvid 2004. In NIST TRECVID, 2004.
• 3: M. Campbell, A. Haubold, M. Liu, A. P. Natsev, J. R. Smith, J. Tesic, L. Xie, , R. Yan, and J. Yang. IBM research TRECVID-2007 video retrieval system. In NIST TRECVID-2007, Nov 2007.
• 4: T.-S. Chua, S.-Y. Neo, H.-K. Goh, M. Zhao, Y. Xiao, and G. Wang. Trecvid 2005 by nus pris. In NIST TRECVID-2005, Nov 2005.
• 5: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 6: C. Halaschek-Wiener, J. Golbeck, A. Schain, M. Grove, B. Parsia, and J. Hendler. Photostuff - an image annotation tool for the semantic web. In Proc. of 4th international semantic web conference, 2005.
• 7: Alexander G. Hauptmann , Wei-Hao Lin , Rong Yan , Jun Yang , Ming-Yu Chen, Extreme video retrieval: joint maximization of human and computer performance, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180721]
• 8: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 9: P. Over, T. Ianeva, W. Kraaij, and A. F. Smeaton. Trecvid 2006 overview. In NIST TRECVID-2006, 2006.
• 10: J. Philbin, O. Chum, J. Sivic, V. Ferrari, M. Marin, A. Bosch, N. Apostolof, and A. Zisserman. Oxford trecvid 2007 notebook paper. In NIST TRECVID-2007, Nov 2007.
• 11: A. F. Smeaton , N. Murphy , N. E. O'Connor , S. Marlow , H. Lee , K. McDonald , P. Browne , J. Ye, The físchlár digital video system: a digital library of broadcast TV programmes, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.312-313, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379696]
• 12: C. Snoek, M. Worring, J. Geusebroek, D. Koelma, and F. Seinstra. The mediamill TRECVID 2004 semantic viedo search engine. In Proc. of TRECVID, 2004.
• 13: Timo Volkmer , John R. Smith , Apostol (Paul) Natsev, A web-based system for collaborative annotation of large image and video collections: an evaluation and user study, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101341]
• 14: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 15: Howard D. Wactlar , Michael G. Christel , Yihong Gong , Alexander G. Hauptmann, Lessons Learned from Building a Terabyte Digital Video Library, Computer, v.32 n.2, p.66-73, February 1999 [doi>10.1109/2.745722]
• 16: Rong Yan , Alexander G. Hauptmann, A review of text and image retrieval approaches for broadcast news video, Information Retrieval, v.10 n.4-5, p.445-484, October 2007 [doi>10.1007/s10791-007-9031-y]
• 17: R. Yan, A. Natsev, and M. Campbell. A learning-based hybrid tagging and browsing approach for e±cient manual image annotation. In Proceedings of the 2006 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
• 18: Yiming Yang , Jan O. Pedersen, A Comparative Study on Feature Selection in Text Categorization, Proceedings of the Fourteenth International Conference on Machine Learning, p.412-420, July 08-12, 1997
• 19: J. Yuan, L. Xiao, D. Wang, D. Ding, Y. Zuo, Z. Tong, X. Liu, S. Xu, W. Zheng, X. Li, Z. Si, J. Li, F. Lin, and B. Zhang. Tsinghua university at TRECVID 2005. In NIST TRECVID 2005, Nov 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Performance, Theory

Keywords:
hybrid annotation, interactive image and video retrieval, multi-query retrieval