Experiments in interactive video search by addition and subtraction

ABSTRACT

We have developed an interactive video search system that allows the searcher to rapidly assess query results and easily pivot off those results to form new queries. The system is intended to maximize the use of the discriminative power of the human searcher. This is accomplished by providing a hierarchical segmentation, streamlined interface, and redundant visual cues throughout. The typical search scenario includes a single searcher with the ability to search with text and content-based queries. In this paper, we evaluate new variations on our basic search system. In particular we test the system using only visual content-based search capabilities, and using paired searchers in a realtime collaboration. We present analysis and conclusions from these experiments.

REFERENCES

• 1: J. Adcock, A. Girgensohn, M. Cooper, T. Liu, L. Wilcox, and E. Rieffel. Fxpal experiments for trecvid 2004. In Proceedings of the TREC Video Retrieval Evaluation (TRECVID), pages 70--81, Washington D.C., 2004. NIST.
• 2: J. Adcock, A. Girgensohn, M. Cooper, and L. Wilcox. Interactive video search using multilevel indexing. In International Conference on Image and Video Retrieval, pages 205--214, 2005.
• 3: Javed A. Aslam , Mark Montague, Models for metasearch, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.276-284, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384007]
• 4: H. Bay, T. Tuytelaars, and L. V. Gool. Surf: Speeded up robust features. In European Conference on Computer Vision, 2006.
• 5: Michael W. Berry , Susan T. Dumais , Gavin W. O'Brien, Using linear algebra for intelligent information retrieval, SIAM Review, v.37 n.4, p.573-595, Dec. 1995 [doi>10.1137/1037127]
• 6: L. Bottou and Y. Bengio. Convergence properties of the K-means algorithms. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 585--592. The MIT Press, 1995.
• 7: C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
• 8: Shih-Fu Chang , Dan Ellis , Wei Jiang , Keansub Lee , Akira Yanagawa , Alexander C. Loui , Jiebo Luo, Large-scale multimodal semantic concept detection for consumer video, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290118]
• 9: Michael G. Christel , Rong Yan, Merging storyboard strategies and automatic retrieval for improving interactive video search, Proceedings of the 6th ACM international conference on Image and video retrieval, p.486-493, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282351]
• 10: M. Cooper, J. Adcock, and F. Chen. Fxpal at trecvid 2006. In Proceedings of the TREC Video Retrieval Evaluation (TRECVID), 2006.
• 11: M. Cooper, J. Adcock, H. Zhou, and R. Chen. Fxpal at trecvid 2005. In Proceedings of the TREC Video Retrieval Evaluation (TRECVID), 2005.
• 12: Phillip I. Good, Permutation, Parametric, and Bootstrap Tests of Hypotheses (Springer Series in Statistics), Springer-Verlag New York, Inc., Secaucus, NJ, 2004
• 13: Alexander G. Hauptmann , Wei-Hao Lin , Rong Yan , Jun Yang , Ming-Yu Chen, Extreme video retrieval: joint maximization of human and computer performance, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180721]
• 14: Jing Huang , S. Ravi Kumar , Mandar Mitra , Wei-Jing Zhu , Ramin Zabih, Image Indexing Using Color Correlograms, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.762, June 17-19, 1997
• 15: M. A. H. Huijbregts, R. J. F. Ordelman, and F. M. G. de Jong. Annotation of heterogeneous multimedia content using automatic speech recognition. In Proceedings of the Second International Conference on Semantic and Digital Media Technologies, SAMT 2007, Genoa, Italy, volume 4816 of Lecture Notes in Computer Science, pages 78--90. Springer Verlag, 2007.
• 16: Xirong Li , Dong Wang , Jianmin Li , Bo Zhang, Video search in concept subspace: a text-like paradigm, Proceedings of the 6th ACM international conference on Image and video retrieval, p.603-610, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282366]
• 17: H.-B. Luan, S.-X. Lin, S. Tang, S.-Y. Neo, and T.-S. Chua. Interactive spatio-temporal visual map model for web video retrieval. In Proc. IEEE International Conference on Multimedia and Expo, pages 560--563, 2-5 July 2007.
• 18: Lucene. Jakarta lucene. http://jakarta.apache.org/lucene/docs/index.html.
• 19: M. R. Morris. Interfaces for collaborative exploratory web search: Motivations and directions for multi-user designs. In CHI 2007 Workshop on Exploratory Search and HCI, 2007.
• 20: M. R. Naphade, L. Kennedy, J. R. Kender, S.-F. Chang, J. Smith, P.Over, and A.Hauptmann. A light scale concept ontology for multimedia understanding for trecvid 2005. Computer Science Technical Report RC23612 W0505-104, IBM, 2005.
• 21: P. Over, G. Awad, W. Kraiij, and A. Smeaton. Trecvid 2007 an overview. In Proceedings of the TRECVID 2007 Workshop, Nov. 2007.
• 22: C. Petersohn. Fraunhofer hhi at trecvid 2004: Shot boundary detection system. In Proceedings of the TREC Video Retrieval Evaluation (TRECVID), pages 64--69, Washington D.C., 2004. NIST.
• 23: Alan F. Smeaton , Paul Browne, A usage study of retrieval modalities for video shot retrieval, Information Processing and Management: an International Journal, v.42 n.5, p.1330-1344, September 2006 [doi>10.1016/j.ipm.2005.11.003]
• 24: A. F. Smeaton, H. Lee, C. Foley, S. McGivney, and C. Gurrin. Físchlár-diamondtouch: collaborative video searching on a table. In SPIE Electronic Imaging - Multimedia Content Analysis, Management, and Retrieval, pages 15--19, 2006.
• 25: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 26: C. Snoek, M. Worring, D. Koelma, and A. Smeulders. A learned lexicon-driven paradigm for interactive video retrieval. IEEE Trans. on Multimedia, 9(2):280--292, Feb. 2007.
• 27: C. G. M. Snoek, I. Everts, J. C. van Gemert, J.-M. Geusebroek, B. Huurnink, D. C. Koelma, M. van Liempt, O. de Rooij, K. E. A. van de Sande, A. W. M. Smeulders, J. R. R. Uijlings, and M. Worring. The mediamill trecvid 2007 semantic video search engine. In Proceedings of the 5th TRECVID Workshop, November 2007.
• 28: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 29: Dacheng Tao , Xiaoou Tang , Xuelong Li , Xindong Wu, Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.7, p.1088-1099, July 2006 [doi>10.1109/TPAMI.2006.134]
• 30: Kent Wittenburg , Clifton Forlines , Tom Lanning , Alan Esenther , Shigeo Harada , Taizo Miyachi, Rapid serial visual presentation techniques for consumer digital video devices, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.115-124, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964709]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Design, Experimentation, Human Factors

Keywords:
collaborative information retrieval, interactive video search, multimedia information retrieval