How many high-level concepts will fill the semantic gap in news video retrieval?

ABSTRACT

A number of researchers have been building high-level semantic concept detectors such as outdoors, face, building, etc., to help with semantic video retrieval. Using the TRECVID video collection and LSCOM truth annotations from 300 concepts, we simulate performance of video retrieval under different assumptions of concept detection accuracy. Even low detection accuracy provides good retrieval results, when sufficiently many concepts are used. Considering this extrapolation under reasonable assumptions, this paper arrives at the conclusion that "concept-based" video retrieval with fewer than 5000 concepts, detected with minimal accuracy of 10% mean average precision is likely to provide high accuracy results, comparable to text retrieval on the web, in a typical broadcast news collection. We also derive evidence that it should be feasible to find sufficiently many new, useful concepts that would be helpful for retrieval.

REFERENCES

• 1: The thesaurus for graphic materials: Its history, use, and future. Cataloging & Classification Quarterly, 31(3/4):189--212, 2001.
• 2: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 3: Steven M. Beitzel , Eric C. Jensen , Ophir Frieder , Abdur Chowdhury , Greg Pass, Surrogate scoring for improved metasearch precision, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076139]
• 4: S. F. Chang, R. Manmatha, and T. S. Chua. Combining text and audio-visual features in video indexing. In IEEE ICASSP 2005, 2005.
• 5: A. G. Hauptmann, R. Baron, M.-Y. Chen, M. Christel, P. Duygulu, C. Huang, R. Jin, W.-H. Lin, T. Ng, N. Moraveji, N. Papernick, C. Snoek, G. Tzanetakis, J. Yang, R. Yan, and H. Wactlar. Informedia at TRECVID 2003: Analyzing and searching broadcast news video. In Proc. of TRECVID, 2003.
• 6: Waltraud Huyer , Arnold Neumaier, Global Optimization by Multilevel Coordinate Search, Journal of Global Optimization, v.14 n.4, p.331-355, June 1999 [doi>10.1023/A:1008382309369]
• 7: G. R. Institute. Art and architecture thesaurus on line, 2006.
• 8: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 9: John R. Kender , Milind R. Naphade, Visual Concepts for News Story Tracking: Analyzing and Exploiting the NIST TRECVID Video Annotation Experiment, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.1174-1181, June 20-26, 2005 [doi>10.1109/CVPR.2005.371]
• 10: Proceedings of the International Conference on Image and Video Retrieval, July 2002
• 11: C. Lin, B. Tseng, and J. Smith. VideoAnnEx: IBM MPEG-7 annotation tool for multimedia indexing and concept learning. In IEEE International Conference on Multimedia and Expo, 2003.
• 12: Wei-Hao Lin , Alexander Hauptmann, News video classification using SVM-based multimodal classifiers and combination strategies, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641075]
• 13: Christopher D. Manning , Hinrich Schütze, Foundations of statistical natural language processing, MIT Press, Cambridge, MA, 1999
• 14: Marjo Markkula , Eero Sormunen, End-User Searching Challenges Indexing Practices inthe Digital Newspaper Photo Archive, Information Retrieval, v.1 n.4, p.259-285, January 2000 [doi>10.1023/A:1009995816485]
• 15: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 16: M. R. Naphade, T. Kristjansson, B. Frey, and T. Huang. Probabilistic multimedia objects (multijects): A novel approach to video indexing and retrieval in multimedia systems. In Proc. of ICIP, 1998.
• 17: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 18: Apostol (Paul) Natsev , Milind R. Naphade , Jelena TešiĆ, Learning the semantics of multimedia queries and concepts from a small number of examples, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101288]
• 19: NIST. Overview of trecvid 2006, 2006.
• 20: C. of Australian State Libraries. Australian pictorial thesaurus, 2005.
• 21: P. Over, T. Ianeva, W. Kraaij, and A. Smeaton. Trecvid 2005 - an overview. In Proceedings of TRECVID 2005. NIST, USA, 2005.
• 22: Kerry Rodden , Wojciech Basalaj , David Sinclair , Kenneth Wood, Does organisation by similarity assist image browsing?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.190-197, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365097]
• 23: A. Smeaton and P. Over. TRECVID: Benchmarking the effectiveness of information retrieval tasks on digital video. In Proc. of the Intl. Conf. on Image and Video Retrieval, 2003.
• 24: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 25: J. R. Smith, C. Y. Lin, M. R. Naphade, P. Natsev, and B. Tseng. Advanced methods for multimedia signal processing. In Intl. Workshop for Digital Communications IWDC, Capri, Italy, 2002.
• 26: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 27: Howard D. Wactlar , Michael G. Christel , Yihong Gong , Alexander G. Hauptmann, Lessons Learned from Building a Terabyte Digital Video Library, Computer, v.32 n.2, p.66-73, February 1999 [doi>10.1109/2.745722]
• 28: Yi Wu , Edward Y. Chang , Kevin Chen-Chuan Chang , John R. Smith, Optimal multimodal fusion for multimedia data analysis, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027665]
• 29: Rong Yan , Alexander G. Hauptmann, Probabilistic models for combining diverse knowledge sources in multimedia retrieval, Carnegie Mellon University, Pittsburgh, PA, 2006
• 30: Rong Yan , Alexander G. Hauptmann, The combination limit in multimedia retrieval, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957086]
• 31: J. Yang, M. Y. Chen, and A. G. Hauptmann. Finding person x: Correlating names with visual appearances. In Intl. Conf. on Image and Video Retrieval (CIVR'04), Ireland, 2004.
• 32: G. K. Zipf. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Hafner Pub. Co, 1972.

INDEX TERMS

General Terms:
Experimentation

Keywords:
LSCOM, concept-based video retrieval, high-level semantic concepts, semantic gap