Learning structured concept-segments for interactive video retrieval

ABSTRACT

Now with a large lexicon of over 300 semantic concepts available for indexing purpose, video retrieval can be made easier by leveraging on the available semantic indices. However, any successful concept-based video retrieval approach must take the following into account: though improving continuously, these concept indexing results are still far from perfect; more concepts are awaiting for detection instead of being detected due to the limited amount of annotated data. If possible, a structured query formulation other than a simple AND logic of some chosen concepts is more desirable to model the complex query need with the fixed concept lexicon. In this paper, we propose a concept-based interactive video retrieval approach to tackle these problems. To better represent the query information need, the proposed approach learns through the feedback information a structured formulation which consists of multiple semantic concept combination terms. Instead of taking the top-ranked items from the selected concepts, it leverages on a simple mining algorithm to drill down to concept-segments where the positive examples are most densely populated than the negative examples. We evaluate the proposed method on the large scale TRECVid 05&06 data sets, and achieve promising results. Retrieval in concept-segment level has a 14% improvement upon the concept-level. Structured query formulation improves around 13% compared with the simple logical AND formulation. The learning and retrieval process only takes 300ms, satisfying the real-time interactive search need.

REFERENCES

• 1: H. Bay, T. Tuytelaars, and L. Gool. 2006. Surf: Speeded up robust features. In Proc. of ECCV, 2006.
• 2: M. Campbell and et al.2006. IBM research trecvid-2006 video retrieval system. In Proc. Of TRECVid, 2006.
• 3: T. Hastie, R. Tibshirani, and J. Friedman.2001. The Elements of Statistical Learning. Springer series in statistics. Springer, New York, 2001.
• 4: L. Hollink, G.P. Nguyen, D.C. Koelma, A. Th. Schreiber, and M. Worring.2004. User Strategies in Video Retrieval: a Case Study. In Proc. of CIVR, 2004.
• 5: J. Kender.2007. A Large Scale Concept Ontology for News Stories: Empirical Methods, Analysis, and Improvements. In Proc of ICME, 2007
• 6: Xirong Li , Dong Wang , Jianmin Li , Bo Zhang, Video search in concept subspace: a text-like paradigm, Proceedings of the 6th ACM international conference on Image and video retrieval, p.603-610, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282366]
• 7: Donald Metzler , W. Bruce Croft, Combining the language model and inference network approaches to retrieval, Information Processing and Management: an International Journal, v.40 n.5, p.735-750, September 2004 [doi>10.1016/j.ipm.2004.05.001]
• 8: D. Metzler and R. Manmatha.2004. An inference network approach to image retrieval. In Proc. of CIVR, 2004.
• 9: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 10: Apostol (Paul) Natsev , Milind R. Naphade , Jelena TešiĆ, Learning the semantics of multimedia queries and concepts from a small number of examples, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101288]
• 11: Apostol (Paul) Natsev , Alexander Haubold , Jelena Tešić , Lexing Xie , Rong Yan, Semantic concept-based query expansion and re-ranking for multimedia retrieval, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291448]
• 12: S.-Y. Neo, J. Zhao, M.-Y. Kan, and T.-S. Chua.2006. Video retrieval using high level features: Exploiting query matching and condence-based weighting. In Proc. of CIVR, 2006.
• 13: Y. Rui, T. S. Huang, M. Ortega, and S. Mehrotra.1998. Relevance Feedback: A Power Tool for Interactive Content-Based Image Retrieval. IEEE Trans. on Circuits and Systems for Video Technology 8(5) (Sep. 1998):644--655
• 14: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 15: C. G. Snoek, M. Worring, D. C. Koelma, and A. W. Smeulders.2007. A learned lexicon-driven paradigm for interactive video retrieval. IEEE Trans. Multimedia, 2007.
• 16: D. Wang, J. Li, and B. Zhang.2006. Relay boost fusion for learning rare concepts in multimedia. In Proc. of CIVR, 2006.
• 17: Dong Wang , Xirong Li , Jianmin Li , Bo Zhang, The importance of query-concept-mapping for automatic video retrieval, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291293]
• 18: Dong Wang , Xiaobing Liu , Linjie Luo , Jianmin Li , Bo Zhang, Video diver: generic video indexing with diverse features, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290094]
• 19: J. Yuan et al.2007. THU and ICRC at TRECVid 2007, In Proc. of TRECVid, 2007
• 20: W. Zheng, J. Li, Z. Si, F. Lin, and B. Zhang.2006. Using high-level semantic features in video retrieval. In Proc. of CIVR, 2006.
• 21: X. S. Zhou, T. S. Huang.2003. Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems, Vol. 8, No. 6. (1 April 2003), pp. 536--544.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Multimedia databases; Query processing

General Terms:
Algorithms, Design, Experimentation

Keywords:
interactive video retrieval, mining structured concepts, relevance feedback