Probabilistic optimized ranking for multimedia semantic concept detection via RVM

ABSTRACT

We present a probabilistic ranking-driven classifier for the detection of video semantic concept, such as airplane, building, etc. Most existing concept detection systems utilize Support Vector Machines (SVM) to perform the detection and ranking of retrieved video shots. However, the margin maximization principle of SVM does not perform ranking optimization but merely classification error minimization. To tackle this problem, we exploit the sparse Bayesian kernel model, namely the relevance vector machine (RVM), as the classifier for semantic concept detection. Based on automatic relevance determination principle, RVM outputs the posterior probabilistic prediction of the semantic concepts. This inference output is optimal for ranking the target video shots, according to the Probabilistic Ranking Principle. The probability output of RVM on individual uni-modal features also facilitates probabilistic fusion of multi-modal evidences to minimize Bayes risk. We demonstrate both theoretically and empirically that RVM outperforms SVM for video semantic concept detection. The testings on TRECVID 07 dataset show that RVM produces statically significant improvements in MAP scores over the SVM-based methods.

REFERENCES

• 1: A. Agarwal and W. Triggs. Hyperfeatures - multilevel local coding for visual recognition. In ECCV International Workshop on Statistical Learning in Computer Vision, 2006.
• 2: Christopher M. Bishop , Michael E. Tipping, Variational Relevance Vector Machines, Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, p.46-53, June 30-July 03, 2000
• 3: S.-F. Chang, W. Hsu, L. Kennedy, L. Xie, A. Yanagawa, E. Zavesky, and D.-Q. Zhang. Columbia university trecvid-2005 video search and high-level feature extraction. In TREC Video Retrieval Evaluation Proceedings, March 2006.
• 4: T.-S. Chua, S.-Y. Neo, Y.-T Zheng, H.-K. Goh, Y. Xiao, S. Tang, and M. Zhao. Trecvid-2006 by nus-i2r. In TREC Video Retrieval Evaluation Proceedings, March 2006.
• 5: Gordon V. Cormack , Thomas R. Lynam, Validity and power of t-test for comparing MAP and GMAP, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277892]
• 6: Chitra Dorai , Svetha Venkatesh, Guest Editors' Introduction: Bridging the Semantic Gap with Computational Media Aesthetics, IEEE MultiMedia, v.10 n.2, p.15-17, March 2003 [doi>10.1109/MMUL.2003.1195157]
• 7: M. Evans and T. Swartz. Methods for approximating integrals in statistics with special emphasis on Bayesian integration problems. Statistical Science, 10:254--272, 1995.
• 8: Stephan Fischer , Rainer Lienhart , Wolfgang Effelsberg, Automatic recognition of film genres, Proceedings of the third ACM international conference on Multimedia, p.295-304, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215283]
• 9: S. Gao and Q. Sun. Classifier optimization for multimedia semantic concept detection. In Proceedings of the International Conference on Multimedia and Expo (ICME), pages 61--74, 2006.
• 10: A. G. Hauptmann, M.-Y. Chen, M. Christel, W.-H. Lin, R. Yan, and J. Yang. Multi-lingual broadcast news retrieval. In Proceedings of TREC Video Retrieval Evaluation, March 2006.
• 11: Y.-G. Jiang, X. Wei, C.-W. Ngo, H.-K. Tan, W. Zhao, and X. Wu. Modeling local interest points for semantic detection and video search at trecvid 2006. In TREC Video Retrieval Evaluation Proceedings, March 2006.
• 12: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Affine-Invariant Local Descriptors and Neighborhood Statistics for Texture Recognition, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.649, October 13-16, 2003
• 13: HD. Le, S. Satoh, and T. Matsui. Nii-ism, japan at trecvid 2007: High level feature extraction. In TREC Video Retrieval Evaluation Proceedings, Nov 2007.
• 14: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 15: J. Luo, M. R. Boutell, R. T. Gray, and C. M. Brown. Image transform bootstrapping and its applications to semantic scene classification. IEEE Transactions on SMC, 35(3):563--570, 2005.
• 16: T. Mei, X. Hua, W. Lai, , L. Yang, Z. Zha, Y. Liu, Z. Gu, G. Qi, M. Wang, J. Tang, , X. Yuan, Z. Lu, and J. Liu. Msra-ustc-sjtu at trecvid 2007: High-level feature extraction and search. In TREC Video Retrieval Evaluation Proceedings, Nov 2007.
• 17: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 18: Radford M. Neal, Bayesian Learning for Neural Networks, Springer-Verlag New York, Inc., Secaucus, NJ, 1996
• 19: C. Ngo, Y. Jiang, X. Wei, F. Wang, W. Zhao, H. Tan, and X. Wu. Experimenting vireo-374: Bag-of-visual-words and visual-based ontology for semantic video indexing and search. In TREC Video Retrieval Evaluation Proceedings, Nov 2007.
• 20: J. Platt. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Margin Classiers, MIT Press, pages 61--74, 2000.
• 21: Brian D. Ripley , N. L. Hjort, Pattern Recognition and Neural Networks, Cambridge University Press, New York, NY, 1995
• 22: S. E. Robertson, The probability ranking principle in IR, Readings in information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 23: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 24: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 25: Cees G. M. Snoek , Marcel Worring , Arnold W. M. Smeulders, Early versus late fusion in semantic video analysis, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101236]
• 26: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 27: M. Tipping. The relevance vector machine. In Advances in Neural Information Processing Systems, San Mateo, CA. Morgan Kaufmann, 2000.
• 28: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 29: H. C. Wu , Robert W. P. Luk , K. F. Wong, Probability ranking principle via optimal expected rank, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277872]
• 30: Hwanjo Yu, SVM selective sampling for ranking with application to data retrieval, Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081911]
• 31: J. Yuan, Z. Guo, L. Lv, W. Wan, T. Zhang, D. Wang, X. Liu, C. Liu, S. Zhu, D. Wang, Y. Pang, N. Ding, Y. Liu, J. Wang, X. Zhang, X. Tie, Z. Wang, H.Wang, T. Xiao, Y. Liang, J. Li, F. Lin, , B. Zhang, L. JianGuo, W. WeiXin, T. XiaoFeng, D. DaYong, C. YuRong, W. Tao, , and Z. Yimin. Thu and icrc at trecvid 2007. In TREC Video Retrieval Evaluation Proceedings, Nov 2007.
• 32: Yisong Yue , Thomas Finley , Filip Radlinski , Thorsten Joachims, A support vector method for optimizing average precision, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277790]
• 33: Zhang Ling , Zhang Bo, Relationship between support vector set and kernel functions in SVM, Journal of Computer Science and Technology, v.17 n.5, p.549-555, May 2002 [doi>10.1007/BF02948823]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Experimentation

Keywords:
high level feature detection, learning to rank, relevance vector machine, video concept detection