Combining multimodal preferences for multimedia information retrieval

ABSTRACT

Representing and fusing multimedia information is a key issue to discover semantics in multimedia. In this paper we address more specifically the problem of multimedia content retrieval by first defining a novel preference-based representation particularly adapted to the fusion problem, and then, by investigating the RankBoost algorithm to combine those preferences and a learn multimodal retrieval model. The approach has been tested on annotated images and on the complete TRECVID 2005 corpus and compared with SVM-based fusion strategies. The results show that our approach equals SVM performance but, contrary to SVM, is parameter free and faster.

REFERENCES

• 1: L. Boldareva and D. Hiemstra. Interactive content-based retrieval using pre-computed object-object similarities. In Conference on Image and Video Retrieval, CIVR'04, pages 308--316, Dublin, Ireland, 2004.
• 2: Eric Bruno, Nicolas Moenne-Loccoz, and Stéphane Marchand Maillet. Learning user queries in multimodal dissimilarity spaces. In Proceedings of the 3rd International Workshop on Adaptive Multimedia Retrieval, AMR'05, Glasgow, UK, July 2005.
• 3: Eric Bruno , Nicolas Moenne-Loccoz , Stéphane Marchand-Maillet, Unsupervised event discrimination based on nonlinear temporal modeling of activity content, Pattern Analysis & Applications, v.7 n.4, p.402-410, December 2004 [doi>10.1007/s10044-005-0242-9]
• 4: Eric Bruno, Nicolas Moenne-Loccoz, and Stéphane Marchand-Maillet. Asymmetric learning and dissimilarity spaces for content-based retrieval. In Proc. of International Conference on Image and Video Retrieval (CIVR), pages 330--339, Tempe, AZ, July 2006.
• 5: E. Y. Chang, B. Li, G. Wu, and K. Go. Statistical learning for effective visual information retrieval. In Proceedings of the IEEE International Conference on Image Processing, 2003.
• 6: R.P.W. Duin. The combining classifier: To train or not to train? In Proceedings of the 16th International Conference on Pattern Recognition, ICPR'02, volume II, pages 765--770, Quebec City, 2004. IEEE Computer Socity Press.
• 7: Yoav Freund , Raj Iyer , Robert E. Schapire , Yoram Singer, An efficient boosting algorithm for combining preferences, The Journal of Machine Learning Research, 4, p.933-969, 12/1/2003
• 8: J. Gu, L. Lu, H.J Zhang, and J. Yang. Dominant feature vectors based audio similarity measure. In PCM, number 2, pages 890--897, 2004.
• 9: D Heesch and S Rueger. NNk networks for content-based image retrieval. In 26th European Conference on Information Retrieval, Sunderland, UK, 2004.
• 10: Winston H. Hsu and Shih-Fu Chang. Generative, discriminative, and ensemble learning on multi-modal perceptual fusion toward news video story segmentation. In ICME, Taipei, Taiwan, June 2004.
• 11: Anil K. Jain , Aditya Vailaya , Xiong Wei, Query by video clip, Multimedia Systems, v.7 n.5, p.369-384, September 1999 [doi>10.1007/s005300050139]
• 12: Josef Kittler , Mohamad Hatef , Robert P. W. Duin , Jiri Matas, On Combining Classifiers, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.3, p.226-239, March 1998 [doi>10.1109/34.667881]
• 13: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 14: Nicolas Moënne-Loccoz, Eric Bruno, and Stéphane Marchand-Maillet. Interactive partial matching of video sequences in large collections. In IEEE International Conference on Image Processing, Genova, Italy, 11-14 September 2005.
• 15: Giang P. Nguyen , Marcel Worring , Arnold W. M. Smeulders, Similarity learning via dissimilarity space in CBIR, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178695]
• 16: N.C. Oza, R. Polikar, J. Kittler, and F. Roli. Multiple classifier systems. In Series: Lecture Notes in Computer Science, volume 3541. Springer, 2005.
• 17: Elzbieta Pekalska , Pavel Paclik , Robert P. W. Duin, A generalized kernel approach to dissimilarity-based classification, The Journal of Machine Learning Research, 2, 3/1/2002
• 18: C. Petersohn. Fraunhofer HHI at TRECVID 2004: Shot boundary detection system. In TREC Video Retrieval Evaluation Online Proceedings, 2004.
• 19: Philip Resnik. Using information content to evaluate semantic similarity in a taxonomy. In 14th International Joint Conference on Artificial Intelligence, IJCAI, pages 448--453, Montreal, Canada, 1995.
• 20: Robert E. Schapire , Yoram Singer, Improved Boosting Algorithms Using Confidence-rated Predictions, Machine Learning, v.37 n.3, p.297-336, Dec. 1999 [doi>10.1023/A:1007614523901]
• 21: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 22: J. R. Smith, A. Jaimes, C.-Y. Lin, M. Naphade, A. Natsev, and B. Tseng. Interactive search fusion methods for video database retrieval. In IEEE International Conference on Image Processing (ICIP), 2003.
• 23: Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
• 24: Yi Wu , Edward Y. Chang , Kevin Chen-Chuan Chang , John R. Smith, Optimal multimodal fusion for multimedia data analysis, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027665]
• 25: Rong Yan , Alexander G. Hauptmann , Rong Jin, Negative pseudo-relevance feedback in content-based video retrieval, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957087]
• 26: J. Yang and A.G. Hauptmann. Multi-modality analysis for person type classification in news video. In Electronic Imaging'05 - Conference on Storage and Retrieval Methods and Applications for Multimedia, San Jose, USA, Jan 2005.
• 27: X.S. Zhou, A. Garg, and T.S. Huang. A discussion of nonlinear variants of biased discriminant for interactive image retrieval. In Proc. of the 3rd Conference on Image and Video Retrieval, CIVR'04, pages 353--364, 2004.
• 28: X.S. Zhou and T.S. Huang. Small sample learning during multimedia retrieval using biasmap. In Proceedings of the IEEE Conference on Pattern Recognition and Computer Vision, CVPR'01, volume I, pages 11--17, Hawaii, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Theory

Keywords:
RankBoost, multimedia indexing and retrieval, multimodal fusion