Classification of video events using 4-dimensional time-compressed motion features

ABSTRACT

Among the various types of semantic concepts modeled, events pose the greatest challenge in terms of computational power needed to represent the event and accuracy that can be achieved in modeling it. We introduce a novel low-level visual feature that summarizes motion in a shot. This feature leverages motion vectors from MPEG-encoded video, and aggregates local motion vectors over time in a matrix, which we refer to as a motion image. The resulting motion image is representative of the overall motion in a video shot, having compressed the temporal dimension while preserving spatial ordering. Building motion models using this feature permits us to combine the power of discriminant modeling with the dynamics of the motion in video shots that cannot be accomplished by building generative models over a time series of motion features from multiple frames in the video shot. Evaluation of models built using several motion image features in the TRECVID 2005 dataset shows that use of this novel motion feature results an average improvement in concept detection performance by 140% over existing motion features. Furthermore, experiments also reveal that when this motion feature is combined with static feature representations of a single keyframe from the shot such as color and texture features, the fused detection results in an improvement between 4 to 12% over the fusion across the static features alone.

REFERENCES

• 1: Adams, W. H., Amir, A., Dorai, C., Ghoshal, S., Iyengar, G., Jaimes, A., Lang, C. Lin, C. Y., Naphade, M. R., Natsev, A., Neti, C., Nock, H. J., Permutter, H., Singh, R., Srinivasan, S., Smith, J. R., Tseng, B. L., Varadaraju, A. T., and Zhang, D. IBM Research TREC-2002 Video Retrieval System. In Proceedings of the Text Retrieval Conference (TREC) (Gaithersburg, MD, November 2002), NIST Special Publications, SP 500-251, 2002, 289--298.
• 2: Amir, A., Berg, M., Chang, S. F., Iyengar, G., Lin, C., Naphade, M. R., Natsev, A., Neti, C., Nock, H., Hsu, W., Sachdev, I., Smith, J. R., Tseng, B., Wu, Y., and Zhang, D. IBM Research TRECVID-2003 Video Retrieval System. In Proceedings of the TRECVID 2003 Workshop (Gaithersburg, MD, November 2003), NIST Special Publications, 2003.
• 3: Amir, A., Argillander J., Berg, M., Chang, S. F., Iyengar, G., Lin, C., Naphade, M. R., Natsev, A., Hsu, W., Smith, J. R., Tešić, J., Yan, R., Zhang, D. IBM Research TRECVID-2004 Video Retrieval System. In Proceedings of the TRECVID 2004 Workshop (Gaithersburg, MD, November 2004), NIST Special Publications, 2004.
• 4: Amir A., Argillander J., Campbell M., Haubold A., Iyengar G., Ebadollahi S., Kang F., Naphade M. R., Natsev A., Smith J. R., Tešić J., and Volkmer T. IBM Research TRECVID-2005 Video Retrieval System. In Proceedings of the TRECVID 2005 Workshop (Gaithersburg, MD, November 2005), NIST Special Publications, 2005.
• 5: Bresenham, J. E. Algorithm for computer control of a digital plotter. In IBM Systems Journal, Vol. 4 (1), 1965, 25--30.
• 6: Campbell M., Haubold A., Ebadollahi S., Naphade M. R., Natsev P., Smith J. R., Tešić J., and Xie L. IBM Research TRECVID-2006 Video Retrieval System. In Proceedings of the TRECVID 2006 Workshop (Gaithersburg, MD, November 2006), NIST Special Publications, 2006.
• 7: Ewerth, R., Beringer, C., Kopp, T., Nievergall, M., Stadelmann, T., and Freisleben, B. University of Marburg at TRECVID 2005: Shot Boundary Detection and Camera Motion Estimation Results. In Proceedings of the TRECVID 2005 Workshop, NIST Special Publications, Gaithersburg, MD, Nov. 2005.
• 8: Jing Huang , S. Ravi Kumar , Mandar Mitra , Wei-Jing Zhu , Ramin Zabih, Spatial Color Indexing and Applications, International Journal of Computer Vision, v.35 n.3, p.245-268, Dec. 1999 [doi>10.1023/A:1008108327226]
• 9: Cheng Lu , Mark S. Drew , James Au, Classification of summarized videos using hidden markov models on compressed chromaticity signatures, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500217]
• 10: Yu-Fei Ma , Hong-Jiang Zhang, Motion pattern-based video classification and retrieval, EURASIP Journal on Applied Signal Processing, v.2003 n.1, p.199-208, January 2003 [doi>10.1155/S1110865703211021]
• 11: Naphade, M. R., Huang, M. Discovering Recurrent Events in Video Using Unsupervised Methods. In Proceedings of the International Conference on Image Processing (ICIP '02) (Rochester, NY, September 22--25, 2002), IEEE Press, New York, NY, 2002, II-13--II-16.
• 12: Naphade, M. R., Kennedy, L., Kender, J. R., Chang, S. F., Smith, J. R., Over P., and Hauptmann, A. LSCOM-lite: A Light Scale Concept Ontology for Multimedia Understanding for TRECVID 2005. IBM Research Technical Report, RC23612 (W0505-104), May, 2005.
• 13: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 14: Milind R. Naphade , Roy Wang , Thomas S. Huang, Supporting audiovisual query using dynamic programming, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500202]
• 15: Snoek, C. G. M., and Worring, M. Multimedia Event-Based Video Indexing using Time Intervals. In IEEE Transactions on Multimedia, 7(4) (Aug. 2005), 638--647.
• 16: J. W. Davis. Hierarchical Motion History Images for Recognizing Human Motion. In Proceedings of the IEEE Workshop on Detection and Recognition of Events in Video (Vancouver, Canada, July 8, 2001). IEEE Press, New York, NY, 2001, 39--46.
• 17: J. C. Niebles, H. Wang, L. Fei-Fei. Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words. In Proceedings of the British Machine Vision Conference (BMVC '06) (Edinburgh, United Kingdom, September 4--7, 2006). British Machine Vision Association, 2001
• 18: Lihi Zelnik-Manor , Michal Irani, Statistical Analysis of Dynamic Actions, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.9, p.1530-1535, September 2006 [doi>10.1109/TPAMI.2006.194]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
      I.2.6 Learning

General Terms:
Algorithms, Design, Experimentation, Performance, Verification

Keywords:
LSCOM, MPEG motion vectors, TRECVID, motion features