Automatically extracting highlights for TV Baseball programs

ABSTRACT

In today's fast-paced world, while the number of channels of television programming available is increasing rapidly, the time available to watch them remains the same or is decreasing. Users desire the capability to watch the programs time-shifted (on-demand) and/or to watch just the highlights to save time. In this paper we explore how to provide for the latter capability, that is the ability to extract highlights automatically, so that viewing time can be reduced.

We focus on the sport of baseball as our initial target—it is a very popular sport, the whole game is quite long, and the exciting portions are few. We focus on detecting highlights using audio-track features alone without relying on expensive-to-compute video-track features. We use a combination of generic sports features and baseball-specific features to obtain our results, but believe that may other sports offer the same opportunity and that the techniques presented here will apply to those sports. We present details on relative performance of various learning algorithms, and a probabilistic framework for combining multiple sources of information. We present results comparing output of our algorithms against human-selected highlights for a diverse collection of baseball games with very encouraging results.

REFERENCES

• 1: WebTV, http:/www.webtv.eom.
• 2: TVReplay, http://www.tvreplay.eorn
• 3: Tivo Inc., http://www.tivo.com
• 4: Francis C. Li , Anoop Gupta , Elizabeth Sanocki , Li-wei He , Yong Rui, Browsing digital video, Proceedings of the SIGCHI conference on Human factors in computing systems, p.169-176, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332425]
• 5: Zhang, H., et al. Automatic Parsing of News Video. in IEEE Conference on Multimedia Computing and Systems. 1994.
• 6: MPEG-7: Context and Objectives (V.7). in ISO/IEC JTCI/SC29/WGI I N2207, MPEG98. March 1998.
• 7: ATVEF, http://www.atvef.eom
• 8: Yong Rui , Thomas S. Huang , Sharad Mehrotra, Constructing table-of-content for videos, Multimedia Systems, v.7 n.5, p.359-368, September 1999 [doi>10.1007/s005300050138]
• 9: Howard D. Wactlar , Michael G. Christel , Yihong Gong , Alexander G. Hauptmann, Lessons Learned from Building a Terabyte Digital Video Library, Computer, v.32 n.2, p.66-73, February 1999 [doi>10.1109/2.745722]
• 10: McGee, T. and N. Dimitrova. Parsing TV Program Structures for Identification and Removal of Non-Story Segments. in SPIE Conf. on Storage and Retrieval for Image and Video Databases. 1999. San Jo80, CA.
• 11: Extracting story units from long programs for video browsing and navigation, Proceedings of the 1996 International Conference on Multimedia Computing and Systems (ICMCS '96), p.0296, June 17-23, 1996
• 12: Arons, B. Pitch-based Emphasis Detection for Segmenting Speech Recordings. in International Conference on Spoken Language Processing. 1994.
• 13: Liwei He , Elizabeth Sanocki , Anoop Gupta , Jonathan Grudin, Auto-summarization of audio-video presentations, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.489-498, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319691]
• 14: Zhu Liu , Yao Wang , Tsuhan Chen, Audio Feature Extraction and Analysis for Scene Segmentation and Classification, Journal of VLSI Signal Processing Systems, v.20 n.1-2, p.61-79, Oct. 1998 [doi>10.1023/A:1008066223044]
• 15: Savitha Srinivasan , Dragutin Petkovic , Dulce Ponceleon, Towards robust features for classifying audio in the CueVideo system, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.393-400, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319658]
• 16: Tong Zhang , C.-C. Jay Kuo, Heuristic approach for generic audio data segmentation and annotation, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.67-76, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319471]
• 17: Chua Hock Chuan, Automatic Parsing of TV Soccer Programs, Proceedings of the International Conference on Multimedia Computing and Systems, p.167, May 15-18, 1995 [doi>10.1109/MMCS.1995.484921]
• 18: Integrated Image and Speech Analysis for Content-Based Video Indexing, Proceedings of the 1996 International Conference on Multimedia Computing and Systems (ICMCS '96), p.0306, June 17-23, 1996
• 19: Rui, Y. and P. Anandan. Segmenting Visual Action Units Based on Spatial-Temporal Motion Patterns. in IEEE Conf on Computer Vision and Pattern Recgonition. 2000.
• 20: Lawrence Rabiner , Biing-Hwang Juang, Fundamentals of speech recognition, Prentice-Hall, Inc., Upper Saddle River, NJ, 1993
• 21: Dellaert, F., T. Polzin, and A. Walbel. Recognizing Emotion in Speech. in IEEE ICASSP. 1995.
• 22: Droppo, J. and A. Acero. Maximum A Posterior Pitch Tracking. in IEEE ICASSP. 1999.
• 23: Huang, L.-S. and C.-H. Yang. A Novel Approach to Robust Speech Endpoint Detection in Car Environments (submitted). in IEEE ICASSP. 2000.
• 24: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 25: Christopher M. Bishop, Neural networks for pattern recognition, Oxford University Press, Oxford, 1996
• 26: Burges, C., A Tutorial on Support Vector Machines for Pattern Recognition, U. Fayyad, Editor. 1999, Klnwer Academic Publishers: Boston.
• 27: Platt, J.C., Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods, in Advances in Large Margin Classifiers, P.B. Alexander J. Smola, Bernhard Seholkopf and Dale Schuurmans, Editor. 1999.
• 28: Yoshiharu Ishikawa , Ravishankar Subramanya , Christos Faloutsos, MindReader: Querying Databases Through Multiple Examples, Proceedings of the 24rd International Conference on Very Large Data Bases, p.218-227, August 24-27, 1998

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis
  I.4 IMAGE PROCESSING AND COMPUTER VISION

  J. Computer Applications

General Terms:
Algorithms, Human Factors, Languages, Measurement

Keywords:
audio, baseball, highlights, summarization, television, video