Panoramic video capturing and compressed domain virtual camera control

ABSTRACT

A system for capturing panoramic video and a novel method for corresponding compressed domain virtual camera control is presented. It targets applications such as classroom lectures and video conferencing. The proposed method is based on the FlyCam panoramic video system that is designed to produce high resolution and wide-angle video sequences by stitching the video pictures from multiple stationary cameras. The panoramic video sequence is compressed into an MPEG-2 stream for delivery. The proposed method integrates region of Interest (ROI) detection, tracking, and virtual camera control, and works on compressed domain information only. It first detects the ROI in the P (predictive coded) picture using only the macroblock type information, It then up-samples this detection result to obtain the ROI of the whole video stream. The ROI is tracked using a Kalman filter. The Kalman filter estimation results are used for virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. The proposed system has been implemented for real time processing.

REFERENCES

• 1: James R. Bergen , P. Anandan , Keith J. Hanna , Rajesh Hingorani, Hierarchical Model-Based Motion Estimation, Proceedings of the Second European Conference on Computer Vision, p.237-252, May 19-22, 1992
• 2: Burt, P. J., and Adelson, E. H., "The Laplacian Pyramid as a Compact Image Code," IIB<E Trans. On Conrmunications. 3 I, pp. 532-540, 1983.
• 3: L. W. Campbell , A. F. Bobick, Recognition of human body motion using phase space constraints, Proceedings of the Fifth International Conference on Computer Vision, p.624, June 20-23, 1995
• 4: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• 5: T. Darrell , G. Gordon , M. Harville , J. Woodfill, Integrated Person Tracking Using Stereo, Color, and Pattern Detection, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.601, June 23-25, 1998
• 6: Foote, J. and Kimber, D.,"FlyCam: practical panoramic video and automatic camera control," Proc ICME '2000, pp. 1419- 1422,200O.
• 7: Barry G. Haskell , Atul Puri , Arun N. Netravali, Digital Video: An introduction to MPEG-2, Chapman & Hall, Ltd., London, UK, 1996
• 8: Aditi Majumder , W. Brent Seales , M. Gopi , Henry Fuchs, Immersive teleconferencing: a new algorithm to generate seamless panoramic video imagery, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.169-178, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319485]
• 9: MPEG: http:llwww.mpeg.org
• 10: Shree K. Nayar, Catadioptric Omnidirectional Camera, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.482, June 17-19, 1997
• 11: Nicolescu , M., and Medioni, G. ,'Electronic pan-tilt-zoom: a solution for intelligent room systems,"in l'roc. ICMI" '2000, pp. 158 l-l 584,200O.
• 12: O'Rourke, J., and Badler., N.I., 'Model-based image analysis of human motion using constraint propagation," IEEE Trans. Pa&v-n Analysis and Machine Zntelligence, 2(6), pp. 522-536, 1980.
• 13: Sony EVI-D30: www.sony.com.
• 14: Rainer Stiefelhagen , Jie Yang , Alex Waibel, Modeling focus of attention for meeting indexing, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.3-10, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319464]
• 15: Sun, X., and Manjunath, B. S. "Motion Quantized Alpha Histogram as a Video Unit Descriptor," ZsoilliC JTCIiXT29/WGl l/P75, " MPEG7 group, 1999.
• 16: Swaminathan, R. and Nayar, S., "Non-metric Calibration of Wide-angle Lenses and Polycameras," in Proc C:VZU' '99, pp. 413-419,1999.
• 17: Laura Teodosio , Walter Bender, Salient video stills: content and context preserved, Proceedings of the first ACM international conference on Multimedia, p.39-46, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.166270]
• 18: Wang, C. and Brandstein , M S. "A Hybrid Real-Time Face Tracking System,' in Proc. ICSSP '98, pp. 3737-3740, 1998.
• 19: Wang, H. and Chang , S-F., "A Highly Efficient System for Face Region Detection in MPEG Video," IEEE Trans c'ircuifs and LSys. ,I;,r Video 'lkch, 7(4), pp. 6 15-628, 1997
• 20: George Wolberg, Digital Image Warping, IEEE Computer Society Press, Los Alamitos, CA, 1994
• 21: Hongjiang Zhang , Chien Yong Low , Stephen W. Smoliar, Video parsing and browsing using compressed data, Multimedia Tools and Applications, v.1 n.1, p.89-111, March 1995 [doi>10.1007/BF01261227]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.4 Restoration
          Subjects: Kalman filtering

Additional Classification:
  C. Computer Systems Organization
  C.3 SPECIAL-PURPOSE AND APPLICATION-BASED SYSTEMS
      Subjects: Real-time and embedded systems

  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications
          Subjects: Computer conferencing, teleconferencing, and videoconferencing

General Terms:
Algorithms, Design, Performance