Multi-camera spatio-temporal fusion and biased sequence-data learning for security surveillance

ABSTRACT

We present a framework for multi-camera video surveillance. The framework consists of three phases: detection, representation, and recognition. The detection phase handles multi-source spatio-temporal data fusion for efficiently and reliably extracting motion trajectories from video. The representation phase summarizes raw trajectory data to construct hierarchical, invariant, and content-rich descriptions of the motion events. Finally, the recognition phase deals with event classification and identification on the data descriptors. Because of space limits, we describe only briefly how we detect and represent events, but we provide in-depth treatment on the third phase: event recognition. For effective recognition, we devise a sequence-alignment kernel function to perform sequence data learning for identifying suspicious events. We show that when the positive training instances (i.e., suspicious events) are significantly outnumbered by the negative training instances (benign events), then SVMs (or any other learning methods) can suffer a high incidence of errors. To remedy this problem, we propose the kernel boundary alignment (KBA) algorithm to work with the sequence-alignment kernel. Through empirical study in a parking-lot surveillance setting, we show that our spatio-temporal fusion scheme and biased sequence-data learning method are highly effective in identifying suspicious events.

REFERENCES

• 1: S. Amari , S. Wu, Improving support vector machine classifiers by modifying kernal functions, Neural Networks, v.12 n.6, p.783-789, July 1999 [doi>10.1016/S0893-6080(99)00032-5]
• 2: Ronald T Azuma, Predictive Tracking for Augmented Reality, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1995
• 3: Y. Bengio. Markovian models for sequential data. Neural computing serveys, 1998.
• 4: J. E. Boyd , E. Hunter , P. H. Kelly , L.-C. Tai , C. B. Phillips , R. C. Jain, MPI-Video Infrastructure for Dynamic Environments, Proceedings of the IEEE International Conference on Multimedia Computing and Systems, p.249, June 28-July 01, 1998 [doi>10.1109/MMCS.1998.693651]
• 5: Tolga Bozkaya , Meral Ozsoyoglu, Distance-based indexing for high-dimensional metric spaces, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.357-368, May 11-15, 1997, Tucson, Arizona, United States
• 6: R. G. Brown. Introduction to Random Signal Analysis and Kalman filtering. Wiley, New York, NY, 1983.
• 7: Christopher J. C. Burges, Geometry and invariance in kernel based methods, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 8: Darya Chudova , Padhraic Smyth, Pattern discovery in sequences under a Markov assumption, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775070]
• 9: E. Church. Revised Geometry of the Aerial Photograph. Bulletin of Aerial Photogrammetry, 15, 1945.
• 10: H. Cohn. Conformal Mapping on Riemann Surfaces. Dover Pubns, 1980.
• 11: R. T. Collins, A. J. Lipton, T. Kanade, H. Fujiyoshi, D. Duggins, Y. Tsin, D. Tolliver, N. Enomoto, O. Hasegawa, P. Burt, and L. Wixson. A system for video surveillance and monitoring (VSAM project final report). CMU Technical Report CMU-RI-TR-00-12, 2000.
• 12: Daniel F. Dementhon , Larry S. Davis, Model-based object pose in 25 lines of code, International Journal of Computer Vision, v.15 n.1-2, p.123-141, June 1995 [doi>10.1007/BF01450852]
• 13: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 14: Gerald E. Farin, Curves and Surfaces for Computer-Aided Geometric Design: A Practical Code, Academic Press, Inc., Orlando, FL, 1996
• 15: Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, Cambridge, MA, 1993
• 16: James D. Foley , Andries van Dam , Steven K. Feiner , John F. Hughes, Computer graphics: principles and practice (2nd ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1990
• 17: Keinosuke Fukunaga, Introduction to statistical pattern recognition (2nd ed.), Academic Press Professional, Inc., San Diego, CA, 1990
• 18: R. Haralick, H. Joo, C. Lee, X. Zhuang, V. Vaidya, and M. Kim. Pose estimation from corresponding point data. IEEE Trans. Syst., Man, Cybern., 19:1426--46, 1989.
• 19: Radu Horaud , Fadi Dornaika , Bart Lamiroy, Object Pose: The Link between Weak Perspective,Paraperspective, and Full Perspective, International Journal of Computer Vision, v.22 n.2, p.173-189, March 1997 [doi>10.1023/A:1007940112931]
• 20: Tommi Jaakkola , Mark Diekhans , David Haussler, Using the Fisher Kernel Method to Detect Remote Protein Homologies, Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology, p.149-158, August 06-10, 1999
• 21: Tommi S. Jaakkola , David Haussler, Exploiting generative models in discriminative classifiers, Proceedings of the 1998 conference on Advances in neural information processing systems II, p.487-493, July 1999
• 22: T. S. Jaakola and D. Haussler. Probabilistic kernel regression models. In Conference of AI and Statistics, 1999.
• 23: S. J. Julier, J. K. Uhlmann, and H. F. Durrant-Whyte. A New Approach for Filtering Nonlinear Systems. In Proc. Amiercan Control Conference, Seattle, WA, 1995.
• 24: K. Kanatani. Optimal homography computation with a reliability measure. Proc. IAPR Workshop Machine Vision Applications, Nov. 1998.
• 25: V. Kettnaker and R. Zabih. Bayesian multi-camera surveillance. CVPR, 1999.
• 26: Lily Lee , Raquel Romano , Gideon Stein, Monitoring Activities from Multiple Video Streams: Establishing a Common Coordinate Frame, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.758-767, August 2000 [doi>10.1109/34.868678]
• 27: Leslie, Christin, E. Eskin, A. Cohen, J. Weston, and W. S. Noble. Mismatch string kernels for svm protein classification. Neural Information Processing Systems, 2002.
• 28: C. Leslie, E. Eskin, and W. S. Noble. The spectrum kernel: a string kernel for svm protein classification. Proceedings of the Pacific Symposium on Biocomputing. World Scientific, 2002.
• 29: J. Lou, H. Yang, W. Hu, and T. Tan. Visual vehicle tracking using an improved ekf. ACCV, 2002.
• 30: S. Maybank, A. Worrall, and G. Sullivan. Filter for car tracking based on acceleration and steering angle. Proc. of British Machine Vision Conference, 1996.
• 31: P. S. Maybeck. Stochastic Models, Estimation, and Control, vol. 1. Academic Press, New York, NY, 1979.
• 32: I. Pavlidis and V. Morellas. Two examples of indoor and outdoor surveillance systems: Motivation, design, and testing. Proceedings 2nd European Workshop on Advanced Video-Based Surveillance, 2001.
• 33: I. Pavlidis, V. Morellas, P. Tsiamyrtzis, and S. Harp. Urban surveillance systems: From the laboratory to the commercial world. Proc. of the IEEE, 89(10), 2001.
• 34: C. Regazzoni and P. K. Varshney. Multisensor surveillance systems based on image and video data. Proc. of the IEEE Conf. on Image Proc., 2002.
• 35: F. W. Sears. Optics, 3rd Ed. Addison-Wesley, Reading, MA, 1958.
• 36: T. Starner and A. Pentland. Visual recognition of american sign language using hidden markov models. Technical Report Master's Thesis, MIT, Program in Media Arts & Sciences, MIT Media Laboratory, 1994.
• 37: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 38: C. Watkins. Dynamic alignment kernels. Technical Report, Department of Computer Science, University of London, 1999.
• 39: Greg Welch , Gary Bishop, An Introduction to the Kalman Filter, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1995
• 40: G. Wu and E. Chang. Adaptive feature-space conformal transformation for learning imbalanced data. International Conference of Machine Learning (ICML), August 2003.
• 41: Gang Xu , Zhengyou Zhang, Epipolar Geometry in Stereo, Motion, and Object Recognition: A Unified Approach, Kluwer Academic Publishers, Norwell, MA, 1996
• 42: Zhengyou Zhang, A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.11, p.1330-1334, November 2000 [doi>10.1109/34.888718]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Feature representation

General Terms:
Algorithms, Performance, Theory