Timeline-based information assimilation in multimedia surveillance and monitoring systems

ABSTRACT

Most surveillance and monitoring systems nowadays utilize multiple types of sensors. However, due to the asynchrony among and diversity of sensors, information assimilation - how to combine the information obtained from asynchronous and multifarious sources is an important and challenging research problem. In this paper, we propose a hierarchical probabilistic method for information assimilation in order to detect events of interest in a surveillance and monitoring environment. The proposed method adopts a bottom-up approach and performs assimilation of information at three different levels - media-stream level, atomic-event level and compound-event level.To detect an event, our method uses not only the current media streams but it also utilizes their two important properties - first, accumulated past history of whether they have been providing the concurring or contradictory evidences, and - second, the system designer's confidence in them. A compound event, which comprises of two or more atomic-events, is detected by first estimating probabilistic decisions for the atomic-events based on individual streams, and then by aligning these decisions along a timeline and hierarchically assimilating them. The experimental results show the utility of our method.

REFERENCES

• 1: CNN news, July 2005. http://www.cnn.com/2005/LAW/07/05/crime.prevention.ap/index.html.
• 2: Pradeep K. Atrey , Mohan S. Kankanhalli, Probability fusion for correlated multimedia streams, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027624]
• 3: P. K. Atrey and M. S. Kankanhalli. Goal based optimal selection of media streams. In IEEE International Conference on Multimedia and Expo, Amsterdam, The Netherlands, July 2005.
• 4: J. A. Benediktsson and I. Kanellopoulos. Classification of multisource and hyperspectral data based on decision fusion. IEEE Trans. on GeoScience and Remote Sensing, 37(3):1367--1377, May 1999.
• 5: D. A. Bloch and H. C. Kraemer. 2 x 2 Kappa coefficients: Measures of agreement or association. Journal of Biometrics, 45(1):269--287, 1989.
• 6: Z. Chair and P. Varshney. Optimal data fusion in multiple sensor detection systems. IEEE Transactions on Aerospace and Electronic Systems, 22:98--101, 1986.
• 7: N. Checka, K. W. Wilson, M. R. Siracusa, and T. Darrell. Multiple person and speaker activity tracking with a particle filter. In International Conference on Acoustics Speech and Signal Processing, Montreal, Canada, May 2004.
• 8: C. Genest and J. V. Zidek. Combining probability distributions: A critique and annotated bibliography. Journal of Statistical Science, 1(1):114--118, 1986.
• 9: J. Hershey, H. Attias, N. Jojic, and T. Krisjianson. Audio visual graphical models for speech processing. In IEEE International Conference on Speech, Acoustics, and Signal Processing, Montreal, Canada, May 2004.
• 10: P. KaewTraKulPong and R. Bowden. An improved adaptive background mixture model for real-time tracking with shadow detection. In European Workshop on Advanced Video Based Surveillance Systems, London, UK, September 2001.
• 11: M. Kam, Q. Zhu, and W. Gray. Optimal data fusion of correlated local decisions in multiple sensor detection systems. IEEE Transactions on Aerospace and Electronic Systems, 28(3):916--920, July 1992.
• 12: L. I.-K. Lin. A concordance correlation coefficient to evaluate reproducibility. Journal of Biometrics, 45(1):255--268, 1989.
• 13: A. V. Nefian, L. Liang, X. Pi, X. Liu, and K. Murphy. Dynamic bayesian networks for audio-visual speech recognition. In EURASIP Journal on Applied Signal Processing, volume 11, pages 1274--1288, 2002.
• 14: H. J. Nock , G. Iyengar , C. Neti, Assessing face and speech consistency for monologue detection in video, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641070]
• 15: B. S. Rao and H. D. Whyte. A decentralized bayesian algorithm for identification of tracked objects. IEEE Transactions on Systems, Man and Cybernetics, 23(6):1683--1698, November-December 1993.
• 16: M. Siegel and H. Wu. Confidence fusion. In IEEE International Workshop on Robot Sensing, pages 96--99, Graz, Austria, May 2004.
• 17: C. Stauffer and W. E. L. Grimson. Adaptive background mixture models for real-time tracking. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, pages 252--258, Ft. Collins, CO, USA, 1999.
• 18: M. Valera and S. A. Velastin. Intelligent distributed surveillance systems: A review. IEE Proceedings on Visual Image Signal Processing, 152(2):192--204, April 2005.
• 19: Yi Wu , Edward Y. Chang , Kevin Chen-Chuan Chang , John R. Smith, Optimal multimodal fusion for multimedia data analysis, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027665]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Security

Keywords:
agreement coefficient, compound and atomic events, confidence fusion, event detection, information assimilation, multimedia surveillance