Multimodal observation systems

ABSTRACT

In recent years, we have seen a significant research interest in a number of multimodal sensing applications like surveillance, video ethnography, tele-presence, assisted living, life blogging etc. However, these applications are currently evolving as separate silos with no interconnection. Further, the individual application-centric architectures typically tend to focus on specific sensors, specific (hardwired) queries and deal with specific environments. We present a generic sensing architecture 'Observation System', which allows multiple users to undertake different applications through abstracted interaction with a common set of sensors. The observation system observes behavior of various objects in an environment and keeps a record of important events and activities in an eventbase. In this system, multifarious data collected from disparate sensors and other sources are correlated to understand and gain insights in the environment. The observation system has applications in many areas including but not limited to surveillance, traffic monitoring, ethnography, marketing, and healthcare. In this paper, we present the architecture and functionality of such a system and present details of activity detection using multiple sensor streams in a distributed sensing environment. We also present results of such an approach and potential extensions to the analysis of more complex activities and events.

REFERENCES

• 1: Philippe Bonnet , Johannes Gehrke , Praveen Seshadri, Towards Sensor Database Systems, Proceedings of the Second International Conference on Mobile Data Management, p.3-14, January 08-10, 2001
• 2: O. Gnawali, R. Govindan, and J. S. Heidemann:. Implementing a sensor database system using a generic data Figure 7: Telepresence View dissemination mechanism. IEEE Data Enginering Bulletin, 28(1):70--75, 2005.
• 3: Forouzan Golshani , Nevenka Dimitrova, A Language for Content-Based Video Retrieval, Multimedia Tools and Applications, v.6 n.3, p.289-312, May 1998 [doi>10.1023/A:1009612532460]
• 4: A. Hampapur, S. Borger, L. Brown, C. Carlson, J. Connell, M. Lu, A. Senior, V. Reddy, C. Shu, and Y. Tian. S3: The ibm smart surveillance system: From transactional systems to observational systems. In Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. IEEE International Conference on, 2007.
• 5: R. Jain. Out-of-the-box data engineering - events in eterogeneous data environments. In International Conference on Data Engineering, pages 8--21, 2003.
• 6: Ramesh Jain , Pilho Kim , Zhao Li, Experiential meeting system, Proceedings of the 2003 ACM SIGMM workshop on Experiential telepresence, November 07, 2003, Berkeley, California [doi>10.1145/982484.982486]
• 7: Simon Moncrieff , Svetha Venkatesh , Geoff West , Stewart Greenhill, Multi-modal emotive computing in a smart house environment, Pervasive and Mobile Computing, v.3 n.2, p.74-94, March, 2007 [doi>10.1016/j.pmcj.2006.07.003]
• 8: E. Ochs, A. Graesch, A. Mittmann, T. Bradbury, and R. Repetti. Video ethnography and ethnoarchaeological tracking. The Work and Family Handbook: Multi-Disciplinary Perspective, Methods, and Approaches, pages 387--409, 2006.
• 9: Conrad Parker , Silvia Pfeiffer, Video Blogging: Content to the Max, IEEE MultiMedia, v.12 n.2, p.4-8, April 2005 [doi>10.1109/MMUL.2005.41]
• 10: E. Saykol, U. Gĺudĺukbay, and O. Ulusoy. A database model for querying visual surveillance by integrating semantic and low-level features. In International Workshop on Multimedia Information Systems, pages 163--176, 2005.
• 11: M. Valera and S. Velastin. Intelligent distributed surveillance systems: a review. In IEE Proceedings on Vision, Image and Signal Processing, pages 192--204, 2005.
• 12: Utz Westermann , Ramesh Jain, {\rm E} - A Generic Event Model for Event-Centric Multimedia Data Management in eChronicle Applications, Proceedings of the 22nd International Conference on Data Engineering Workshops, p.106, April 03-07, 2006 [doi>10.1109/ICDEW.2006.1]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

General Terms:
Design, Management, Performance, Security

Keywords:
application, events, query, surveillance, system