Human action analysis, annotation and modeling in video streams based on implicit user interaction

ABSTRACT

This paper proposes an integrated framework for analyzing human actions in video streams. Despite most current approaches that are just based on automatic spatiotemporal analysis of sequences, the proposed method introduces the implicit user-in-the-loop concept for dynamically mining semantics and annotating video streams. This work sets a new and ambitious goal: to recognize, model and properly use "average user's" selections, preferences and perception, for dynamically extracting content semantics. The proposed approach is expected to add significant value to hundreds of billions of non-annotated or inadequately annotated video streams existing in the Web, file servers, databases etc. Furthermore expert annotators can gain important knowledge relevant to user preferences, selections, styles of searching and perception.

REFERENCES

• 1: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 2: Arnab Bhattacharya , Vebjorn Ljosa , Jia-Yu Pan , Mark R. Verardo , Hyungjeong Yang , Christos Faloutsos , Ambuj K. Singh, ViVo: Visual Vocabulary Construction for Mining Biomedical Images, Proceedings of the Fifth IEEE International Conference on Data Mining, p.50-57, November 27-30, 2005 [doi>10.1109/ICDM.2005.151]
• 3: Jia Li , James Z. Wang, Real-time computerized annotation of pictures, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180841]
• 4: Dhiraj Joshi , James Z. Wang , Jia Li, The Story Picturing Engine---a system for automatic text illustration, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.68-89, February 2006 [doi>10.1145/1126004.1126008]
• 5: Stevenson, K., Leung, C., "Comparative evaluation of Web image search engines for multimedia applications," IEEE International Conference on Multimedia and Expo, July 2005.
• 6: Search Engine Statistics For 2006-07," SEO Weekly Article, www.accuracast.com.
• 7: comScore's qSearch 2.0 service", comScore's Report Article, www.comscore.com
• 8: Bernard J. Jansen , Amanda Spink , Tefko Saracevic, Real life, real users, and real needs: a study and analysis of user queries on the web, Information Processing and Management: an International Journal, v.36 n.2, p.207-227, Jan.1.2000 [doi>10.1016/S0306-4573(99)00056-4]
• 9: Yining Deng , b. s. Manjunath, Unsupervised Segmentation of Color-Texture Regions in Images and Video, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.8, p.800-810, August 2001 [doi>10.1109/34.946985]
• 10: N. Tsapatsoulis, C. Pattichis, A. Kounoudes, C. Loizou, A. Constantinides, J. G. Taylor "Visual Attention based Region of Interest Coding for Video-telephony Applications", 5th International Symposium on Communication Systems, Networks and Digital Signal Processing (CSNDSP'06), Patras, Greece, July 2006.
• 11: N. Tsapatsoulis, Y. Avrithis and S. Kollias, "Facial Image Indexing in Multimedia Databases," Pattern Analysis and Applications, Vol. 4, Issue 2/3, pp 93--107, 2001.
• 12: A. D. Doulamis, N. D. Doulamis and S. D. Kollias, "On Line Retrainable Neural Networks: Improving the Performance of Neural Network in Image Analysis problems," IEEE Trans. on Neural Networks, Vol. 11, No. 1, pp. 137--155, January 2000.
• 13: D. C. Park, M. A. EL-Sharkawi, and R. J. Marks II, "An Adaptively Trained Neural Network," IEEE Trans. on Neural Networks, vol. 2, pp. 334--345, 1991.
• 14: D. J. Luenberger. Linear and non Linear Programming. Addison-Wesley 1984.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Information filtering; Retrieval models; Selection process

General Terms:
Algorithms, Design, Theory

Keywords:
action modeling, human action analysis, human object detection, user transparent interaction, video annotation