Dynamic pictorial ontologies for video digital libraries annotation

ABSTRACT

In this paper, we present the dynamic pictorial ontology paradigm for video annotation. Ontologies are often used to describe a given domain for different goals, including description of multimedia data. In the case of video annotation, the visual knowledge cannot be described using only abstract concepts but is more effectively represented in a visual form. To this aim, we introduce visual concepts, elicited from the data set as the most representative prototypes that specialize abstract concepts. The ontology created is intrinsically dynamic since it must embrace the perceptual and visual experience during annotation. Thus visual concepts can change, adapting to the multimedia content analyzed. Motivation for this new ontology paradigm are discussed together with a proposal of a framework for ontology creation, maintenance, and automatic annotation of video. The creation and usage of dynamic pictorial ontologies have been tested for soccer domain exploiting low level perceptual features and higher level domain features.

REFERENCES

• 1: S. Dasiopoulou, V. K. Papastathis, V. Mezaris, I. Kompatsiaris, M. G. Strintzis, "An Ontology Framework For Knowledge-Assisted Semantic Video Analysis and Annotation", Proc.of 4th International Workshop on SemAnnot at the 3rd International Semantic Web Conference, Hiroshima, Japan, 2004.
• 2: S. Dasiopoulou, V. Mezaris, I. Kompatsiaris, V. K. Papastathis M. G. Strintzis, "Knowledge-Assisted Semantic Video Object Detection", IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on Analysis and Understanding for Video Adaptation, vol. 15, no. 10, pp. 1210--1224, October 2005
• 3: C. Tsinaraki, P. Polydoros, S. Christodoulakis, "Interoperability support for Ontology-based Video Retrieval Applications", Proc of Conference on Image and Video Retrieval (CIVR), 2004
• 4: A. Jaimes , J. R. Smith, Semi-automatic, data-driven construction of multimedia ontologies, Proceedings of the 2003 International Conference on Multimedia and Expo, p.781-784, July 06-09, 2003
• 5: T. h. Athanasiadis, V. Tzouvaras, K. Petridis, F. Precioso, Y. Avrithis Y. Kompatsiaris, "Using a Multimedia Ontology Infrastructure for Semantic Annotation of Multimedia Content", Proc. of 5th International Workshop on Knowledge Markup and Semantic Annotation (SemAnnot '05), Galway, Ireland, November 2005
• 6: "Towards a common multimedia ontology framework" analysis of the contribution 27/4/06 report on ACEMedia project
• 7: "Ontological and Epistemological Foundations", report on DELOS project http://www.idi.ntnu.no/grupper/su/publ/html/totland/ch032.htm
• 8: Rudy Hirschheim , Heinz K. Klein , Kalle Lyytinen, Information systems development and data modeling: conceptual and philosophical foundations, Cambridge University Press, New York, NY, 1995
• 9: E. G. Guba, Y. S. Lincoln, "Competing Paradigms in Qualitative Research", in (Denzin and Lincoln, 1994), pp. 105--117, 1994
• 10: Ludwig Wittgenstein , David Francis Pears , Brian McGuinness , Bertrand Russell, Tractatus Logico Philosophicus (Routledge Classics) (Routledge Classics), Routledge, New York, NY, 2001
• 11: Y. Zhai, J. Liu, X. Cao, A. Basharat, A. Hakeem, S. Ali, M. Shah, C. Grana, R.Cucchiara, "Video Understanding and Content-based Retrieval", TREC Video Retrieval Evaluation Online Proceedings, http://wwwnlpir.nist.gov/projects/tvpubs/tv.pubs.org.html, 2005
• 12: C. Grana, R. Cucchiara, "Linear Transition Detection as a Unified Shot Detection Approach" in IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, n. 4, pp. 483--489, 2007
• 13: M. Bertini, R. Cucchiara, A. Del Bimbo C. Torniai, "Video Annotation with Pictorially Enriched Ontologies". In Proc. of International Conference on Multimedia and Expo (ICME), 2005.
• 14: Costantino Grana , Roberto Vezzani , Rita Cucchiara, Enhancing HSV histograms with achromatic points detection for video retrieval, Proceedings of the 6th ACM international conference on Image and video retrieval, p.302-308, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282327]
• 15: Ding Zhou , Jia Li , Hongyuan Zha, A new Mallows distance based metric for comparing clusterings, Proceedings of the 22nd international conference on Machine learning, p.1028-1035, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102481]
• 16: Marco Bertini , Alberto Del Bimbo , Carlo Torniai, Automatic video annotation using ontologies extended with visual information, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101235]
• 17: James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Kluwer Academic Publishers, Norwell, MA, 1981
• 18: A. Jaimes, B. Tseng, J. Smith, "Modal keywords, Ontologies, and Reasoning for Video Understanding" in Proc. of Conference on Image and Video Retrieval (CIVR), July 2003
• 19: A. Benitez, S.-F. Chang, "Automatic Multimedia Knowledge Discovery, Summarization and Evaluation" IEEE Transactions on Multimedia, Submitted
• 20: John R. Kender , Milind R. Naphade, Visual Concepts for News Story Tracking: Analyzing and Exploiting the NIST TRECVID Video Annotation Experiment, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.1174-1181, June 20-26, 2005 [doi>10.1109/CVPR.2005.371]
• 21: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 22: Douglas B. Lenat , R. V. Guha, Building Large Knowledge-Based Systems; Representation and Inference in the Cyc Project, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 23: J. Strintzis, S. Bloehdorn, S. Handschuh, S. Staab, N. Simou, V. Tzouvaras, K. Petridis, I. Kompatsiaris, Y. Avrithis, "Knowledge representation for semantic multimedia content analysis and reasoning," in Proc. of European Workshop on the Integration of Knowledge, Semantics and Digital Media Technology, Nov. 2004.
• 24: S. Vembu, M. Kiesel, M. Sintek, S. Bauman, "Towards bridging the semantic gap in multimedia annotation and retrieval," in Proc. of First International Workshop on Semantic Web Annotations for Multimedia (SWAMM), Edinburgh (Scotland), May 2006.
• 25: V. Mezaris, I. Kompatsiaris, N. Boulgouris, M. Strintzis, "Real-time compressed-domain spatiotemporal segmentation and ontologies for video indexing and retrieval," IEEE Transactions on Circuits and Systems for Video Technology, vol. 14,no. 5, pp. 606--621, 2004.
• 26: N. Simou, C. Saathoff, S. Dasiopoulou, E. Spyrou, N. Voisine, V. Tzouvaras, I. Kompatsiaris, Y. Avrithis, S. Staab, "An ontology infrastructure for multimedia reasoning," in Proc. International Workshop VLBV, Sardinia (Italy), September, 2005.
• 27: D. Kosmopoulos, S. Petridis, I. Pratikakis, V. Gatos, S. Perantonis, V. Karkaletsis, G. Paliouras, "Knowledge Acquisition from Multimedia Content using an Evolution Framework," in Proc. of 3rd IFIP Conference on Artificial Intelligence Applications & Innovations (AIAI), June, 2006.
• 28: K. Petridis, S. Bloehdorn, C. Saathoff, N. Simou, S. Dasiopoulou, V. Tzouvaras, S. Handschuh, Y. Avrithis, I. Kompatsiaris, S. Staab, "Knowledge Representation and Semantic Annotation of Multimedia Content," IEE Proceedings on Vision Image and Signal Processing, Special issue on Knowledge-based Digital Media Processing, vol. 153, no. 3, pp. 255--262, June 2006.
• 29: C. Snoek, B. Huurnink, L. Hollink, M. de Rijke, G. Schreiber, M. Worring, "Adding semantics to detectors for video retrieval," IEEE Transactions on Multimedia, vol. 9, no. 5, August, 2007.
• 30: C. Grana, D. Bulgarelli, R. Cucchiara, "Video clip clustering for assisted creation of mpeg-7 pictorially enriched ontologies," in Proc. Second International Symposium on Communications, Control and Signal Processing (ISCCSP), Marrakech, Morocco, March 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases

General Terms:
Algorithms

Keywords:
multimedia extraction and annotation, multimedia ontologies, video annotation, video retrieval