Designing annotation before it's needed

ABSTRACT

This paper considers the automated and semi-automated annotation of audiovisual media in a new type of production framework, A4SM (Authoring System for Syntactic, Semantic and Semiotic Modelling). We present the architecture of the framework and outline the underlying XML-Schema based content description structures of A4SM. We then describe tools for a news and demonstrate how video material can be annotated in real time and how this information can not only be used for retrieval but also can be used during the different phases of the production process itself. Finally, we discuss the pros and cons of our approach of evolving semantic networks as the basis for audio- visual content description.

REFERENCES

• 1: Thomas G. Aguierre Smith , Glorianna Davenport, The Stratification System - A Design Emvironment for Random Access, Proceedings of the Third International Workshop on Network and Operating System Support for Digital Audio and Video, p.250-261, November 12-13, 1992
• 2: Philippe Aigrain , Philippe Joly , Véronique Longueville, Medium knowledge-based macro-segmentation of video into sequences, Intelligent multimedia information retrieval, MIT Press, Cambridge, MA, 1997
• 3: Arnheim, R. (1956). Art and Visual Perception: A Psychology of the creative eye. London: Faber & Faber.
• 4: Bloch, G. R. (1986). Elements d'une Machine de Montage Pour I'Audio-Visuel. Ph.D., Ecole Nationale Superieure Des Telecommunications.
• 5: Bloom, P.J. (1985). High-quality digital audio in the entertainment industry: an overview of achievements and challenges, IEEE Acoust. Speech Signal Process. Mag.. 2, 2- 25 (1985)
• 6: Jan Borchers , Max Mühlhäuser, Design Patterns for Interactive Musical Systems, IEEE MultiMedia, v.5 n.3, p.36-46, July 1998 [doi>10.1109/93.713303]
• 7: Bordwell, D. (1989). Making Meaning - Inference and Rhetoric in the Interpretation of Cinema. Cambridge, Massachusetts: Harvard University Press.
• 8: Ronald J. Brachman , Hector J. Levesque, Readings in Knowledge Representation, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1985
• 9: Brooks, KM (1999). Metalinear Cinematic Narrative: Theory, Process, and Tool. MIT Ph.D. Thesis.
• 10: Carlo Colombo , Alberto Del Bimbo , Pietro Pala, Semantics in Visual Information Retrieval, IEEE MultiMedia, v.6 n.3, p.38-53, July 1999 [doi>10.1109/93.790610]
• 11: Marc Eliot Davis, Media streams: representing video for retrieval and repurposing, Massachusetts Institute of Technology, Cambridge, MA, 1995
• 12: Alberto Del Bimbo, Visual information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 13: Eco, U. (1977). A Theory of Semiotics. London: The Macmillan Press.
• 14: Fehlis, H. (I 999). Hybrides Trackingsystem fur virtuelle Studios. Fernseh- + Kinotechnik; Bd. 53, Nr. 5
• 15: Amarnath Gupta , Ramesh Jain, Visual information retrieval, Communications of the ACM, v.40 n.5, p.70-79, May 1997 [doi>10.1145/253769.253798]
• 16: Greimas, J. (1983). Structural Semantics: An Attempt at a Method. Lincoln: University of Nebraska Press.
• 17: Frank,G. Halasz, Reflections on NoteCards: seven issues for the next generation of hypermedia systems, Communications of the ACM, v.31 n.7, p.836-852, July 1988 [doi>10.1145/48511.48514]
• 18: Hirata, K. (1995). Towards Formalizing Jazz Piano Knowledge witha Deductive Object-Oriented Approach. Proceedings of Artificial intelligence and Music, IJCAI, pp. 77 - 80, Montreal.
• 19: Jane Hunter , Liz Armstrong, A comparison of schemas for video metadata representation, Proceeding of the eighth international conference on World Wide Web, p.1431-1451, May 1999, Toronto, Canada
• 20: Jane Hunter , Carl Lagoze, Combining RDF and XML schemas to enhance interoperability between metadata application profiles, Proceedings of the 10th international conference on World Wide Web, p.457-466, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372100]
• 21: IS0 MPEG-7(2001). Text of ISO/IEC FCD 15938-2 Information Technology - Multimedia Content Description Interface - Part 2 Description Definition Language, ISO/IEC JTC I/SC 29iWG 11 N4002, March 2001
• 22: ISO MPEG-7(2001). Tex to fISOiIEC 15938-5/FCD Information Technology - Multimedia Content Description Interface - Part 5 Multimedia Description Schemes, ISOilEC JTC l/SC 29/WG I1 N3966, March 2001
• 23: Johnson, S.E., Jourlin, P., Spgrk jones, K. 7 Woodland P.C. (2000). Audio Indexing and retrieval of Complete Broadcast News Shows. RIAO' 2000 Conference proceedings, Vol 2, pp. 1 163 - 1177, CollCge de France, Paris, France, April 12- 14 2000
• 24: Lindley, C. (2000). A Video Annotation Methodology for Interactive Video Sequence Generation, BCS Computer Graphics & Displays Group Conference on Digital Content Creation, Bradford, UK, 12-13 April 2000.
• 25: LemstrGm, K. & Tarhio, J. (2000). Searching Monophonic Patterns within Polyphonic Sources. RIAO' 2000 Conference proceedings, Vol 2, pp. 1163 - 1177, CollCge de France, Paris, France, April 12-14 2000
• 26: MPEG Requirements Group (2000). Overview of the MPEG-7 Standard (version4.0), Dot. ISO/MPEG N3752, MPEG La Baule Meeting, October 2000.
• 27: Melucci, M. & Orio, N. (2000). SMILE: a System for Content-based Musical Information Retrieval Environments. RIAO' 2000 Conference proceedings, Vol 2, pp. 1261 -1279, CollCge de France, Paris, France, April 12-14 2000
• 28: Mills, T.J., Pye, D., hollinghurst, N.J. & Wood, K.R. (2000). At&TV: Broadcast Television and Radio Retrieval. RIAO' 2000 Conference proceedings, Vol 2, pp. 1135 - 1144, CollCge de France, Paris, France, April 12-14 2000
• 29: Nack, F. (1996). "AUTEUR: The Application of Video Semantics and Theme Representation in Automated Video Editing," Ph.D., Lancaster University, 1996.
• 30: Nack, F. and Parkes, A. (1997). Towards the Automated Editing of Theme-Oriented Video Sequences. In Applied Artificial Intelligence (AAI) {Ed: Hiroaki Kitano}, Vol. 11, No. 4, pp. 33 1-366.
• 31: Nack, F. and A. Steinmetz (1998). Approaches on Intelligent Video Production, Proceedings of ECAI-98 Workshop on AI/Alife and Entertainment, August 24, 1998, Brighton.
• 32: Frank Nack , Adam T. Lindsay, Everything You Wanted to Know About MPEG-7: Part 1, IEEE MultiMedia, v.6 n.3, p.65-77, July 1999 [doi>10.1109/93.790612]
• 33: Nack, F. & C. Lindley (2000) Environments for the production and maintenance of interactive stories, Workshop on Digital Storytelling, Darmstadt, Germany, 15-16/6/2000.
• 34: Akio Nagasaka , Yuzuru Tanaka, Automatic Video Indexing and Full-Video Search for Object Appearances, Proceedings of the IFIP TC2/WG 2.6 Second Working Conference on Visual Database Systems II, p.113-127, September 30-October 03, 1991
• 35: Pachet, F. and Cazzly, D. (2000). A Taxonomy of Musical Genres. RIAO' 2000 Conference proceedings, Vol 2, pp. 1238 - 1245, CollCge de France, Paris, France, April 12-14 2000
• 36: Alan Philip Parkes, An artificial intelligence approach to the conceptual description of videodisc images, 1988
• 37: A. Parkes, Settings and the setting structure: the description and automated propagation of networks for perusing videodisk image states, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.229-238, June 25-28, 1989, Cambridge, Massachusetts, United States
• 38: Peirce, C. S. (1960). The Collected Papers of Charles Sanders Peirce - I Principles of Philosophy and 2 Elements of Logic, Edited by Charles Hartshone and Paul Weiss. Cambridge, MA: The Belknap Press of Harvard University Press.
• 39: Silvia Pfeiffer , Stephan Fischer , Wolfgang Effelsberg, Automatic audio content analysis, Proceedings of the fourth ACM international conference on Multimedia, p.21-30, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244139]
• 40: Robertson, J., De Quincey, A., Stapleford T. & Wiggins, G. (1998). Real-Time Music Generation for a Virtual Environment. Proceedings of ECAI-98 Workshop on AI/Alife and Entertainment, August 24, 1998, Brighton.
• 41: Sack, W. (1993). Coding News And Popular Culture. In The International Joint Conference on Artificial Intelligence (IJCA93) Workshop on Models of Teaching and Models of Learning. Chambery, Savoie, France.
• 42: Simone Santini , Ramesh Jain, Integrated Browsing and Querying for Image Databases, IEEE MultiMedia, v.7 n.3, p.26-39, July 2000 [doi>10.1109/93.879766]
• 43: SMPTE (1999). Dynamic Data Dictonary Structure, 6. Draft, September 1999.
• 44: J. F. Sowa, Conceptual structures: information processing in mind and machine, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1984
• 45: TALC (1999). http://www.de.ibm.com/ide/solutions/dmsc/
• 46: Yoshinobu Tonomura , Akihito Akutsu , Yukinobu Taniguchi , Gen Suzuki, Structured Video Computing, IEEE MultiMedia, v.1 n.3, p.34-43, September 1994 [doi>10.1109/MMUL.1994.318984]
• 47: Erling Wold , Thom Blum , Douglas Keislar , James Wheaton, Content-Based Classification, Search, and Retrieval of Audio, IEEE MultiMedia, v.3 n.3, p.27-36, September 1996 [doi>10.1109/93.556537]
• 48: XML Schema Part 0 (2000). Primer, W3C Candidate Recommendation, 24 October 2000, http://www.w3.org/TR/xmlschema-0/
• 49: XML Schema Part 1 (2000). Structures W3C Candidate Recommendation, 24 October 2000, htta://www.w3.orp/TK/xmlschema-1/
• 50: XML Schema Part 2 (2000). Datatypes W3C Candidate Recommendation, 24 October 2000, http//www.w3.org/TR/xmlschema-2/
• 51: Yeung, M. M., Yeo, B., Wolf, W. & Liu, B. (1995). Video Browsing using Clustering and Scene Transitions on Compressed Sequences. In Proceedings IS&T/SPIE '95 Multimedia Computing and Networking, San Jose. SPIE (2417), 399 - 413.
• 52: Zhang, II., Gong, Y., Smoliar, S. W. (1994). Automated parsing of news video. In IEEE International Conference on Multimedia Computing and Systems, (pp. 45 - 54). Boston: IEEE Computer Society Press.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Multimedia databases
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Design, Experimentation, Performance

Keywords:
MPEG-7, XML Schema, automated annotation, news production, semantic networks