High-level authoring of illustrated documents

ABSTRACT

This paper starts by introducing a class of future document authoring systems that will allow authors to specify the content and form of a text+pictures document at a high level of abstraction, while leaving responsibility for linguistic and graphical details to the system. Next, we describe two working prototypes that implement parts of this functionality, based on semantic modeling of the pictures and the text of the document; one of these two, the ILLUSTRATE prototype, is a multimedia extension of previous text authoring systems in the What You See Is What You Meant (WYSIWYM) tradition. The paper concludes with an exploration of the ways in which Multimedia WYSIWYM can be further enhanced, allowing it to approximate the ‘ideal’ systems that were sketched earlier in the paper. Applications of Multimedia WYSIWYM to general-purpose picture retrieval (in the context of the Semantic Web, for example) are also discussed.

REFERENCES

• 1: ABPI (1997) The Association of the British Pharmaceutical Industry, 1996-1997 Compendium of Patient Information Leaflets.
• 2: Elisabeth André , Thomas Rist, Generating coherent presentations employing textual and visual material, Artificial Intelligence Review, v.9 n.2-3, p.147-165, June 1995 [doi>10.1007/BF00849177]
• 3: Barker-Plummer, D. and Greeves, M. (1995) Architectures for heterogeneous reasoning. In: Lee, J. (ed.), Proc. First International Workshop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications (IMMI-1), Edinburgh, UK.
• 4: Bateman, J., Delin, J. and Allen, P. (2000) Constraints on layout in multimodal document generation. Proc. First Int. Conf. on Natural Language Generation, Workshop on Coherence in Generated Multimedia. Mitzpe Ramon, Israel.
• 5: John Bateman , Jörg Kleinz , Thomas Kamps , Klaus Reichenberger, Towards constructive text, diagram, and layout generation for information presentation, Computational Linguistics, v.27 n.3, p.409-449, September 2001 [doi>10.1162/089120101317066131]
• 6: Berners-Lee, T., Hendler, J. and Lassila, O. (2001) A new form of Web content that is meaningful to computers will unleash a revolution of new possibilities. Sci. Am. May.
• 7: M. Bordegoni , G. Faconti , S. Feiner , M. T. Maybury , T. Rist , S. Ruggieri , P. Trahanias , M. Wilson, A standard reference model for intelligent multimedia presentation systems, Computer Standards & Interfaces, v.18 n.6-7, p.477-496, Dec. 1997 [doi>10.1016/S0920-5489(97)00013-5]
• 8: Bouayad-Agha, N., Power, N., Scott, D. and Belz, A. (2002) PILLS: Multilingual generation of medical information documents with overlapping content. Proc. LREC 2002, pp. 2111- 2114.
• 9: Caroline Brun , Marc Dymetman , Veronika Lux, Document structure and multilingual authoring, Proceedings of the first international conference on Natural language generation, June 12-16, 2000, Mitzpe Ramon, Israel [doi>10.3115/1118253.1118258]
• 10: Lynne Cahill , John Carroll , Roger Evans , Daniel Paiva , Richard Power , Donia Scott , Kees van Deemter, From RAGS to RICHES: exploiting the potential of a flexible generation architecture, Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, p.106-113, July 06-11, 2001, Toulouse, France [doi>10.3115/1073012.1073027]
• 11: CLIM (1994) Common Lisp Interface Manager: User Guide. Franz Inc.
• 12: Dale, R. and Reiter, E. (1995) Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Sci. 18: 233-263.
• 13: Enser, P. (1995) Progress in documentation; pictorial information retrieval. J. Documentation 51(2): 126-170.
• 14: Gregor Erbach, ProFIT: prolog with features, inheritance and templates, Proceedings of the seventh conference on European chapter of the Association for Computational Linguistics, March 27-31, 1995, Dublin, Ireland [doi>10.3115/976973.976999]
• 15: ETAI (1997, 1998) ETAI News Journal on Intelligent User Interfaces 1 (1-2).
• 16: Steven K. Feiner , Kathleen R. McKeown, Automating the generation of coordinated multimedia explanations, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 17: Gentner, D. (1992) Interfaces for learning: motivation and the locus of control. In: Engel, F. L., Bouwhuis, D. G., Boesser, T. and d'Ydewalle, G., editors, Cognitive Modelling and Interactive Environments in Language Learning. NATO ASI series Vol. F87. Springer-Verlag.
• 18: Graf, W. H., Neurohr, S. and Goebel, R. (1996) A constraint-based tool for the pagination of Yellow-Page directories. In: Geske, U. and Simonis, H., editors, Proc. KI96 Workshop on Declarative Constraint Programming. GMD-Studien 297, St. Augustin, Germany.
• 19: Kamp, K. and Reyle, U. (1993) From Discourse to Logic. Kluwer Academic.
• 20: Stephan Kerpedjiev , Steven F. Roth, Mapping communicative goals into conceptual tasks to generate graphics in discourse, Proceedings of the 5th international conference on Intelligent user interfaces, p.157-164, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325814]
• 21: H J Levesque, Making believers out of computers, Artificial Intelligence, v.30 n.1, p.81-108, Oct. 1986 [doi>10.1016/0004-3702(86)90068-8]
• 22: Masthoff, J. (2000) Analytical Usability Evaluations of WYSIWYM. ITRI internal report (http://www.itri.bton.ac.uk/research.html#WYSIWYM).
• 23: Mark T. Maybury, Planning multimedia explanations using communicative acts, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 24: Mark T. Maybury , Wolfgang Wahlster, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 25: McGregor, R. and Bates, R. (1987) The LOOM knowledge representation language. Proc. Knowledge-Based Systems Workshop, St. Louis, MI.
• 26: McKevitt, P. (ed.) Integration of Natural Language and Vision Processing: Intelligent Multimedia. Artif. Intelli. Rev. 9 (2-3).
• 27: Vibhu O. Mittal , Giuseppe Carenini , Johanna D. Moore , Steven Roth, Describing complex charts in natural language: a caption generation system, Computational Linguistics, v.24 n.3, September 1998
• 28: Paraboni, I. and van Deemter, K. (2002) Towards the generation of document-deictic references. In: van Deemter, K. and Kibble, R., editors, Information Sharing: Reference and Presupposition in Language Generation and Interpretation, CSLI Publications, Stanford.
• 29: Pineda, L. (2000) Spatial language, deixis and illustration. Proc. Workshop "Coherence in Generated Multimedia", Mitzpe Ramon, Israel.
• 30: Paul Piwek, A formal semantics for generating and editing plurals, Proceedings of the 18th conference on Computational linguistics, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/992730.992734]
• 31: Piwek, P., Evans, R., Cahill, L. and Tipper, N. (2000) Natural language generation in the MILE system. Proc. 'IMPACTS in NLG' Workshop, Schloss Dagstuhl, Germany.
• 32: Richard Power , Donia Scott, Multilingual authoring using feedback texts, Proceedings of the 17th international conference on Computational linguistics, p.1053-1059, August 10-14, 1998, Montreal, Quebec, Canada
• 33: Steven F. Roth , William E Hefley, Intelligent multimedia presentation systems: research and principles, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 34: A. Th. (Guus) Schreiber , Barbara Dubbeldam , Jan Wielemaker , Bob Wielinga, Ontology-Based Photo Annotation, IEEE Intelligent Systems, v.16 n.3, p.66-74, May 2001 [doi>10.1109/5254.940028]
• 35: Scott, D., Power, R. and Evans, R. (1998) Generation as a solution to its own problem. Proc. 9th International Workshop on Natural Language Generation. Niagara-on-the-Lake, Canada, pp. 256-265.
• 36: Donia Scott, The Multilingual Generation Game: Authoring Fluent Texts in Unfamiliar Languages, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.1407-1411, July 31-August 06, 1999
• 37: Oliviero Stock, Natural language and exploration of an information space: the AlFresco interactive systems, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 38: van Deemter, K. (1998) Retrieving pictures for document generation. Proc. 14th Twente Workshop on Language Technology, on Multimedia Information Retrieval (TWLT14), Twente, The Netherlands.
• 39: Kees van Deemter, Document Generation and Picture Retrieval, Proceedings of the Third International Conference on Visual Information and Information Systems, p.632-640, June 02-04, 1999
• 40: Kees van Deemter , Richard Power, Inclusion of Picture Sequences in Generated Documents, Proceedings of the 9th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, p.99-112, September 21-24, 1999
• 41: Kees van Deemter , Richard Power, Authoring multimedia documents using WYSIWYM editing, Proceedings of the 18th conference on Computational linguistics, p.222-228, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990853]
• 42: van de Waal, H. (1995) ICONCLASS; An iconographic classification system. Amsterdam 1973- 1985 (17 vols).
• 43: C. J. van Rijsbergen, Towards an information logic, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.77-86, June 25-28, 1989, Cambridge, Massachusetts, United States
• 44: Venters, C. and Cooper, M. (2000) A Review of Content-Based Image Retrieval Systems. http://www.jtap.ac.uk/reports/htm/jtap-054.html.
• 45: Wolfgang Wahlster , Elisabeth André , Wolfgang Finkler , Hans-Jürgen Profitlich , Thomas Rist, Plan-based integration of natural language and graphics generation, Artificial Intelligence, v.63 n.1-2, p.387-427, Oct. 1993 [doi>10.1016/0004-3702(93)90022-4]
• 46: Michelle X. Zhou , Steven Feiner, IMPROVISE: Automated Generation of Animated Graphics for Coordinated Multimedia Presentations, Revised Papers from the Second International Conference on Cooperative Multimodal Communication, p.43-63, January 28-30, 1998