Automated authoring of coherent multimedia discourse in conversation systems

ABSTRACT

We are building a full-fledged multimedia conversation framework called Responsive Information Architect (RIA), using a combination of AI and multimedia techniques. Here we describe RIA's capability of automated authoring of a coherent multimedia discourse, which is used by RIA to express itself when conversing with a user. Specifically, we focus on explaining three unique features of our automated authoring approach: automated authoring of multimedia inter¿action acts, dynamic insertion of multimedia punctuation acts, and systematic design of cross-media acts.

REFERENCES

• 1: D. Arijon. Grammar of the Film Language. Silman-James Press, 1976.
• 2: M. Bordegoni , G. Faconti , S. Feiner , M. T. Maybury , T. Rist , S. Ruggieri , P. Trahanias , M. Wilson, A standard reference model for intelligent multimedia presentation systems, Computer Standards & Interfaces, v.18 n.6-7, p.477-496, Dec. 1997 [doi>10.1016/S0920-5489(97)00013-5]
• 3: Isabel F. Cruz , Wendy T. Lucas, A visual approach to multimedia querying and presentation, Proceedings of the fifth ACM international conference on Multimedia, p.109-120, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266346]
• 4: Mukesh Dalal , Steven Feiner , Kathleen McKeown , Shimei Pan , Michelle Zhou , Tobias Höllerer , James Shaw , Yong Feng , Jeanne Fromer, Negotiation for automated generation of temporal multimedia presentations, Proceedings of the fourth ACM international conference on Multimedia, p.55-64, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244147]
• 5: Pete Faraday , Alistair Sutcliffe, An empirical study of attending and comprehending multimedia presentations, Proceedings of the fourth ACM international conference on Multimedia, p.265-275, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244225]
• 6: Steven K. Feiner , Kathleen R. McKeown, Automating the Generation of Coordinated Multimedia Explanations, Computer, v.24 n.10, p.33-41, October 1991 [doi>10.1109/2.97249]
• 7: Won Chul Kim , James D. Foley, Providing high-level control and expert assistance in the user interface presentation design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.430-437, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169346]
• 8: J. Levin, G. Anglin, and R. Carney. On empirically validating functions of pictures in prose. In D. Willows and H. Houghton, editors, The Psychology of Illustration: Basic Research, volume 1, chapter 2, pages 5 l-86. Springer-Verlag, New York, 1987.
• 9: Mark T. Maybury , Wolfgang Wahlster, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 10: K. McKeown, S. Feiner, J. Robin, D. Seligmann, and M. Tanenblatt. Generating cross-references for multimedia explanation. In Proc. AAAI '92, pages 12-17, 1992.
• 11: K. McKeown and S. Pan. Prosody modeling in concept-to-speech generation: Methodological issues. Philosophical Trunsacrions of the Royal Society, Series A, 358( 1769), 2000.
• 12: S. Pan and K. McKeown. Spoken language generation in a multimedia system. In Proc. ICSLP '96, pages 374-377, 1996.
• 13: Lloyd Rutledge , Jacco van Ossenbruggen , Lynda Hardman , Dick C. A. Bulterman, A framework for generating adaptable hypermedia documents, Proceedings of the fifth ACM international conference on Multimedia, p.121-130, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266348]
• 14: Stuart G. Towns , Charles B. Callaway , James C. Lester, Generating coordinated natural language and 3D animations for complex spatial explanations, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.112-119, July 1998, Madison, Wisconsin, United States
• 15: Michelle X. Zhou, Visual Planning: A Practical Approach to Automated Presentation Design, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.634-641, July 31-August 06, 1999
• 16: M. Zhou and S. Ma. Representing and retrieving visual presentations for example-based graphics generation. In Proc. 1st Intl. Syp. on Smart Graphics 2001, pages 87-94, 2001.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.2 Document Preparation
          Subjects: Multi/mixed media

General Terms:
Design, Performance, Theory

Keywords:
automated generation of multimedia discourse, graphics generation, multimedia authoring, multimedia conversation/dialogue, spoken language generation