Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

To aid users in exploring large and complex data sets, we are building an intelligent multimedia conversation system. Given a user request, our system dynamically creates a multimedia response that is tailored to the interaction context. In this paper, we focus on the problem of media allocation, a process that assigns one or more media, such as graphics or speech, to best convey the intended response content. Specifically, we develop a graph-matching approach to media allocation, whose goal is to find a set of data-media mappings that maximizes the satisfaction of various allocation constraints (e.g., data-media compatibility and presentation consistency constraints). Compared to existing rule-based or plan-based approaches to media allocation, our work offers three unique contributions. First, we provide an extensible computational framework that optimizes media assignments by dynamically balancing all relevant constraints. Second, we use feature-based metrics to uniformly model various allocation constraints, including those cross-content and cross-media constraints, which often require special treatment in existing approaches. Third, we further improve the quality of a response by automatically detecting and repairing undesired allocation results. We have applied our approach to two different applications and our preliminary study has shown the promise of our work.

REFERENCES

• 1: Elisabeth André , Thomas Rist, Generating coherent presentations employing textual and visual material, Artificial Intelligence Review, v.9 n.2-3, p.147-165, June 1995 [doi>10.1007/BF00849177]
• 2: Y. Arens, E. Hovy, and S. van Mulken. Structure and rules in automated multimedia presentation planning. In Proc. IJCAI'93, pages 1253--1259, 1993.
• 3: Yigal Arens , Eduard Hovy , Mira Vossers, On the knowledge underlying multimedia presentations, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 4: Context-Based Multimodal Input Understanding in Conversational Systems, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.87, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166974]
• 5: Christian Elting , Jan Zwickel , Rainer Malaka, Device-dependant modality selection for user-interfaces: an empirical study, Proceedings of the 7th international conference on Intelligent user interfaces, January 13-16, 2002, San Francisco, California, USA [doi>10.1145/502716.502728]
• 6: Peter Faraday , Alistair Sutcliffe, Designing effective multimedia presentations, Proceedings of the SIGCHI conference on Human factors in computing systems, p.272-278, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258753]
• 7: Steven K. Feiner , Kathleen R. McKeown, Automating the Generation of Coordinated Multimedia Explanations, Computer, v.24 n.10, p.33-41, October 1991 [doi>10.1109/2.97249]
• 8: H. Gardner. Multiple Intelligences: The Theory in Practice. Basic Books, 1993.
• 9: Steven Gold , Anand Rangarajan, A Graduated Assignment Algorithm for Graph Matching, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.4, p.377-388, April 1996 [doi>10.1109/34.491619]
• 10: C. Karagiannidis, A. Koumpis, and C. Stephanidis. Decision making in intelligent. Computer Standards and Interfaces: Special Issue on Intellitent Multimedia Presentation, 18(6), 1997.
• 11: Stephan Kerpedjiev , Giuseppe Carenini , Steven F. Roth , Johanna D. Moore, Integrating planning and task-based design for multimedia presentation, Proceedings of the 2nd international conference on Intelligent user interfaces, p.145-152, January 06-09, 1997, Orlando, Florida, United States [doi>10.1145/238218.238316]
• 12: Srdjan Kovačevic, A compositional model of human-computer dialogues, Multimedia interface design, ACM Press, New York, NY, 1992 [doi>10.1145/146022.146102]
• 13: Mark T. Maybury, Planning multimedia explanations using communicative acts, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 14: K. McKeown, S. Feiner, J. Robin, D. Seligmann, and M. Tanenblatt. Generating cross-references for multimedia explanation. In Proc. AAAI '92, pages 12--17, 1992.
• 15: J. Miller. Time course of coactivation in bimodal divided attention. Perception and Psychophysics, 40(5):331--343, 1986.
• 16: K. Mullet and D. Sano. Designing Visual Interfaces. SunSoft Press, 1995.
• 17: S. Pan. A multi-layer conversation management approach for information-seeking applications. In Proc. ICSLP '04, 2004. To appear.
• 18: S. Pan and J. Shaw. SEGUE: A hybrid case-based natural language generator. In Proc. INLG '04, pages 130--140, 2004.
• 19: Christopher P. Robinson , Ray E. Eberts, Comparison of speech and pictorial displays in a cockpit environment, Human Factors, v.29 n.1, p.31-44, Feb. 1987
• 20: Steven F. Roth , William E Hefley, Intelligent multimedia presentation systems: research and principles, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 21: Alistair Sutcliffe , Peter Faraday, Designing presentation in multimedia interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.92-98, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191710]
• 22: Louis Weitzman , Kent Wittenburg, Grammar-based articulation for multimedia document design, Multimedia Systems, v.4 n.3, p.99-111, June 1996 [doi>10.1007/s005300050015]
• 23: C. Wickens. Processing resources in attention. In R. Pararsuraman and D. Davis, editors, Varieties of Attention. Academic Press, 1983.
• 24: C. Wickens, D. Sandry, and M. Vidulich. Compatibility and resource competition between modalities of input, central processing and output. Human Factors, 25:227--248, 1983.
• 25: Michelle X. Zhou , Vikram Aggarwal, An optimization-based approach to dynamic data content selection in intelligent multimedia interfaces, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029673]
• 26: M. Zhou and M. Chen. Automated generation of graphic sketches by examples. In Proc. IJCAI '03, pages 65--71, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications
          Subjects: Information browsers
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

General Terms:
Algorithms, Human Factors

Keywords:
automated generation of multimedia presentations, intelligent multimedia interfaces, media allocation