A semantic approach to the dynamic design of interaction controls in conversation systems

ABSTRACT

To support a full-fledged, multimedia human-computer conversation, we are building an intelligent framework, called Responsive Information Architect (RIA), which can automatically synthesize multimedia responses during the conversation. As part of its visual response generation, RIA dynamically creates context-sensitive interaction controls that are visual interfaces through which users can further interact with RIA. To enable the systematic design of inter¿action controls, we study and abstract control properties. In particular, in this paper we present a semantic model that captures the intentional, presentational, and behavioral char¿acteristics of interaction controls. Using this model, we show how to systematically sketch the semantics of an interaction control.

REFERENCES

• 1: M. C. Chuah , S. F. Roth, On the semantics of interactive visualizations, Proceedings of the 1996 IEEE Symposium on Information Visualization (INFOVIS '96), p.29, October 28-29, 1996
• 2: Mei C. Chuah , Steven F. Roth , Joe Mattis , John Kolojejchick, SDM: selective dynamic manipulation of visualizations, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.61-70, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215654]
• 3: Mark Derthick , Steven F. Roth, Example based generation of custom data analysis appliances, Proceedings of the 6th international conference on Intelligent user interfaces, p.57-64, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.360116]
• 4: James Foley , Won Chul Kim , Srdjan Kovacevic , Kevin Murray, Defining Interfaces at a High Level of Abstraction, IEEE Software, v.6 n.1, p.25-32, January 1989 [doi>10.1109/52.16899]
• 5: James D. Foley , Andries van Dam , Steven K. Feiner , John F. Hughes, Computer graphics (2nd ed. in C): principles and practice, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 6: Scott E. Hudson , Roger King, An Semantic Feedback in the Higgens UIMS, IEEE Transactions on Software Engineering, v.14 n.8, p.1188-1206, August 1988 [doi>10.1109/32.7628]
• 7: Robert J. K. Jacob , Leonidas Deligiannidis , Stephen Morrison, A software model and specification language for non-WIMP user interfaces, ACM Transactions on Computer-Human Interaction (TOCHI), v.6 n.1, p.1-46, March 1999 [doi>10.1145/310641.310642]
• 8: Stephan Kerpedjiev , Steven F. Roth, Mapping communicative goals into conceptual tasks to generate graphics in discourse, Proceedings of the 5th international conference on Intelligent user interfaces, p.157-164, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325814]
• 9: J. Mackinlay, S. Card, and G. Robertson. A semantic analysis of the design space of input devices. Human Computer Interaction, 5:145- 190, 1990.
• 10: Gary Marchionini, Information seeking in electronic environments, Cambridge University Press, New York, NY, 1995
• 11: George G. Robertson , Stuart K. Card , Jack D. Mackinlay, Information visualization using 3D interactive animation, Communications of the ACM, v.36 n.4, p.57-71, April 1993 [doi>10.1145/255950.153577]
• 12: Ben Shneiderman, Designing the user interface (2nd ed.): strategies for effective human-computer interaction, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1992
• 13: Maureen C. Stone , Ken Fishkin , Eric A. Bier, The movable filter as a user interface tool, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.306-312, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191774]
• 14: Michelle X. Zhou, Visual Planning: A Practical Approach to Automated Presentation Design, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.634-641, July 31-August 06, 1999
• 15: M. Zhou and S. Feiner. Efficiently planning coherent visual discourse. J. of Knowledge-Based Systems, 10(5):275-286, March 1998.
• 16: Michelle X. Zhou , Shimei Pan, Automated authoring of coherent multimedia discourse in conversation systems, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500241]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.2 Semantics of Programming Languages
          Subjects: Operational semantics

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis

General Terms:
Design, Human Factors, Performance, Theory

Keywords:
automated generation of interactive visual presen"tations, interaction controls, semantic analysis of interfaces