Specification and dialogue control of visual interaction through visual rewriting systems

ABSTRACT

Computers are increasingly being seen not only as computing tools but more so as communication tools, thus placing special emphasis on human-computer interaction (HCI). In this article, the focus is on visual HCI, where the messages exchanged between human and computer are images appearing on the computer screen, as usual in current popular user interfaces. We formalize interactive sessions of a human-computer dialogue as a structured set of legal visual sentences, i.e., as a visual language, and show how rewriting systems can be generalized to specify both the pictorial and the computational aspects of visual languages. To this end, Visual Conditional Attributed Rewriting (VCARW) systems are introduced, and use for specification of visual languages. These specifications are given as inputs to a procedure illustrated in the article as a system of algorithms, which automatically generates control mechanisms of the interaction, thus favoring the design of more reliable and usable systems.

REFERENCES

• 1: ABOWD, G. AND BEALE, R. 1991. Users, systems and interfaces: A unifying framework for interaction. In Proceedings of HCI '91: People and Computers VI, D. Diaper and N. Hammond, Eds. 73-87.
• 2: Marc Andries , Gregor Engels , Jan Rekers, How to represent a visual specification, Visual language theory, Springer-Verlag New York, Inc., New York, NY, 1998
• 3: AREFI, F., MILANI, M., AND STARY, C. 1991. Towards customized user interface design environments. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 146-151.
• 4: BURNETT, M. AND AMBLER, A. 1994. Interactive visual data abstraction in a declarative visual programming language. J. Vis. Lang. Comput. 5, 26-60.
• 5: James C. Brancheau , Carol V. Brown, The management of end-user computing: status and directions, ACM Computing Surveys (CSUR), v.25 n.4, p.437-482, Dec. 1993 [doi>10.1145/162124.162138]
• 6: BIANCHI, N., BOTTONI, P., MussIo, P., AND PROTTI, M. 1993. Cooperative visual environments for the design of effective visual systems. J. Vis. Lang. Comput. 4, 357-382.
• 7: BIANCHI, N., BOTTONI, P., SPINU, C., GARBAY, C., AND MUSSIO, P. 1998. Situated image understanding in a multi-agent framework. Int. J. Pattern Recog. Art. Intell. 12, 5, 595-624.
• 8: P. Bottoni , S. Levialdi , S.-K. Chang , M. F. Costabile , P. Mussio, On the Specification of Dynamic Visual Languages, Proceedings of the IEEE Symposium on Visual Languages, p.14, September 01-04, 1998
• 9: P. Bottoni , S. Levialdi , P. Mussio , M. F. Costabile, Visual Conditional Attributed Rewriting Systems in Visual Language Specification, Proceedings of the 1996 IEEE Symposium on Visual Languages, p.156, September 03-06, 1996
• 10: P. Bottoni , M. F. Costabile , S. Levialdi , P. Mussio, A visual approach to HCI, ACM SIGCHI Bulletin, v.28 n.3, p.50-55, July 1996 [doi>10.1145/231132.231145]
• 11: BOTTONI, P., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1997. Defining visual languages for interactive computing. IEEE Trans. Syst. Man. Cybernet. A-27, 6, 773-782.
• 12: Paolo Bottoni , Maria Francesca Costabile , Stefano Levialdi , Piero Mussio, Specification of visual languages as means for interaction, Visual language theory, Springer-Verlag New York, Inc., New York, NY, 1998
• 13: BOTTONI, P., COSTABILE, M. F., LEVIALDI, S., AND MUSSIO, P. 1998c. Specifying dialog control in visual interactive systems. J. Vis. Lang. Comput. 9, 5, 553-564.
• 14: P. Bottoni , P. Mussio , B. Olivieri , M. Protti, A completely visual environment for agent-based computing, Proceedings of the working conference on Advanced visual interfaces, May 24-27, 1998, L'Aquila, Italy [doi>10.1145/948496.948532]
• 15: John Tang Boyland, Conditional attribute grammars, ACM Transactions on Programming Languages and Systems (TOPLAS), v.18 n.1, p.73-108, Jan. 1996 [doi>10.1145/225540.225544]
• 16: Alan Carle , Lori Pollock, On the optimality of change propagation for incremental evaluation of hierarchical attribute grammars, ACM Transactions on Programming Languages and Systems (TOPLAS), v.18 n.1, p.16-29, Jan. 1996 [doi>10.1145/225540.225542]
• 17: CHANG, S. K. AND MUSSIO, P. 1996. Customized visual language design. In Proceedings of the 8th International Conference on Software Engineering and Knowledge Engineering (SEKE '96). 553-562.
• 18: CHANG, S. K., TAUBER, M. J., Yu, B., AND YU, J.S. 1988. The SIL-Icon compiler--An objectoriented system generator. Int. J. Patt. Recognit. Art. Intell. 2, 241-273.
• 19: CHANG, S. K., TORTORA, G., Yu, B., AND GUERICIO, A. 1987. Icon purity--Towards a formal definition of Icon. Int. J. Patt. Recognit. Art. Intell. 1, 377-392.
• 20: Sitt Sen Chok , K. Marriott, Automatic construction of user interfaces from constraint multiset grammars, Proceedings of the 11th International IEEE Symposium on Visual Languages, p.242, September 05-09, 1995
• 21: Gennaro Costagliola , Genoveffa Tortora , Sergio Orefice , Andrea De Lucia, Automatic Generation of Visual Programming Environments, Computer, v.28 n.3, p.56-66, March 1995 [doi>10.1109/2.366162]
• 22: Gennaro Costagliola , Andrea De Lucia , Sergio Orefice , Genoveffa Tortor, A Parsing Methodology for the Implementation of Visual Systems, IEEE Transactions on Software Engineering, v.23 n.12, p.777-799, December 1997 [doi>10.1109/32.637392]
• 23: Joëlle Coutaz, Architectural Design for User Interfaces, Proceedings of the 3rd European Software Engineering Conference, p.7-22, October 21-24, 1991
• 24: JFurgen Dassow , G. Paun, Regulated Rewriting in Formal Language Theory, Springer-Verlag New York, Inc., Secaucus, NJ, 1990
• 25: DIx, A. 1991. Formal Methods for Interactive Systems. Academic Press.
• 26: DUKE, K. J. AND HARRISON, M.D. 1995. Interaction and task requirements. In DSV-IS '95: Eurographics Workshop on Design, Specification and Verification of Interactive Systems, P. Palanque and R. Bastide, Eds. Springer, 54-75.
• 27: FACONTI, G. P. AND DUKE, D.J. 1996. Device models. In Design, Specification and Verification of Interactive Systems, F. Bodart and J. Vanderdonckt, Eds. Springer, 73-91.
• 28: James D. Foley , Richard L. Phillips , John F. Hughes , Andries van Dam , Steven K. Feiner, Introduction to Computer Graphics, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1994
• 29: GOLIN, E.J. 1991. Parsing visual languages with picture layout grammars. J. Vis. Lang. Comput. 2, 371-394.
• 30: GREEN, M. 1985. Report on dialogue specification tools. In User Interface Management Systems. Springer, 9-20.
• 31: Michael Harrison , Alan Dix, A state model of direct manipulation in interactive systems, Formal methods in human-computer interaction, Cambridge University Press, New York, NY, 1990
• 32: HELM, R. AND MARRIOTT, K. 1991. A declarative specification and semantics for visual languages. J. Vis. Lang. Comput. 2, 311-332.
• 33: HIRAKAWA, M., NISHIMURA, Y., KADO, M., AND ICHIKAWA, T. 1991. Interpretation of icon overlapping in iconic programming. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 254-259.
• 34: KAHN, K. AND SARASWAT, V.A. 1990. Complete visualisations of concurrent programs and their executions. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 7-15.
• 35: Takuya Katayama, Translation of attribute grammars into procedures, ACM Transactions on Programming Languages and Systems (TOPLAS), v.6 n.3, p.345-369, July 1984 [doi>10.1145/579.586]
• 36: Kim Marriott , Bernd Meyer , Kent B. Wittenburg, A survey of visual language specification and recognition, Visual language theory, Springer-Verlag New York, Inc., New York, NY, 1998
• 37: José Meseguer, Conditional rewriting logic as a unified model of concurrency, Theoretical Computer Science, v.96 n.1, p.73-155, 6 April 1992
• 38: MEVEL, A. AND GUEGUEN, T. 1987. Smalltalk-80. MacMillan Education.
• 39: Mark Minas, Diagram Editing with Hypergraph Parser Support, Proceedings of the 1997 IEEE Symposium on Visual Languages (VL '97), p.226, April 23-26, 1997
• 40: Pierro Mussio , Maria Finadri , Pietro Gentini , Franco Colombo, A bootstrap approach to visual user-interface design and development, The Visual Computer: International Journal of Computer Graphics, v.8 n.2, p.75-93, Feb. 1992 [doi>10.1007/BF01900550]
• 41: MussIo, P., PIETROGRANDE, M., AND PROTTI, M. 1991. Simulation ofhepatological models: A study in visual interactive exploration of scientific problems. J. Vis. Lang. Comput. 2, 75-95.
• 42: NAJORK, M.A. AND KAPLAN, S.M. 1993. Specifying visual languages with conditional set rewrite systems. In Proceedings of the IEEE Workshop on Visual Languages. IEEE, New York, 12-18.
• 43: NORMAN, D.A. 1988. The Psychology of Everyday Things. Basic Books.
• 44: PAPAKONSTANTINOU, a. 1979. A poor man's realization of attribute grammars. Softw. Pract. Exper. 9, 9, 719-728.
• 45: James Lyle Peterson, Petri Net Theory and the Modeling of Systems, Prentice Hall PTR, Upper Saddle River, NJ, 1981
• 46: J. Rekers , A. Schurr, A graph grammar approach to graphical parsing, Proceedings of the 11th International IEEE Symposium on Visual Languages, p.195, September 05-09, 1995
• 47: J. Rekers , A.. Schuerr, A graph based framework for the implementation of visual environments, Proceedings of the 1996 IEEE Symposium on Visual Languages, p.148, September 03-06, 1996
• 48: Alexander Repenning , Tamara Sumner, Agentsheets: A Medium for Creating Domain-Oriented Visual Languages, Computer, v.28 n.3, p.17-25, March 1995 [doi>10.1109/2.366152]
• 49: Arto Salomaa, Formal languages, Academic Press Professional, Inc., San Diego, CA, 1987
• 50: David Canfield Smith , Allen Cypher , Jim Spohrer, KidSim: programming agents without a programming language, Communications of the ACM, v.37 n.7, p.54-67, July 1994 [doi>10.1145/176789.176795]
• 51: J. A. Serrano, The use of semantic constraints on diagram editors, Proceedings of the 11th International IEEE Symposium on Visual Languages, p.211, September 05-09, 1995
• 52: Ben Shneiderman, Designing the user interface (videotape), University of Maryland at College Park, College Park, MD, 1987
• 53: TONDL, L. 1981. Problems of Semantics. Reidel.
• 54: WANG, D., LEE, g. R., AND ZEEVAT, H. 1995. Reasoning with diagrammatic representations. In Diagrammatic Reasoning, J. Glasgow, N. H. Narayanan, and B. Chandrasekaran, Eds. AAAI Press, Menlo Park, CA, 339-393.
• 55: Peter Wegner, Interaction as a basis for empirical computer science, ACM Computing Surveys (CSUR), v.27 n.1, p.45-48, March 1995 [doi>10.1145/214037.214092]
• 56: Peter Wegner, Why interaction is more powerful than algorithms, Communications of the ACM, v.40 n.5, p.80-91, May 1997 [doi>10.1145/253769.253801]
• 57: Kent B. Wittenburg , Louis M. Weitzman, Relational grammars: theory and practice in a visual language interface for process modeling, Visual language theory, Springer-Verlag New York, Inc., New York, NY, 1998

INDEX TERMS

Primary Classification:
  D. Software
  D.1 PROGRAMMING TECHNIQUES

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces
      D.2.6 Programming Environments
          Subjects: Interactive environments

  F. Theory of Computation
  F.4 MATHEMATICAL LOGIC AND FORMAL LANGUAGES

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design; Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Human Factors, Languages

Keywords:
control automaton, dialogue control, visual languages