A software model and specification language for non-WIMP user interfaces

ABSTRACT

We present a software model and language for describing and programming the fine-grained aspects of interaction in a non-WIMP user interface, such as a virtual environment. Our approach is based on our view that the essence of a non-WIMP dialogue is a set of continuous relationships—most of which are temporary. The model combines a data-flow or constraint-like component for the continuous relationships with an event-based component for discrete interactions, which can enable or diable individual continuous relationships. To demonstrate our approach, we present the PMIW user interface management system for non-WIMP interactions, a set of examples running under it, a visual editor for our user interface description language, and a discussion of our implemantation and our restricted use of constraints for a performance-driven interactive situation. Our goal is to provide a model and language that captures the formal structure of non-WIMP interactions in the way that various previous techniques have captured command-based, textual, and event-based styles and to suggest that using it need and not compromise real-time performance.

REFERENCES

• 1: Gregory D. Abowd , Alan J. Dix, Integrating status and event phenomena in formal specifications of interactive systems, ACM SIGSOFT Software Engineering Notes, v.19 n.5, p.44-52, Dec. 1994
• 2: Ben B. Bederson , Larry Stead , James D. Hollan, Pad++: advances in multiscale interfaces, Conference companion on Human factors in computing systems, p.315-316, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/259963.260379]
• 3: Lawrence D. Bergman , Jane S. Richardson , David C. Richardson , Frederick P. Brooks, Jr., VIEW: an exploratory molecular visualization system with user-definable interaction sequences, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.117-126, September 1993 [doi>10.1145/166117.166132]
• 4: David A. Carr, Specification of interface interaction objects, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.372-378, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191793]
• 5: CARR, D. A., JOG, N., KUMAR, H. P., TEITTINEN, M., AND AHLBERG, C. 1994. Using interaction object graphs to specify and develop graphical widgets. Tech. Rep. ISR-TR-94-69. Institute for Systems Research, University of Maryland, College Park, MD.
• 6: Stéphane Chatty, Extending a graphical toolkit for two-handed interaction, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.195-204, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192500]
• 7: CITRIN, W., DOHERTY, M., AND ZORN, B. 1995. Design of a completely visual object-oriented programming language. In Visual Object-Oriented Programming, A. Burnett, A. Goldberg, and T. Lewis, Eds. Prentice-Hall, New York, NY.
• 8: Matthew Conway , Randy Pausch , Rich Gossweiler , Tommy Burnette, Alice: a rapid prototyping system for building virtual environments, Conference companion on Human factors in computing systems, p.295-296, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/259963.260503]
• 9: James Cremer , Joseph Kearney , Yiannis Papelis, HCSM: a framework for behavior and scenario control in virtual environments, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.5 n.3, p.242-267, July 1995 [doi>10.1145/217853.217857]
• 10: James Cremer , Joseph Kearney , Yiannis Papelis, Driving Simulation: Challenges for VR Technology, IEEE Computer Graphics and Applications, v.16 n.5, p.16-20, September 1996 [doi>10.1109/38.536270]
• 11: Conal Elliott , Greg Schechter , Ricky Yeung , Salim Abi-Ezzi, TBAG: a high level framework for interactive, animated 3D graphics applications, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.421-434, July 1994 [doi>10.1145/192161.192276]
• 12: S. K. Feiner , Clifford Beshers, Worlds within worlds: metaphors for exploring n-dimensional virtual worlds, Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology, p.76-83, October 03-05, 1990, Snowbird, Utah, United States [doi>10.1145/97924.97933]
• 13: Mark A. Flecchia , R. Daniel Bergeron, Specifying complex dialogs in ALGAE, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.229-234, April 05-09, 1987, Toronto, Ontario, Canada
• 14: James Foley , Won Chul Kim , Srdjan Kovacevic , Kevin Murray, Defining Interfaces at a High Level of Abstraction, IEEE Software, v.6 n.1, p.25-32, January 1989 [doi>10.1109/52.16899]
• 15: FREEMAN-BENSON, B. AND BORNING, A. 1992. The design and implementation of Kaleidoscope'90, a constraint imperative programming language. In Proceedings of the IEEE Computer Society International Conference on Computer Languages (Apr.). IEEE Press, Piscataway, NJ, 174-180.
• 16: Michael Gleicher, A graphics toolkit based on differential constraints, Proceedings of the 6th annual ACM symposium on User interface software and technology, p.109-120, December 1993, Atlanta, Georgia, United States [doi>10.1145/168642.168653]
• 17: GOSSWEILER, R., LONG, C., KOGA, S., AND PAUSCH, R. 1993. DIVER: A distributed virtual environment research platform. In Proceedings of the IEEE Symposium on Research Frontiers in Virtual Reality (San Jose, CA., Oct. 25-26). IEEE Press, Piscataway, NJ.
• 18: Mark Green, The University of Alberta user interface management system, ACM SIGGRAPH Computer Graphics, v.19 n.3, p.205-213, Jul. 1985
• 19: Mark Green , Robert Jacob, SIGGRAPH '90 Workshop report: software architectures and metaphors for non-WIMP user interfaces, ACM SIGGRAPH Computer Graphics, v.25 n.3, p.229-235, July 1991 [doi>10.1145/126640.126677]
• 20: Mark Green , Sean Halliday, A geometric modeling and animation system for virtual reality, Communications of the ACM, v.39 n.5, p.46-53, May 1996 [doi>10.1145/229459.229465]
• 21: Ralph D. Hill, Supporting concurrency, communication, and synchronization in human-computer interaction—the Sassafras UIMS, ACM Transactions on Graphics (TOG), v.5 n.3, p.179-210, July 1986 [doi>10.1145/24054.24055]
• 22: Ralph D. Hill, The Rendezvous constraint maintenance system, Proceedings of the 6th annual ACM symposium on User interface software and technology, p.225-234, December 1993, Atlanta, Georgia, United States [doi>10.1145/168642.168665]
• 23: Ralph D. Hill , Tom Brinck , Steven L. Rohall , John F. Patterson , Wayne Wilner, The Rendezvous architecture and language for constructing multiuser applications, ACM Transactions on Computer-Human Interaction (TOCHI), v.1 n.2, p.81-125, June 1994 [doi>10.1145/180171.180172]
• 24: Ken Hinckley , Randy Pausch , John C. Goble , Neal F. Kassell, A survey of design issues in spatial input, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.213-222, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192501]
• 25: Deborah Hix , James N. Templeman , Robert J. K. Jacob, Pre-screen projection: from concept to testing of a new interaction technique, Proceedings of the SIGCHI conference on Human factors in computing systems, p.226-233, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223933]
• 26: HUDSON, S. AND SMITH, I. 1994. Practical system for compiling one-way constraint into C+ + objects. Tech. Rep. Visualization and Usability Center, Georgia Tech Graphics.
• 27: S. E. Hudson, Graphical specification of flexible user interface displays, Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology, p.105-114, November 13-15, 1989, Williamsburg, Virginia, United States [doi>10.1145/73660.73673]
• 28: Scott E. Hudson, Incremental attribute evaluation: a flexible algorithm for lazy update, ACM Transactions on Programming Languages and Systems (TOPLAS), v.13 n.3, p.315-341, July 1991 [doi>10.1145/117009.117012]
• 29: Robert J.K. Jacob, Executable specifications for a human-computer interface, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.28-34, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801575]
• 30: Robert J. K. Jacob, Using formal specifications in the design of a human-computer interface, Communications of the ACM, v.26 n.4, p.259-264, April 1983 [doi>10.1145/2163.358093]
• 31: JACOB, R. J. K. 1985. An executable specification technique for describing human-computer interaction. In Advances in Human-Computer Interaction, H. R. Hartson, Ed. Ablex Publishing Corp., Norwood, NJ, 211-242.
• 32: Robert J. K. Jacob, A specification language for direct-manipulation user interfaces, ACM Transactions on Graphics (TOG), v.5 n.4, p.283-317, Oct. 1986 [doi>10.1145/27623.27624]
• 33: JACOB, R. J. K. 1993. Eye movement-based human-computer interaction techniques: Toward non-command interfaces. In Advances in Human-Computer Interaction, H. R. Hartson and D. Hix, Eds. Ablex Publishing Corp., Norwood, NJ, 151-190. http://www.eecs.tufts.edu/ -j acob/p ape rs/hartson, txt; h ttp ://www. e ec s. tuft s. e duFj acob/p ape rs/hartson, ps.
• 34: Robert J. K. Jacob, A Visual Language for Non-WIMP User Interfaces, Proceedings of the 1996 IEEE Symposium on Visual Languages, p.231, September 03-06, 1996
• 35: JACOB, R. J. K., LEGGETT, J. J., MYERS, B. A., AND PAUSCH, R. 1993. Interaction styles and input/output devices. Behav. Inf. Tech. 12, 2, 69-79. http://www'eecs'tufts'eduFjac~b/ papers/bit.txt; http://www.eecs.tufts.eduFj acob/papers/bit.ps.
• 36: David J. Kasik, A user interface management system, ACM SIGGRAPH Computer Graphics, v.16 n.3, p.99-106, July 1982
• 37: Michael Kass, CONDOR: constraint-based dataflow, ACM SIGGRAPH Computer Graphics, v.26 n.2, p.321-330, July 1992
• 38: J. Bryan Lewis , Lawrence Koved , Daniel T. Ling, Dialogue structures for virtual worlds, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.131-136, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108864]
• 39: LIANG, J. AND GREEN, M. 1994. JDCAD: A highly interactive 3D modeling system. Comput. Graph. 18, 4, 499-506.
• 40: MACKINLAY, J. D., CARD, S. K., AND ROBERTSON, G. G. 1990. A semantic analysis of the design space of input devices. Human-Comput. Interact. 5, 145-190.
• 41: Stephen Andrew Morrison , Robert J. Jacob, A specification paradigm for design and implementation of non-wimp human-computer interactions, 1998
• 42: Stephen A. Morrison , Robert J. K. Jacob, A specification paradigm for design and implementation of non-WIMP user interfaces, CHI 98 conference summary on Human factors in computing systems, p.357-358, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286826]
• 43: Brad A. Myers, Creating user interfaces by demonstration, Academic Press Professional, Inc., San Diego, CA, 1988
• 44: Brad A. Myers, User interface software tools, ACM Transactions on Computer-Human Interaction (TOCHI), v.2 n.1, p.64-103, March 1995 [doi>10.1145/200968.200971]
• 45: Brad A. Myers , Dario A. Giuse , Roger B. Dannenberg , David S. Kosbie , Edward Pervin , Andrew Mickish , Brad Vander Zanden , Philippe Marchal, Garnet: Comprehensive Support for Graphical, Highly Interactive User Interfaces, Computer, v.23 n.11, p.71-85, November 1990 [doi>10.1109/2.60882]
• 46: NEWMAN, W. M. 1968. A system for interactive graphical programming. In Proceedings of the Spring Joint Computer Conference. AFIPS Press, Arlington, VA, 47-54.
• 47: Jakob Nielsen, Noncommand user interfaces, Communications of the ACM, v.36 n.4, p.83-99, April 1993 [doi>10.1145/255950.153582]
• 48: Dan Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1992
• 49: REISNER, P. 1981. Formal grammar and human factors design of an interactive graphics system. IEEE Trans. Softw. Eng. SE-7, 2, 229-240.
• 50: James R. Rhyne, Extensions to C for interface programming, Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software, p.30-45, October 17-19, 1988, Alberta, Canada [doi>10.1145/62402.62405]
• 51: John Rohlf , James Helman, IRIS performer: a high performance multiprocessing toolkit for real-time 3D graphics, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.381-394, July 1994 [doi>10.1145/192161.192262]
• 52: Chris Shaw , Mark Green , Jiandong Liang , Yunqi Sun, Decoupled simulation in virtual reality with the MR toolkit, ACM Transactions on Information Systems (TOIS), v.11 n.3, p.287-317, July 1993 [doi>10.1145/159161.173948]
• 53: SHNEIDERMAN, B. 1982. Multi-party grammars and related features for defining interactive systems. IEEE Trans. Syst. Man Cybern. SMC-12, 2, 148-154.
• 54: John L. Sibert , William D. Hurley , Teresa W. Bleser, An object-oriented user interface management system, ACM SIGGRAPH Computer Graphics, v.20 n.4, p.259-268, Aug. 1986
• 55: Marc P. Stevens , Robert C. Zeleznik , John F. Hughes, An architecture for an extensible 3D interface toolkit, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.59-67, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192449]
• 56: Bernard Sulfrin , Jifeng He, Specification, analysis and refinement of interactive processes, Formal methods in human-computer interaction, Cambridge University Press, New York, NY, 1990
• 57: Pedro Szekely, Modular implementation of presentations, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.235-240, April 05-09, 1987, Toronto, Ontario, Canada
• 58: TANIMOTO, S. L. 1990. VIVA: A visual language for image processing. J. Visual Lang. Comput. 1, 2 (June), 127-139.
• 59: Brad Vander Zanden , Brad A. Myers , Dario A. Giuse , Pedro Szekely, Integrating pointer variables into one-way constraint models, ACM Transactions on Computer-Human Interaction (TOCHI), v.1 n.2, p.161-213, June 1994 [doi>10.1145/180171.180174]
• 60: Colin Ware , Ravin Balakrishnan, Reaching for objects in VR displays: lag and frame rate, ACM Transactions on Computer-Human Interaction (TOCHI), v.1 n.4, p.331-356, Dec. 1994 [doi>10.1145/198425.198426]
• 61: YUNTEN, T. AND HARTSON, H. R. 1985. A supervisory methodology and notation (SUPERMAN) for human-computer system development. In Advances in Human-Computer Interaction, H. R. Hartson, Ed. Ablex Publishing Corp., Norwood, NJ, 243-281.
• 62: Pamela Zave , Michael Jackson, Conjunction as composition, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.4, p.379-411, Oct. 1993 [doi>10.1145/158431.158438]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.1 Specifying and Verifying and Reasoning about Programs
          Subjects: Specification techniques

  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality

General Terms:
Design, Human Factors, Languages

Keywords:
PMIW, interaction techiques, non-WIMP interface, specification language, state transition diagram, user interface management system (UIMS)