Creating user interfaces using programming by example, visual programming, and constraints

ABSTRACT

Peridot is an experimental tool that allows designers to create user interface components without conventional programming. The designer draws pictures of what the interface should look like and then uses the mouse and other input devices to demonstrate how the interface should operate. Peridot generalizes from these example pictures and actions to create parameterized procedures, such as those found in conventional user interface libraries such as the Macintosh Toolbox. Peridot uses visual programming, programming by example, constraints, and plausible inferencing to allow nonprogrammers to create menus, buttons, scroll bars, and many other interaction techniques easily and quickly. Peridot created its own interface and can create almost all of the interaction techniques in the Macintosh Toolbox. Therefore, Peridot demonstrates that it is possible to provide sophisticated programming capabilities to nonprogrammers in an easy-to-use manner and still have sufficient power to generate interesting and useful programs.

REFERENCES

• 1: Apple Computer Inc., Inside MacIntosh, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1994
• 2: Paul S. Barth, An object-oriented approach to graphical interfaces, ACM Transactions on Graphics (TOG), v.5 n.2, p.142-172, April 1986 [doi>10.1145/22949.22951]
• 3: BIERMANN, h. W., AND KRISHNASWAMY, R. Constructing programs from example computations. IEEE Trans. Softw. Eng. SE-2, 3 (Sept. 1976), 141-153.
• 4: BORNING, A. Thinglab--A constraint-oriented simulation laboratory. Tech. Rep. SSL-79-3, Xerox Palo Alto Research Center, Palo Alto, Calif., July 1979.
• 5: Alan Borning, The Programming Language Aspects of ThingLab, a Constraint-Oriented Simulation Laboratory, ACM Transactions on Programming Languages and Systems (TOPLAS), v.3 n.4, p.353-387, Oct. 1981 [doi>10.1145/357146.357147]
• 6: A. Borning, Defining constraints graphically, Proceedings of the SIGCHI conference on Human factors in computing systems, p.137-143, April 13-17, 1986, Boston, Massachusetts, United States
• 7: Alan Borning , Robert Duisberg, Constraint-based tools for building user interfaces, ACM Transactions on Graphics (TOG), v.5 n.4, p.345-374, Oct. 1986 [doi>10.1145/27623.29354]
• 8: O. Peter Buneman , Eric K. Clemons, Efficiently monitoring relational databases, ACM Transactions on Database Systems (TODS), v.4 n.3, p.368-382, Sept. 1979 [doi>10.1145/320083.320099]
• 9: W. Buxton , B. Myers, A study in two-handed input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.321-326, April 13-17, 1986, Boston, Massachusetts, United States
• 10: W. Buxton , M. R. Lamb , D. Sherman , K. C. Smith, Towards a comprehensive user interface management system, Proceedings of the 10th annual conference on Computer graphics and interactive techniques, p.35-42, July 25-29, 1983, Detroit, Michigan, United States
• 11: Luca Cardelli, Building user interfaces by direct manipulation, Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software, p.152-166, October 17-19, 1988, Alberta, Canada [doi>10.1145/62402.62428]
• 12: Luca Cardelli , Rob Pike, Squeak: a language for communicating with mice, ACM SIGGRAPH Computer Graphics, v.19 n.3, p.199-204, Jul. 1985
• 13: Eugene Charniak , Drew McDermott, Introduction to artificial intelligence, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1985
• 14: FOLEY, J. D., AND MCMATH, C.F. Dynamic process visualization. IEEE Comput. Graph. Appl.} 6, 2 (Mar. 1986), 16-25.
• 15: GOSLING, J. Algebraic constraints. Tech. Rep. CMU-CS-83-132, Computer Science Dept., Carnegie Mellon Univ., Pittsburgh, Pa., 1983.
• 16: GOULD, L., AND FINZER, W. Programming by rehearsal. Tech. Rep. SCL-84-1, Xerox Palo Alto Research Center, Palo Alto, Calif., May 1984. (A short version appears in Byte 9, 6 ( June 1984).)
• 17: Daniel Conrad Halbert, Programming by example, 1984
• 18: D. A. Henderson, Jr., The Trillium user interface design environment, Proceedings of the SIGCHI conference on Human factors in computing systems, p.221-227, April 13-17, 1986, Boston, Massachusetts, United States
• 19: Tyson R. Henry , Scott E. Hudson, Using active data in a UIMS, Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software, p.167-178, October 17-19, 1988, Alberta, Canada [doi>10.1145/62402.62429]
• 20: JACOB, R. J.K. A state transition diagram language for visual programming. Computer 18,(Aug. 1985), 51-59.
• 21: Robert J. K. Jacob, A specification language for direct-manipulation user interfaces, ACM Transactions on Graphics (TOG), v.5 n.4, p.283-317, Oct. 1986 [doi>10.1145/27623.27624]
• 22: William Leler, Constraint programming languages: their specification and generation, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988
• 23: LIEBERMAN, H. Constructing graphical user interfaces by example. In Proceedings of Graphics Interface, GI 82 (Toronto, Ontario, Canada, May). 1982, pp. 295-302.
• 24: D. L. Maulsby , I. H. Witten, Inducing programs in a direct-manipulation environment, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.57-62, March 1989
• 25: B. A. Myers, Visual programming, programming by example, and program visualization: a taxonomy, Proceedings of the SIGCHI conference on Human factors in computing systems, p.59-66, April 13-17, 1986, Boston, Massachusetts, United States
• 26: MYERS, B.A. Creating interaction techniques by demonstration. IEEE Comput. Graph. Appl. 7, 9 (Sept. 1987), 51-60.
• 27: MYERS, B.A. Creating user interfaces by demonstration: The Peridot user interface management system. 15 minute video tape. Siggraph Video Rev. 59, 2 (April, 1990). School of Computer Science, Carnegie Mellon Univ., Pittsburgh, Pa.
• 28: Brad A. Myers, Creating user interfaces by demonstration, Academic Press Professional, Inc., San Diego, CA, 1988
• 29: Brad A. Myers, User-Interface Tools: Introduction and Survey, IEEE Software, v.6 n.1, p.15-23, January 1989 [doi>10.1109/52.16898]
• 30: B. A. Myers, Encapsulating interactive behaviors, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.319-324, March 1989
• 31: Brad A. Myers , William Buxton, Creating highly-interactive and graphical user interfaces by demonstration, Proceedings of the 13th annual conference on Computer graphics and interactive techniques, p.249-258, August 1986
• 32: B. A. Myers , B. V. Zanden , R. B. Dannenberg, Creating graphical interactive application objects by demonstration, Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology, p.95-104, November 13-15, 1989, Williamsburg, Virginia, United States [doi>10.1145/73660.73672]
• 33: MYERS, B. A., GIUSE, D., DANNENBERG, R. B., VANDER ZANDEN, B., KOSBIE, D., MARCHAL, P., PERVIN, E., MICKISH, A., AND KOLOJEJCHICK, J.A. The Garnet Toolkit Reference Manuals: Support for Highly-Interactive, Graphical User Interfaces in Lisp. Tech. Rep. CMU-CS. 90-117. Carnegie Mellon School of Computer Science, March 1990.
• 34: Greg Nelson, Juno, a constraint-based graphics system, ACM SIGGRAPH Computer Graphics, v.19 n.3, p.235-243, Jul. 1985
• 35: Robert P. Nix, Editing by example, ACM Transactions on Programming Languages and Systems (TOPLAS), v.7 n.4, p.600-621, Oct. 1985 [doi>10.1145/4472.4476]
• 36: D R Olsen, Jr., Larger issues in user interface management, ACM SIGGRAPH Computer Graphics, v.21 n.2, p.134-137, April 1987 [doi>10.1145/24919.24932]
• 37: Dan R Olsen, Jr. , Elizabeth P. Dempsey, SYNGRAPH: A graphical user interface generator, Proceedings of the 10th annual conference on Computer graphics and interactive techniques, p.43-50, July 25-29, 1983, Detroit, Michigan, United States
• 38: Dan R. Olsen, Jr. , Elizabeth P. Dempsey , Roy Rogge, Input/output linkage in a user interface management system, ACM SIGGRAPH Computer Graphics, v.19 n.3, p.191-197, Jul. 1985
• 39: Theo Pavlidis , Christopher J. Van Wyk, An automatic beautifier for drawings and illustrations, ACM SIGGRAPH Computer Graphics, v.19 n.3, p.225-234, Jul. 1985
• 40: PrAtt, G. R., ED. User Interface Management Systems. Springer-Verlag, New York, 1985.
• 41: C. V. Ramamoorthy , Shashi Shekhar , Vijay Garg, Software development support for AI programs, Computer, v.20 n.1, p.30-40, Jan. 1987 [doi>10.1109/MC.1987.1663354]
• 42: SHAW, D. E., SWARTOUT, W. R., AND CORDELL GREEN, C. Inferring Lisp programs from examples. In Proceedings of the 4th International Joint Conference on Artificial Intelligence, IJCAI 75 (Tbilisi, USSR, Sept.). 1975, pp. 260-267.
• 43: SHNEIDERMAN, B. Direct manipulation: A step beyond programming languages. Computer 16, 8 (Aug. 1983), 57-69.
• 44: G. Singh , M. Green, A high-level user interface management system, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.133-138, March 1989
• 45: SMITH, D. C. Pygmalion: A Computer Program to Model and Stimulate Creative Thought. Birkh~iuser Verlag, Basel, 1977.
• 46: STEFIK, M., BORROW, D. G., AND KAHN, K.M. Integrating access-oriented programming into a multi-paradigm environment. IEEE Softw. 3, i (Jan. 1986), 10-18.
• 47: STEVENS, A., ROBERTS, B., AND STEAD, L. The use of a sophisticated graphics interface in computer-assisted instruction. IEEE Comput. Graph. Appl. 3, 2 (Mar.-Apr. 1983), 25-31.
• 48: Ivan E. Sutherland, Sketch pad a man-machine graphical communication system, Proceedings of the SHARE design automation workshop, p.6.329-6.346, January 1964 [doi>10.1145/800265.810742]
• 49: Pedro Szekely , Brad Myers, A user interface toolkit based on graphical objects and constraints, ACM SIGPLAN Notices, v.23 n.11, p.36-45, Nov. 1988
• 50: TESLER, L. The Smalltalk Environment. Byte Mag. 6, 8 (Aug. 1981), 90-147.
• 51: B. T. Zanden, Constraint grammars–a new model for specifying graphical applications, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.325-330, March 1989
• 52: Anthony I. Wasserman , David T. Shewmake, Rapid prototyping of interactive information systems, ACM SIGSOFT Software Engineering Notes, v.7 n.5, p.171-180, December 1982
• 53: WILLIAMS, G. The Apple Macintosh computer. Byte 9, 2 (Feb. 1984), 30-54.
• 54: XEROX CORPORATION. Interlisp Reference Manual. Xerox Corp., Pasadena, Calif., 1983.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments
          Subjects: Interactive environments

  H. Information Systems
  H.1 MODELS AND PRINCIPLES

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.2 Automatic Programming
          Subjects: Program synthesis
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Design, Human Factors