Informal prototyping of continuous graphical interactions by demonstration

ABSTRACT

Informal prototyping tools have shown great potential in facilitating the early stage design of user interfaces. How-ever, continuous interactions, an important constituent of highly interactive interfaces, have not been well supported by previous tools. These interactions give continuous visual feedback, such as geometric changes of a graphical object, in response to continuous user input, such as the movement of a mouse. We built Monet, a sketch-based tool for proto-typing continuous interactions by demonstration. In Monet, designers can prototype continuous widgets and their states of interest using examples. They can also demonstrate com-pound behaviors involving multiple widgets by direct ma-nipulation. Monet allows continuous interactions to be eas-ily integrated with event-based, discrete interactions. Con-tinuous widgets can be embedded into storyboards and their states can condition or trigger storyboard transitions. Monet achieves these features by employing continuous function approximation and statistical classification techniques, without using any domain specific knowledge or assuming any application semantics. Informal feedback showed that Monet is a promising approach to enabling more complete tool support for early stage UI design.

REFERENCES

• 1: Brian P. Bailey , Joseph A. Konstan , John V. Carlis, DEMAIS: designing multimedia applications with interactive storyboards, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500179]
• 2: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 3: Caetano, A., et al. JavaSketchIt: Issues in Sketching the Look of User Interfaces. In AAAI Spring Symposium on Sketch Understanding. 2002. Palo Alto, CA. pp. 9--14.
• 4: Allen Cypher , Daniel C. Halbert , David Kurlander , Henry Lieberman , David Maulsby , Brad A. Myers , Alan Turransky, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 5: Martin R. Frank , Piyawadee Noi Sukaviriya , James D. Foley, Inference bear: designing interactive interfaces through before and after snapshots, Proceedings of the conference on Designing interactive systems: processes, practices, methods, & techniques, p.167-175, August 23-25, 1995, Ann Arbor, Michigan, United States [doi>10.1145/225434.225453]
• 6: Jason I. Hong , James A. Landay, SATIN: a toolkit for informal ink-based applications, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.63-72, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354412]
• 7: Robert J. K. Jacob , Leonidas Deligiannidis , Stephen Morrison, A software model and specification language for non-WIMP user interfaces, ACM Transactions on Computer-Human Interaction (TOCHI), v.6 n.1, p.1-46, March 1999 [doi>10.1145/310641.310642]
• 8: JAMA. A Java Matrix Package. http://math.nist.gov/javanumerics/jama/.
• 9: Lucas Kovar , Michael Gleicher, Simplicial families of drawings, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502378]
• 10: David Kurlander , Steven Feiner, Inferring constraints from multiple snapshots, ACM Transactions on Graphics (TOG), v.12 n.4, p.277-304, Oct. 1993 [doi>10.1145/159730.159731]
• 11: James A. Landay , Brad A. Myers, Sketching Interfaces: Toward More Human Interface Design, Computer, v.34 n.3, p.56-64, March 2001
• 12: Yang Li , Jason I. Hong , James A. Landay, Topiary: a tool for prototyping location-enhanced applications, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029671]
• 13: Lieberman, H., ed. Your Wish Is My Command: Programming by Example. 2001, Morgan Kaufmann: San Francisco, CA.
• 14: Richard G. McDaniel , Brad A. Myers, Getting more out of programming-by-demonstration, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.442-449, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303127]
• 15: Brad A. Myers, Creating dynamic interaction techniques by demonstration, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.271-278, April 05-09, 1987, Toronto, Ontario, Canada
• 16: Brad A. Myers , Richard G. McDaniel , David S. Kosbie, Marquise: creating complete user interfaces by demonstration, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.293-300, May 1993, Amsterdam, The Netherlands
• 17: Newman, M.W., et al., DENIM: An Informal Web Site Design Tool Inspired by Observations of Practice. Human-Computer Interaction, 2003. 18(3): pp. 259--324.
• 18: Jun-yong Noh , Douglas Fidaleo , Ulrich Neumann, Animated deformations with radial basis functions, Proceedings of the ACM symposium on Virtual reality software and technology, October 22-25, 2000, Seoul, Korea [doi>10.1145/502390.502422]
• 19: Dan R. Olsen, Jr. , Brett Ahlstrom , Douglas Kohlert, Building geometry-based widgets by example, Proceedings of the SIGCHI conference on Human factors in computing systems, p.35-42, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223909]
• 20: Russell, S. and Norvig, P., Statistical Learning Methods, In Artificial Intelligence: A Modern Approach. 2003, Prentice Hall. pp. 712--762.
• 21: Maarten van de Kant , Stephanie Wilson , Mathilde Bekker , Hilary Johnson , Peter Johnson, PatchWork: a software tool for early design, CHI 98 conference summary on Human factors in computing systems, p.221-222, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286711]
• 22: David Wolber, Pavlov: an interface builder for designing animated interfaces, ACM Transactions on Computer-Human Interaction (TOCHI), v.4 n.4, p.347-386, Dec. 1997 [doi>10.1145/267135.267142]
• 23: David Wolber , Gene Fisher, A demonstrational technique for developing interfaces with dynamically created objects, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.221-230, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120806]
• 24: Wolberg, G., Image Morphing: A Survey. Visual Computer, 1998. 14: pp. 360--372.
• 25: Yin Yin Wong, Rough and ready prototypes: lessons from graphic design, Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems, May 03-07, 1992, Monterey, California [doi>10.1145/1125021.1125094]
• 26: Brad Vander Zanden , Brad A. Myers, Demonstrational and constraint-based techniques for pictorially specifying application objects and behaviors, ACM Transactions on Computer-Human Interaction (TOCHI), v.2 n.4, p.308-356, Dec. 1995 [doi>10.1145/212430.212443]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping

Keywords:
continuous interactions, direct manipulation, informal prototyping, pen-based user interfaces, programming by demonstration, sketching