Capturing user tests in a multimodal, multidevice informal prototyping tool

ABSTRACT

Interaction designers are increasingly faced with the challenge of creating interfaces that incorporate multiple input modalities, such as pen and speech, and span multiple devices. Few early stage prototyping tools allow non-programmers to prototype these interfaces. Here we describe CrossWeaver, a tool for informally prototyping multimodal, multidevice user interfaces. This tool embodies the informal prototyping paradigm, leaving design representations in an informal, sketched form, and creates a working prototype from these sketches. CrossWeaver allows a user interface designer to sketch storyboard scenes on the computer, specifying simple multimodal command transitions between scenes. The tool also allows scenes to target different output devices. Prototypes can run across multiple standalone devices simultaneously, processing multimodal input from each one. Thus, a designer can visually create a multimodal prototype for a collaborative meeting or classroom application. CrossWeaver captures all of the user interaction when running a test of a prototype. This input log can quickly be viewed visually for the details of the users' multimodal interaction or it can be replayed across all participating devices, giving the designer information to help him or her analyze and iterate on the interface design.

REFERENCES

• 1: Embarking on Multimodal Interface Design, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.355, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167021]
• 2: Oviatt, S., P. Cohen, L. Wu, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro, Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions. Human-Computer Interaction, 2000. 15(4): p. 263--322.
• 3: Anoop K. Sinha , James A. Landay, Visually prototyping perceptual user interfaces through multimodal storyboarding, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971501]
• 4: Jun Rekimoto, Pick-and-drop: a direct manipulation technique for multiple computer environments, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.31-39, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263505]
• 5: Scott R. Klemmer , Anoop K. Sinha , Jack Chen , James A. Landay , Nadeem Aboobaker , Annie Wang, Suede: a Wizard of Oz prototyping tool for speech user interfaces, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.1-10, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354406]
• 6: James A. Landay , Brad A. Myers, Sketching Interfaces: Toward More Human Interface Design, Computer, v.34 n.3, p.56-64, March 2001
• 7: Moran, L. B., A. J. Cheyer, L. E. Julia, D. L. Martin, and S. Park, Multimodal User Interfaces in the Open Agent Architecture. Knowledge-Based Systems, 1998. 10(5): p. 295--304.
• 8: James A. Landay , Brad A. Myers, Interactive sketching for the early stages of user interface design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.43-50, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223910]
• 9: James Lin , Mark W. Newman , Jason I. Hong , James A. Landay, DENIM: finding a tighter fit between tools and practice for Web site design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.510-517, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332486]
• 10: Scott R. Klemmer , Michael Thomsen , Ethan Phelps-Goodman , Robert Lee , James A. Landay, Where do web sites come from?: capturing and interacting with design history, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503378]
• 11: Brian P. Bailey , Joseph A. Konstan , John V. Carlis, DEMAIS: designing multimedia applications with interactive storyboards, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500179]
• 12: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 13: Clow, J. and S. L. Oviatt. STAMP: an Automated Tool for Analysis of Multimodal System Performance. In Proceedings of the International Conference on Spoken Language Processing. (Sydney, Australia, 1998).
• 14: Marc Rettig, Prototyping for tiny fingers, Communications of the ACM, v.37 n.4, p.21-27, April 1994 [doi>10.1145/175276.175288]
• 15: Reekie, J., M. Shilman, H. Hse, and S. Neuendorffer, Diva, a Software Infrastructure for Visualizing and Interacting with Dynamic Information Spaces. http://www.gigascale.org/diva/, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: Evolutionary prototyping; User interfaces

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen); Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Design, Experimentation, Human Factors

Keywords:
informal prototyping, mobile interface design, multidevice, multimodal, pen and speech input, sketching