Extending a graphical toolkit for two-handed interaction

ABSTRACT

Multimodal interaction combines input from multiple sensors such as pointing devices or speech recognition systems, in order to achieve more fluid and natural interaction. Two-handed interaction has been used recently to enrich graphical interaction. Building applications that use such combined interaction requires new software techniques and frameworks. Using additional devices means that user interface toolkits must be more flexible with regard to input devices and event types. The possibility of parallel interactions must also be taken into account, with consequences on the structure of toolkits. Finally, frameworks must be provided for the combination of events and status of several devices. This paper reports on the extensions we made to the direct manipulation interface toolkit Whizz in order to experiment two-handed interaction. These extensions range from structural adaptations of the toolkit to new techniques for specifying the time-dependent fusion of events.

REFERENCES

• 1: M. Beaudouin-Lafon, Y. Berteaud, and S. Chatty. Creating direct manipulation interfaces with XTV. In Proceedings of EX'90, London, pages 148-155, 1990.
• 2: Michel Beaudouin-Lafon, A TOUR THROUGH THE Avis UIMS, ACM SIGCHI Bulletin, v.23 n.4, p.49-50, Oct. 1991 [doi>10.1145/126729.1056055]
• 3: Eric A. Bier , Steven Freeman, MMM: a user interface architecture for shared editors on a single screen, Proceedings of the 4th annual ACM symposium on User interface software and technology, p.79-86, November 11-13, 1991, Hilton Head, South Carolina, United States [doi>10.1145/120782.120791]
• 4: Eric A. Bier , Maureen C. Stone , Ken Pier , William Buxton , Tony D. DeRose, Toolglass and magic lenses: the see-through interface, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.73-80, September 1993 [doi>10.1145/166117.166126]
• 5: Meera M. Blattner , Roger B. Dannenberg, Multimedia interface design, ACM Press, New York, NY, 1992
• 6: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 7: W. Buxton , B. Myers, A study in two-handed input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.321-326, April 13-17, 1986, Boston, Massachusetts, United States
• 8: Stephane Chatty, Defining the Dynamic Behaviour of Animated Interfaces, Proceedings of the IFIP TC2/WG2.7 Working Conference on Engineering for Human-Computer Interaction, p.95-111, August 10-12, 1992
• 9: Pierre Cointe , Xavier Rodet, Formes: An object and time oriented system for music composition and synthesis, Proceedings of the 1984 ACM Symposium on LISP and functional programming, p.85-95, August 06-08, 1984, Austin, Texas, United States [doi>10.1145/800055.802024]
• 10: Arno P. J. Gourdol , Laurence Nigay , Daniel Salber , Joëlle Coutaz, Two Case Studies of Software Architecture for Multimodal Interactive Systems: VoicePaint and a Voice-enabled Graphical Notebook, Proceedings of the IFIP TC2/WG2.7 Working Conference on Engineering for Human-Computer Interaction, p.271-284, August 10-12, 1992
• 11: Ralph D. Hill, Supporting concurrency, communication, and synchronization in human-computer interaction—the Sassafras UIMS, ACM Transactions on Graphics (TOG), v.5 n.3, p.179-210, July 1986 [doi>10.1145/24054.24055]
• 12: Groupe multimodalitd. In Actes d'IHM'91 - Troisi~mes journdes sur l'ingdnidrie des interfaces hommemachine. 1991.
• 13: Paul Kabbash , William Buxton , Abigail Sellen, Two-handed input in a compound task, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.417-423, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191808]
• 14: Gordon Kurtenbach , Thomas Baudel, HyperMark: issuing commands by drawing marks in HyperCard, Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems, May 03-07, 1992, Monterey, California [doi>10.1145/1125021.1125081]
• 15: Mark A. Linton , John M. Vissides , Paul R. Calder, Composing User Interfaces with InterViews, Computer, v.22 n.2, p.8-22, February 1989 [doi>10.1109/2.19829]
• 16: Brad A. Myers, A new model for handling input, ACM Transactions on Information Systems (TOIS), v.8 n.3, p.289-320, July 1990 [doi>10.1145/98188.98204]
• 17: Laurence Nigay , Joëlle Coutaz, A design space for multimodal systems: concurrent processing and data fusion, Proceedings of the SIGCHI conference on Human factors in computing systems, p.172-178, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169143]
• 18: D. A. Norman. The Design of Everyday Things. Doubleday, New York, 1990.
• 19: Dean Harris Rubine, The automatic recognition of gestures, Carnegie Mellon University, Pittsburgh, PA, 1992
• 20: David Weimer , S. K. Ganapathy, Interaction techniques using hand tracking and speech recognition, Multimedia interface design, ACM Press, New York, NY, 1992 [doi>10.1145/146022.146042]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.4 Graphics Utilities
          Subjects: Graphics editors

General Terms:
Design, Human Factors

Keywords:
direct manipulation, graphical toolkit, interaction styles, multimodal interaction, two-handed interaction