User-defined gestures for surface computing

ABSTRACT

Many surface computing prototypes have employed gestures created by system designers. Although such gestures are appropriate for early investigations, they are not necessarily reflective of user behavior. We present an approach to designing tabletop gestures that relies on eliciting gestures from non-technical users by first portraying the effect of a gesture, and then asking users to perform its cause. In all, 1080 gestures from 20 participants were logged, analyzed, and paired with think-aloud data for 27 commands performed with 1 and 2 hands. Our findings indicate that users rarely care about the number of fingers they employ, that one hand is preferred to two, that desktop idioms strongly influence users' mental models, and that some commands elicit little gestural agreement, suggesting the need for on-screen widgets. We also present a complete user-defined gesture set, quantitative agreement scores, implications for surface technology, and a taxonomy of surface gestures. Our results will help designers create better gesture sets informed by user behavior.

REFERENCES

• 1: Thomas Baudel , Michel Beaudouin-Lafon, Charade: remote control of objects using free-hand gestures, Communications of the ACM, v.36 n.7, p.28-35, July 1993 [doi>10.1145/159544.159562]
• 2: Nicole Beringer, Evoking Gestures in SmartKom - Design of the Graphical User Interface, Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction, p.228-240, April 18-20, 2001
• 3: Paul Dietz , Darren Leigh, DiamondTouch: a multi-user touch technology, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502389]
• 4: Efron, D. (1941) Gesture and Environment. Morningside Heights, New York: King's Crown Press.
• 5: Julien Epps , Serge Lichman , Mike Wu, A study of hand shape use in tabletop gesture interaction, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125601]
• 6: Foley, J.D., van Dam, A., Feiner, S.K. and Hughes, J.F. (1996) The form and content of user-computer dialogues. In Computer Graphics: Principles and Practice. Reading, MA: Addison-Wesley, 392--395.
• 7: Clifton Forlines , Alan Esenther , Chia Shen , Daniel Wigdor , Kathy Ryall, Multi-user, multi-display interaction with a single-user, single-display geospatial application, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166296]
• 8: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 9: Michael D. Good , John A. Whiteside , Dennis R. Wixon , Sandra J. Jones, Building a user-derived interface, Communications of the ACM, v.27 n.10, p.1032-1043, Oct 1984 [doi>10.1145/358274.358284]
• 10: Edwin L. Hutchins , James D. Hollan , Donald A. Norman, Direct manipulation interfaces, Human-Computer Interaction, v.1 n.4, p.311-338, December 1985 [doi>10.1207/s15327051hci0104_2]
• 11: Kendon, A. (1988) How gestures can become like words. In Crosscultural Perspectives in Nonverbal Communication, F. Poyatos (ed). Toronto: C. J. Hogrefe, 131--141.
• 12: Jun Liu , David Pinelle , Samer Sallam , Sriram Subramanian , Carl Gutwin, TNT: improved rotation and translation on digital tables, Proceedings of Graphics Interface 2006, June 07-09, 2006, Quebec, Canada
• 13: Allan Christian Long, Jr. , James A. Landay , Lawrence A. Rowe, Implications for a gesture design tool, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.40-47, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.302985]
• 14: Shahzad Malik , Abhishek Ranjan , Ravin Balakrishnan, Interacting with large displays from a distance with vision-tracked multi-finger gestural input, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095042]
• 15: McNeill, D. (1992) Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press.
• 16: Christophe Mignot , Claude Valot , Noëlle Carbonell, An experimental study of future “natural” multimodal human-computer interaction, INTERACT '93 and CHI '93 conference companion on Human factors in computing systems, p.67-68, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/259964.260075]
• 17: Meredith Ringel Morris , Anqi Huang , Andreas Paepcke , Terry Winograd, Cooperative gestures: multi-user gestural interactions for co-located groupware, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124952]
• 18: Tomer Moscovich , John F. Hughes, Multi-finger cursor techniques, Proceedings of Graphics Interface 2006, June 07-09, 2006, Quebec, Canada
• 19: Nielsen, M., Störring, M., Moeslund, T.B. and Granum, E. (2004) A procedure for developing intuitive and ergonomic gesture interfaces for HCI. Int'l Gesture Workshop 2003, LNCS vol. 2915. Heidelberg: Springer-Verlag, 409--420.
• 20: Isabella Poggi, From a Typology of Gestures to a Procedure for Gesture Production, Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction, p.158-168, April 18-20, 2001
• 21: Jun Rekimoto, SmartSkin: an infrastructure for freehand manipulation on interactive surfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503397]
• 22: Sandrine Robbe-Reiter , Noëlle Carbonell , Pierre Dauchy, Expression constraints in multimodal human-computer interaction, Proceedings of the 5th international conference on Intelligent user interfaces, p.225-228, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325852]
• 23: Sandrine Robbe, An empirical study of speech and gesture interaction: toward the definition of ergonomic design guidelines, CHI 98 conference summary on Human factors in computing systems, p.349-350, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286815]
• 24: Rossini, N. (2004) The analysis of gesture: Establishing a set of parameters. Int'l Gesture Workshop 2003, LNCS vol. 2915. Heidelberg: Springer-Verlag, 124--131.
• 25: Douglas Schuler , Aki Namioka, Participatory Design: Principles and Practices, L. Erlbaum Associates Inc., Hillsdale, NJ, 1993
• 26: John C. Tang, Findings from observational studies of collaborative work, International Journal of Man-Machine Studies, v.34 n.2, p.143-160, Feb. 1991 [doi>10.1016/0020-7373(91)90039-A]
• 27: Edward Tse , Chia Shen , Saul Greenberg , Clifton Forlines, Enabling interaction with single user applications through speech and gestures on a multi-user tabletop, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133336]
• 28: Stephen Voida , Mark Podlaseck , Rick Kjeldsen , Claudio Pinhanez, A study on the manipulation of 2D objects in a projector/camera-based augmented reality environment, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055056]
• 29: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 30: Daniel Wigdor , Darren Leigh , Clifton Forlines , Samuel Shipman , John Barnwell , Ravin Balakrishnan , Chia Shen, Under the table interaction, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166294]
• 31: Andrew D. Wilson, PlayAnywhere: a compact interactive tabletop projection-vision system, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095047]
• 32: Andrew D. Wilson , Shahram Izadi , Otmar Hilliges , Armando Garcia-Mendoza , David Kirk, Bringing physics to the surface, Proceedings of the 21st annual ACM symposium on User interface software and technology, October 19-22, 2008, Monterey, CA, USA [doi>10.1145/1449715.1449728]
• 33: Jacob O. Wobbrock , Htet Htet Aung , Brandon Rothrock , Brad A. Myers, Maximizing the guessability of symbolic input, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057043]
• 34: Mike Wu , Ravin Balakrishnan, Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.193-202, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964718]
• 35: Mike Wu , Chia Shen , Kathy Ryall , Clifton Forlines , Ravin Balakrishnan, Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces, Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems, p.185-192, January 05-07, 2006 [doi>10.1109/TABLETOP.2006.19]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design; Evaluation/methodology

General Terms:
Design, Human Factors

Keywords:
gesture recognition, gestures, guessability, referents, signs, surface, tabletop, think-aloud