Using strokes as command shortcuts: cognitive benefits and toolkit support

ABSTRACT

This paper investigates using stroke gestures as shortcuts to menu selection. We first experimentally measured the performance and ease of learning of stroke shortcuts in comparison to keyboard shortcuts when there is no mnemonic link between the shortcut and the command. While both types of shortcuts had the same level of performance with enough practice, stroke shortcuts had substantial cognitive advantages in learning and recall. With the same amount of practice, users could successfully recall more shortcuts and make fewer errors with stroke shortcuts than with keyboard shortcuts. The second half of the paper focuses on UI development support and articulates guidelines for toolkits to implement stroke shortcuts in a wide range of software applications. We illustrate how to apply these guidelines by introducing the Stroke Shortcuts Toolkit (SST) which is a library for adding stroke shortcuts to Java Swing applications with just a few lines of code.

REFERENCES

• 1: Olivier Bau , Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets, Proceedings of the 21st annual ACM symposium on User interface software and technology, October 19-22, 2008, Monterey, CA, USA [doi>10.1145/1449715.1449724]
• 2: Xiang Cao , Shumin Zhai, Modeling human performance of pen stroke gestures, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240850]
• 3: F. Craik and R. Lockhart. Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6):671--684, 1972.
• 4: Tovi Grossman , Pierre Dragicevic , Ravin Balakrishnan, Strategies for accelerating on-line learning of hotkeys, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240865]
• 5: François Guimbretière , Andrew Martin , Terry Winograd, Benefits of merging command selection and direct manipulation, ACM Transactions on Computer-Human Interaction (TOCHI), v.12 n.3, p.460-476, September 2005 [doi>10.1145/1096737.1096742]
• 6: Ken Hinckley , Shengdong Zhao , Raman Sarin , Patrick Baudisch , Edward Cutrell , Michael Shilman , Desney Tan, InkSeine: In Situ search for active note taking, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240666]
• 7: Jason I. Hong , James A. Landay, SATIN: a toolkit for informal ink-based applications, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.63-72, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354412]
• 8: Paul Kabbash , William Buxton , Abigail Sellen, Two-handed input in a compound task, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.417-423, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191808]
• 9: Per-Ola Kristensson , Shumin Zhai, SHARK²: a large vocabulary shorthand writing system for pen-based computers, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029640]
• 10: Per Ola Kristensson , Shumin Zhai, Command strokes with and without preview: using pen gestures on keyboard for command selection, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240797]
• 11: Gordon Kurtenbach , William Buxton, User learning and performance with marking menus, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.258-264, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191759]
• 12: Gordon Kurtenbach , George W. Fitzmaurice , Russell N. Owen , Thomas Baudel, The Hotbox: efficient access to a large number of menu-items, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.231-237, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303047]
• 13: G. Kurtenbach and T. Moran. Contextual animation of gestural commands. Eurographics Computer Graphics Forum, 13(5):305--314, 1994.
• 14: D. Lane, H. Napier, S. Peres, and A. Sandor. Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. International Journal of Human-Computer Interaction, 18(1):133--144, 2005.
• 15: Allan Christian Long, Jr. , James A. Landay , Lawrence A. Rowe, Implications for a gesture design tool, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.40-47, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.302985]
• 16: A. Chris Long, Jr. , James A. Landay , Lawrence A. Rowe , Joseph Michiels, Visual similarity of pen gestures, Proceedings of the SIGCHI conference on Human factors in computing systems, p.360-367, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332458]
• 17: P. Morrel-Samuels, Clarifying the distinction between lexical and gestural commands, International Journal of Man-Machine Studies, v.32 n.5, p.581-590, May 1990 [doi>10.1016/S0020-7373(05)80034-3]
• 18: Daniel L. Odell , Richard C. Davis , Andrew Smith , Paul K. Wright, Toolglasses, marking menus, and hotkeys: a comparison of one and two-handed command selection techniques, Proceedings of Graphics Interface 2004, p.17-24, May 17-19, 2004, London, Ontario, Canada
• 19: Y Rogers, Evaluating the meaningfulness of icon sets to represent command operations, Proceedings of the Second Conference of the British Computer Society, human computer interaction specialist group on People and computers: designing for usability, p.586-603, December 1986, York, United Kingdom
• 20: Dean Rubine, Specifying gestures by example, ACM SIGGRAPH Computer Graphics, v.25 n.4, p.329-337, July 1991
• 21: C. Tappert. Cursive script recognition by elastic matching. IBM Journal of Research Development, 26(6):765--771, 1982.
• 22: Jacob O. Wobbrock , Andrew D. Wilson , Yang Li, Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes, Proceedings of the 20th annual ACM symposium on User interface software and technology, October 07-10, 2007, Newport, Rhode Island, USA [doi>10.1145/1294211.1294238]
• 23: Catherine G. Wolf, CAN PEOPLE USE GESTURE COMMANDS?, ACM SIGCHI Bulletin, v.18 n.2, p.73-74, Oct. 1986 [doi>10.1145/15683.1044102]
• 24: Catherine G. Wolf , James R. Rhyne , Hamed A. Ellozy, The paper-like interface, Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.), p.494-501, September 1989, Boston, Massachusetts, United States
• 25: Robert Zeleznik , Timothy Miller, Fluid inking: augmenting the medium of free-form inking with gestures, Proceedings of Graphics Interface 2006, June 07-09, 2006, Quebec, Canada
• 26: Shumin Zhai , Per-Ola Kristensson, Shorthand writing on stylus keyboard, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642630]
• 27: Shengdong Zhao , Ravin Balakrishnan, Simple vs. compound mark hierarchical marking menus, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029639]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

General Terms:
Human Factors

Keywords:
gesture, shortcuts, stroke, toolkit