The design and evaluation of multi-finger mouse emulation techniques

ABSTRACT

We explore the use of multi-finger input to emulate full mouse functionality, such as the tracking state, three buttons, and chording. We first present the design space for such techniques, which serves as a guide for the systematic investigation of possible solutions. We then perform a series of pilot studies to come up with recommendations for the various aspects of the design space. These pilot studies allow us to arrive at a recommended technique, the SDMouse. In a formal study, the SDMouse was shown to significantly improve performance in comparison to previously developed mouse emulation techniques.

REFERENCES

• 1: Pär-Anders Albinsson , Shumin Zhai, High precision touch screen interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642631]
• 2: Ravin Balakrishnan , Thomas Baudel , Gordon Kurtenbach , George Fitzmaurice, The Rockin'Mouse: integral 3D manipulation on a plane, Proceedings of the SIGCHI conference on Human factors in computing systems, p.311-318, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258778]
• 3: Hrvoje Benko , Andrew D. Wilson , Patrick Baudisch, Precise selection techniques for multi-touch screens, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124963]
• 4: Peter Brandl , Clifton Forlines , Daniel Wigdor , Michael Haller , Chia Shen, Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces, Proceedings of the working conference on Advanced visual interfaces, May 28-30, 2008, Napoli, Italy [doi>10.1145/1385569.1385595]
• 5: William Buxton , Ralph Hill , Peter Rowley, Issues and techniques in touch-sensitive tablet input, Proceedings of the 12th annual conference on Computer graphics and interactive techniques, p.215-224, July 1985
• 6: Paul Dietz , Darren Leigh, DiamondTouch: a multi-user touch technology, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502389]
• 7: Florian Echtler , Manuel Huber , Gudrun Klinker, Shadow tracking on multi-touch tables, Proceedings of the working conference on Advanced visual interfaces, May 28-30, 2008, Napoli, Italy [doi>10.1145/1385569.1385640]
• 8: Alan Esenther , Kathy Ryall, Fluid DTMouse: better mouse support for touch-based interactions, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133289]
• 9: Fingerworks, Inc. (2008). User's Guide. http://www.fingerworks.com/gesture_guide_mouse.html
• 10: George Fitzmaurice , Azam Khan , Robert Pieké , Bill Buxton , Gordon Kurtenbach, Tracking menus, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.71-79, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964704]
• 11: Clifton Forlines , Daniel Vogel , Ravin Balakrishnan, HybridPointing: fluid switching between absolute and relative pointing with a direct input device, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166286]
• 12: Clifton Forlines , Daniel Wigdor , Chia Shen , Ravin Balakrishnan, Direct-touch vs. mouse input for tabletop displays, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240726]
• 13: Tovi Grossman , Ken Hinckley , Patrick Baudisch , Maneesh Agrawala , Ravin Balakrishnan, Hover widgets: using the tracking state to extend the capabilities of pen-operated devices, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124898]
• 14: Tovi Grossman , Daniel Wigdor , Ravin Balakrishnan, Multi-finger gestural interaction with 3d volumetric displays, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029644]
• 15: Hager-Ross, C., Schieber, M. (2000). Quantifying the Independence of Human Finger Movements: Comparisions of Digits, Hands, and Movement Frequencies. J. of Neuroscience, 20(22), 8542--8550.
• 16: Jefferson Y. Han, Low-cost multi-touch sensing through frustrated total internal reflection, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095054]
• 17: Julien Letessier , François Bérard, Visual tracking of bare fingers for interactive surfaces, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029652]
• 18: Malik, S. (2007). An Exploration of Multi-Finger Interaction on Multi-Touch Surfaces. Univ. of Toronto.
• 19: Shahzad Malik , Abhishek Ranjan , Ravin Balakrishnan, Interacting with large displays from a distance with vision-tracked multi-finger gestural input, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095042]
• 20: Moscovich, T. (2007). Principles and Applications of Multi-touch Interaction. Brown University.
• 21: Tomer Moscovich , John F. Hughes, Multi-finger cursor techniques, Proceedings of Graphics Interface 2006, June 07-09, 2006, Quebec, Canada
• 22: Tomer Moscovich , John F. Hughes, Indirect mappings of multi-touch input using one and two hands, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357254]
• 23: Alex Olwal , Steven Feiner , Susanna Heyman, Rubbing and tapping for precise and rapid selection on touch-screen displays, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357105]
• 24: R. L. Potter , L. J. Weldon , B. Shneiderman, Improving the accuracy of touch screens: an experimental evaluation of three strategies, Proceedings of the SIGCHI conference on Human factors in computing systems, p.27-32, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57171]
• 25: Jun Rekimoto, SmartSkin: an infrastructure for freehand manipulation on interactive surfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503397]
• 26: Andrew Sears , Ben Shneiderman, High precision touchscreens: design strategies and comparisons with a mouse, International Journal of Man-Machine Studies, v.34 n.4, p.593-613, April 1991 [doi>10.1016/0020-7373(91)90037-8]
• 27: Daniel Vogel , Patrick Baudisch, Shift: a technique for operating pen-based interfaces using touch, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240727]
• 28: Christian von Hardenberg , François Bérard, Bare-hand human-computer interaction, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971513]
• 29: Mike Wu , Ravin Balakrishnan, Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.193-202, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964718]
• 30: Mike Wu , Chia Shen , Kathy Ryall , Clifton Forlines , Ravin Balakrishnan, Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces, Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems, p.185-192, January 05-07, 2006 [doi>10.1109/TABLETOP.2006.19]
• 31: Zhengyou Zhang , Ying Wu , Ying Shan , Steven Shafer, Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971522]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Human Factors

Keywords:
mouse emulation, multi-finger input, multi-touch displays