Visual touchpad: a two-handed gestural input device

ABSTRACT

This paper presents the Visual Touchpad, a low-cost vision-based input device that allows for fluid two-handed interactions with desktop PCs, laptops, public kiosks, or large wall displays. Two downward-pointing cameras are attached above a planar surface, and a stereo hand tracking system provides the 3D positions of a user's fingertips on and above the plane. Thus the planar surface can be used as a multi-point touch-sensitive device, but with the added ability to also detect hand gestures hovering above the surface. Additionally, the hand tracker not only provides positional information for the fingertips but also finger orientations. A variety of one and two-handed multi-finger gestural interaction techniques are then presented that exploit the affordances of the hand tracker. Further, by segmenting the hand regions from the video images and then augmenting them transparently into a graphical interface, our system provides a compelling direct manipulation experience without the need for more expensive tabletop displays or touch-screens, and with significantly less self-occlusion.

REFERENCES

• 1: William A. S. Buxton, Telepresence: integrating shared task and person spaces, Proceedings of the conference on Graphics interface '92, p.123-129, September 1992, Vancouver, British Columbia, Canada
• 2: Chen, X., Koike, H., Nakanishi, Y., Oka, K., Sato, Y. (2002). Two-handed drawing on augmented desk system. In Proceedings of Advanced Visual Interfaces (AVI). pp. 219--222.
• 3: Corso, J., Burschka, D., Hager, G. (2003). The 4D Touchpad: Unencumbered HCI with VICs. In Proceedings of CVPR-HCI.
• 4: Olivier Faugeras , Quang-Tuan Luong , T. Papadopoulou, The Geometry of Multiple Images: The Laws That Govern The Formation of Images of A Scene and Some of Their Applications, MIT Press, Cambridge, MA, 2001
• 5: Guiard, Y. (1987). Asymmetric Divison of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model. Journal of Motor Behavior, 19(4). pp. 486--517.
• 6: Christian von Hardenberg , François Bérard, Bare-hand human-computer interaction, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971513]
• 7: Hiroshi Ishii , Minoru Kobayashi, ClearBoard: a seamless medium for shared drawing and conversation with eye contact, Proceedings of the SIGCHI conference on Human factors in computing systems, p.525-532, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142977]
• 8: Paul Kabbash , William Buxton , Abigail Sellen, Two-handed input in a compound task, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.417-423, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191808]
• 9: Kolsch, M., Turk, M. (2002). Keyboards without Keyboards: A Survey of Virtual Keyboards. Technical Report 2002-21. University of California, Santa Barbara.
• 10: Krueger, M. (1991). VIDEOPLACE and the Interface of the Future. In The Art of Human Computer Interface Design. Addison Wesley, Menlo Park, CA. pp. 417--422.
• 11: Gordon Kurtenbach , George Fitzmaurice , Thomas Baudel , Bill Buxton, The design of a GUI paradigm based on tablets, two-hands, and transparency, Proceedings of the SIGCHI conference on Human factors in computing systems, p.35-42, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258574]
• 12: John MacCormick , Michael Isard, Partitioned Sampling, Articulated Objects, and Interface-Quality Hand Tracking, Proceedings of the 6th European Conference on Computer Vision-Part II, p.3-19, June 26-July 01, 2000
• 13: Mysliwiec, T. (1994). FingerMouse: A Freehand Computer Pointing Interface. Technical Report VISLab-94-001, University of Illinois Chicago.
• 14: Kenji Oka , Yoichi Sato , Hideki Koike, Real-Time Tracking of Multiple Fingertips and Gesture Recognition for Augmented Desk Interface Systems, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, p.429, May 20-21, 2002
• 15: Jun Rekimoto, SmartSkin: an infrastructure for freehand manipulation on interactive surfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503397]
• 16: Meredith Ringel , Henry Berg , Yuhui Jin , Terry Winograd, Barehands: implement-free interaction with a wall-mounted display, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634284]
• 17: Nicolas Roussel, Exploring New Uses of Video with VideoSpace, Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, p.73-90, May 11-13, 2001
• 18: Jakub Segen , Senthil Kumar, Gesture VR: vision-based 3D hand interace for spatial interaction, Proceedings of the sixth ACM international conference on Multimedia, p.455-464, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290822]
• 19: Stotts, D., Smith, J., and Jen, D. (2003). The Vis-a-Vid Transparent Video FaceTop. In Proceedings of ACM UIST. pp. 57--58.
• 20: John C. Tang , Scott Minneman, VideoWhiteboard: video shadows to support remote collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.315-322, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108932]
• 21: Emanuele Trucco , Alessandro Verri, Introductory Techniques for 3-D Computer Vision, Prentice Hall PTR, Upper Saddle River, NJ, 1998
• 22: Norimichi Ukita , Masatsugu Kidode, Wearable virtual tablet: fingertip drawing on a portable plane-object using an active-infrared camera, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964475]
• 23: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 24: Mike Wu , Ravin Balakrishnan, Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.193-202, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964718]
• 25: Ka-Ping Yee, Two-handed interaction on a tablet display, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986098]
• 26: Zhengyou Zhang , Ying Wu , Ying Shan , Steven Shafer, Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971522]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.m Miscellaneous

General Terms:
Algorithms, Design, Human Factors

Keywords:
augmented reality, computer vision, direct manipulation, fluid interaction, gestures, hand tracking, perceptual user interface, two hand, virtual keyboard, virtual mouse, visual touchpad