Bare-hand human-computer interaction

ABSTRACT

In this paper, we describe techniques for barehanded interaction between human and computer. Barehanded means that no device and no wires are attached to the user, who controls the computer directly with the movements of his/her hand.Our approach is centered on the needs of the user. We therefore define requirements for real-time barehanded interaction, derived from application scenarios and usability considerations. Based on those requirements a finger-finding and hand-posture recognition algorithm is developed and evaluated.To demonstrate the strength of the algorithm, we build three sample applications. Finger tracking and hand posture recognition are used to paint virtually onto the wall, to control a presentation with hand postures, and to move virtual items on the wall during a brainstorming session. We conclude the paper with user tests, which were conducted to prove the usability of bare-hand human computer interaction.

REFERENCES

• 1: Bérard, F. Vision par ordinateur pour l'interaction homme-machine fortement couplée, Doctoral Theses, Université Joseph Fourier, Grenoble, 1999.
• 2: Andrew Blake , Michael Isard , David Reynard, Learning to track the visual motion of contours, Artificial Intelligence, v.78 n.1-2, p.179-212, Oct. 1995 [doi>10.1016/0004-3702(95)00032-1]
• 3: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
• 4: Crowley, J., Bérard, F., and Coutaz, J. Finger tacking as an input device for augmented reality, Automatic Face and Gesture Recognition, Zürich, 195--200, 1995.
• 5: William T. Freeman , David B. Anderson , Paul A. Beardsley , Chris N. Dodge , Michal Roth , Craig D. Weissman , William S. Yerazunis , Hiroshi Kage , Kazuo Kyuma , Yasunari Miyake , Ken-ichi Tanaka, Computer Vision for Interactive Computer Graphics, IEEE Computer Graphics and Applications, v.18 n.3, p.42-53, May 1998 [doi>10.1109/38.674971]
• 6: Laptev, I. and Lindeberg, T. Tracking of Multi-State Hand Models Using Particle Filtering and a Hierarchy of Multi-Scale Image Features, Technical report ISRN KTH/NA/P-00/ 12-SE, September 2000.
• 7: Lee, J. and Kunii, T. Constraint-based hand animation, in Models and techniques in computer animation, 110--127, Springer Verlag, Tokyo, 1993.
• 8: Lien, C. and Huang, C. Model-Based Articulated Hand Motion Tracking For Gesture Recognition, Image and Vision Computing, vol. 16, no. 2, 121--134, February 1998.
• 9: John MacCormick , Michael Isard, Partitioned Sampling, Articulated Objects, and Interface-Quality Hand Tracking, Proceedings of the 6th European Conference on Computer Vision-Part II, p.3-19, June 26-July 01, 2000
• 10: I. Scott MacKenzie , Colin Ware, Lag as a determinant of human performance in interactive systems, Proceedings of the SIGCHI conference on Human factors in computing systems, p.488-493, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169431]
• 11: Rochelle O'Hagan , Alexander Zelinsky, Finger Track - A Robust and Real-Time Gesture Interface, Proceedings of the 10th Australian Joint Conference on Artificial Intelligence: Advanced Topics in Artificial Intelligence, p.475-484, November 30-December 04, 1997
• 12: Quek, F., Mysliwiec, T. and Zhao, M. Finger mouse: A freehand pointing interface, International Workshop on Automatic Face- and Gesture-Recognition, Zürich, 1995.
• 13: James M. Rehg , Takeo Kanade, DigitEyes: Vision-Based Human Hand Tracking, Carnegie Mellon University, Pittsburgh, PA, 1993
• 14: Yoichi Sato , Yoshinori Kobayashi , Hideki Koike, Fast Tracking of Hands and Fingertips in Infrared Images for Augmented Desk Interface, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.462, March 26-30, 2000
• 15: Jakub Segen , Senthil Kumar, Gesture VR: vision-based 3D hand interace for spatial interaction, Proceedings of the sixth ACM international conference on Multimedia, p.455-464, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290822]
• 16: Stafford-Fraser, J. Video-Augmented Environments, PhD theses, Gonville & Caius College, University of Cambridge, 1996.
• 17: J. Triesch , C. von der Malsburg, Robust classification of hand postures against complex backgrounds, Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG '96), p.170, October 14-16, 1996
• 18: Colin Ware , Ravin Balakrishnan, Reaching for objects in VR displays: lag and frame rate, ACM Transactions on Computer-Human Interaction (TOCHI), v.1 n.4, p.331-356, Dec. 1994 [doi>10.1145/198425.198426]
• 19: Xiaojin Zhu , Jie Yang , Alex Waibel, Segmenting Hands of Arbitrary Color, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.446, March 26-30, 2000

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.5 Implementation
          Subjects: Interactive systems

General Terms:
Algorithms, Design, Experimentation, Human Factors

Keywords:
bare-hand control, computer vision, finger tracking, hand-posture recognition, human-computer interaction, real-time